Listing all orders, customers, articles or anything else with the Shopify API

As the author of two packages for working with the Shopify API -- ShopifySharp for C# apps, and Shopify Prime for TypeScript and JavaScript -- I often field questions about "weird behavior" from developers using the packages. Someimtes there's a bug, sometimes support is missing for a new feature, sometimes an old feature has been removed and stopped working, and sometimes the behavior seems broken but isn't.

The questions about seemingly broken behavior are the questions that I get the most often. One of the most popular questions in that "series" is how do I search for a Shopify order by its name, not its id? But another popular question is "how do I list all of the orders, not just the first 50? I use the .ListAsync() method in ShopifySharp, but it returns 50 when the store has thousands!"

This confusion doesn't just apply to ShopifySharp, because the behavior is actually just the way the Shopify API works. They "paginate" the response of all list calls to 50 objects by default, and up to 250 objects with the limit=250 parameter. From their perspective, a shop could have thousands, tens of thousands, or even hundreds of thousands of whatever object you're trying to list. Not only would that be a yuge payload for Shopify to convert to JSON and send back, but an app might even DDOS itself just trying to read all of that data.

So instead of sending potentially dozens or hundreds of megabytes of JSON to the caller, Shopify limits the response of "list" calls and expects you to loop through each "page" if you want to get them all. To loop through a page, you also need to send the page=XYZ parameter with your request. Both ShopifySharp and Shopify Prime support looping through these pages, but you also need to know how many orders there are in total before you can list them all. Luckily, Shopify also has an API call you can make to count all of your objects before you list them, and both packages support this too.

Let's take a look at some real world examples, one in C# using ShopifySharp, and one in TypeScript using Shopify Prime. These are snippets of code that I use in my own applications.

Listing all Shopify orders with C# and ShopifySharp

public async Task<IEnumerable<ShopifySharp.Order>> GetAllOrders(string domain, string accessToken)
{
    var service = new ShopifySharp.OrderService(domain, accessToken);
    var total = await service.CountAsync();
    var orders = new List<ShopifySharp.Order>();
    int limit = 250;
    int totalPages = total % limit > 0 ? (total / limit) + 1 : total % limit;
    int page = 1;

    while (page <= totalPages)
    {
        var pageOfOrders = await service.ListAsync(new ShopifySharp.Filters.OrderFilter
        {
            Limit = limit,
            Page = page
        });

        orders.AddRange(pageOfOrders);
        page++;
    }
}

Listing all Shopify orders with TypeScript/JavaScript and Shopify Prime

Warning: I have since deprecated and discontinued Shopify Prime, my TypeScript package for working with the Shopify API. I strongly suggest using ShopifySharp instead if you're not tied to JavaScript or TypeScript. The following example is just for historical purposes.

This example is in TypeScript, but you can easily convert it to fully-functional JavaScript by removing the type annotations.

import { Orders } from "shopify-prime";
import { Order } from "shopify-prime/models";

async function listAllOrders(domain: string, accessToken: string): Promise<Order[]> {
    const service = new Orders(domain, accessToken);
    const total = await service.count();
    const limit = 250;
    const totalPages = Math.ceil(total / limit);
    const orders: Order[] = [];
    let page = 1;

    while (page <= totalPages) {
        const pageOfOrders = await service.list({
            limit: limit,
            page: page
        });

        orders.push(pageOfOrders);
        page++;
    }
}

Beware the rate limit

Although the above examples will list all of the orders/customers/foos/bars on a shop, you'll quickly run into Shopify's rate limit and your requests will be denied. In ShopifySharp, this denial will throw an error. Shopify only allows your application to make two calls per second to any shop, with a burst of up to 40 calls followed by a short "cooldown" period.

Once the rate limit is thrown, there are a few different methods you can employ for handling it. The simplest (and probably the most inefficient) solution is to just wait for 2 seconds before trying again, but first you need to catch the error. In ShopifySharp, a special ShopifyRateLimitException is thrown that you can specifically catch:

while (page <= totalPages)
{
    try
    {
        var pageOfOrders = ...;
        orders.AddRange(pageOfOrders);

        // Only move to the next page once the page is retrieved without throwing a rate limit exception
        // WARNING: Your program could be caught here for a very long time, this is very inefficient!
        page++;
    }
    catch (ShopifySharp.ShopifyRateLimitException ex)
    {
        // A rate limit error was thrown. Wait for two seconds and try again.
        Thread.Sleep(2000);
    }
}

Shopify Prime did not have a specific error to throw when a rate limit error occurred. Instead, you would have needed to catch the error and check its status code. Shopify will return a 429 Too Many Requests status once the rate limit has been hit. However, do to the single-threaded, asynchronous nature of JavaScript, it's a lot harder to just wait in a loop; you'd have to make use of promises:

import { ShopifyError } from "shopify-prime";
...

// getPage() is specifically not an async function -- we want to return a promise that completes when we say it does.
const getPage = (page: number, limit: number) => new Promise<Order[]>(async (resolve, reject) => {
    // TODO: Expand function here to set a timeout when a rate limit occurs and then recursively call itself again
    try {
        const pageOfOrders = await service.list({
            limit: limit,
            page: page
        });

        resolve(pageOfOrders);
        return;
    } catch (e) {
        if (e instanceof ShopifyError && e.statusCode === 429) {
            // A rate limit error was thrown. Wait for two seconds and try again.
            setTimeout(() => {
                // Recursion!
                getPage(page, limit).then(resolve);
            }, 2000);
        } else {
            reject(e);
        }

        return;
    }
});

...

while (page < totalPages) {
    const pageOfOrders = await getPage(page, limit);

    orders.push(pageOfOrders);
    page++;
}

Again, implementing a simple "wait" is probably the simplest but most error prone method you can use for handling Shopify's API rate limit. Even the two examples above aren't perfect, there are a lot of little things you need to watch for and get right. In ShopifySharp, you can use the built-in execution policies to handle retries for you. Since this post is about listing all of the objects on a Shopify store, handling implementing retry algorithms is a bit beyond the scope of that goal.

If you're looking to improve this rate limit handling, I'll be writing another post soon that specifically deals with handling the Shopify API rate limit in C# using ShopifySharp. In the mean time, if you want to try your hand at creating a retry strategy on your own, you might want to start with learning about the "leaky bucket" algorithm, which is what ShopifySharp uses for its "smart retry" execution policy.