ShopifySharp Version 5 Migration Guide

I'm happy to announce that the ShopifySharp version 5.0 has officially been released and published on Nuget, which means it's time for another migration guide! As with all major version updates, this one has got a few breaking changes in it, which means updating won't be completely seamless, but it's my hope that this guide will show you how to quickly update from v4 to v5.

And update you should! ShopifySharp v4 is using the very first version of Shopify's REST API, which is scheduled to be deprecated on April 1st, 2020. If you don't update before April 1st, some of your API calls will begin to fail; even worse, Shopify will begin to email your users, telling them your app is unstable and uses outdated APIs.

As much as I would have liked to perform a seamless update to the newest version of their API without requiring devs to deal with breaking changes in ShopifySharp, there are changes in Shopify's API that are just fundamentally different, making v4 interop impossible. But fear not noble developer: I've documented the important changes, and I have several suggestions to both make things easier, and make it possible to mimic some of the v4 behavior that now seems, well, impossible.

Read on!

Changes to listing objects, and working with the paginated results

By far the biggest change in ShopifySharp v5.0 is the way you'll be listing objects from now on. In all previous major versions, you were able to request different "pages" of things like orders; for example, you'd use a filter class to request the 20th page of orders, and Shopify would perform whatever magical database incantations they needed to do to find that page, then return it to you.

Sadly, it is no longer possible to request specific pages when listing objects. It seems that Shopify is now doing a linked list implementation instead of a paginated list. That is to say, each page of objects (e.g. orders) now links to the next page and the previous page. That means if you want to get page 20, you must dolefully walk through each linked page, starting with page one, until you get there; it's impossible to request it directly.

To support this change, we've updated almost all ListAsync methods across the package to accept a new ListFilter<T> object; you must use it when you're making a request to get the next page. This filter class is special, because it has a PageInfo property; without this property, Shopify will only ever return the first page of items, so it's the key to requesting pages beyond the first.

The PageInfo value is just a short string of random letters and numbers, and it always comes from Shopify -- it's never going to be something you create yourself. For every list request, Shopify will be returning this PageInfo value along with the objects, and behind the scenes ShopifySharp will be looking for it.

Excluding a few specific service classes, each list request will now return a ListResult<T> object which contains both the items you requested along with the PageInfo values for getting the next page or the previous page. You plug the value into a new ListFilter<T> and pass it back to the list method, and that gets you the next page.

We've tried to make this as easy as possible for you with ShopifySharp by adding a couple of extension methods to the list result object. You can use listResult.HasNextPage and listResult.HasPreviousPage to check if you're at the end of the list, and then use listResult.GetNextPageFilter() or listResult.GetPreviousPageFilter() to create a filter which can be passed directly to your next list call.

Everybody likes to see examples though, so let's look at some code! One of the most common questions I get from new ShopifySharp users is "how do I list all of the orders/customers/etc on a store". Here's what that now looks like in version 5.0 and above:

var service = new OrderService(domain, accessToken);
var allOrders = new List<Order>();
var page = await service.ListAsync(new OrderListFilter { Limit = 250 });

while (true)
{
    allOrders.AddRange(page.Items);

    if (! list.HasNextPage)
    {
        break;
    }

    page = await service.ListAsync(list.GetNextPageFilter());
}

One very important thing to note is that aside from the Limit and Fields properties, all other properties on a filter become immutable once you list the first page. For example, if you set CreatedAtMin when listing the first page, that value is automatically applied to the next page. This is not an implementation decision we made for ShopifySharp, it's just the way the Shopify API works works now. If you want to change anything other than Limit and Fields, you need to start over with a new filter on page one.

Oh, and since I'm a diehard F# proselytizer, here's how you list all of the orders using F# and ShopifySharp v5.0:

let service = OrderService(domain, accessToken)

let rec getAllOrders (nextPageFilter : ListFilter<Order>) (previousOrders : Order list) = 
    async {
        let! page = 
            service.ListAsync nextPageFilter 
            |> Async.AwaitTask 
        let combinedOrders = 
            List.concat [previousOrders; page.Items]

        match page.HasNextPage with 
        | false ->
            combinedOrders
        | true ->
            let newFilter = page.GetNextPageFilter()

            return! getAllOrders newFilter combinedOrders
    }

let filter = OrderListFilter()
filter.Limit <- Nullable 250 

let! allOrders = getAllOrders filter List.empty

This is still pretty csharpy, but a more functional version of ShopifySharp is coming soon!

If you're curious about what's going on underneath the hood when ShopifySharp is listing things, here's how it works: whenever you make a list request, Shopify will return the very first page of those objects, typically sorted from newest to oldest. You can only list up to 250 objects at a time, so if more than that amount exist, Shopify will append an HTTP header named Link to the response.

The value of that header looks something like this:

Link: "<https://test.myshopify.com/admin/api/2019-07/products.json?limit=3&page_info=abcdefg&fields=id%2Ccustomer%2Ccreated_at>; rel=\"previous\", <https://test.myshopify.com/admin/api/2019-07/products.json?limit=3&page_info=opqrstu&fields=id%2Ccustomer%2Ccreated_at>; rel=\"next\""

It looks a little bit like faux HTML, almost as if you could embed it in a web page.More importantly, you'll see that there's actually two links inside that header, with rel="next" and rel="previous" indicating which one points to the next 250 items and which one points to the previous 250 items.

See where this is going? It's Computer Science 101, they've turned the API into one big linked list where each page contains the information needed to get to the next page or the previous page. There is no longer a concept of page one, page two, page three, and so on. What you would need to do (if you were to implement this yourself instead of using ShopifySharp, which does it all for you) is parse out those links from the returned header and grab the page_info={value} part specifically. That value is the key to unlocking the next or previous pages, so once you have it it's just a simple matter of appending it to the querystring in the subsequent request.

Fancy parsing aside, what about real world impact? Does this truly change anything for those of us with existing Shopify applications? In a word, yes: if your Shopify application shows users a list of things (orders, in my own case), it's no longer possible to let the user jump directly to arbitrary pages. At best, you can only show them the newest 250 items, and then add a link to the next 250 items. They can't just jump to the end of the list anymore, or anywhere in between.

If your app currently implements that sort of behavior like mine does, and you want to preserve that behavior, the best thing you can do is walk through the entire list using the API and import them directly into a database you control. Then you can use ShopifySharp's WebhookService to import new orders/customers/whatevers as they're created. This means you're in control of the data, and you can write the queries to know exactly how many pages are available.

On the other hand, if you're fine replacing that kind of pagination with simple Next Page / Previous Page links, your life is a lot simpler. Remember that page_info={value} thing from the link header discussed above? You're able to get that value directly with ShopifySharp and save it, then reconstruct a list filter using the value at a later point. In practice, I would recommend appending the page_info value to the HTML querystring for next/previous page links.

public class OrdersController : Controller
{
    public IActionResult Index([FromQuery] string page_info)
    {
        int defaultLimit = 250;
        ListFilter<Order> pageFilter;

        if (!string.IsNullOrEmpty(page_info))
        {
            // Reconstruct a ListFilter to continue walking up or down the pages
            // Remember, only the very first page in a list can use filter parameters other than Limit/Fields; 
            // all subsequent requests remember those parameters
            pageFilter = new ListFilter<Order>(page_info, defaultLimit);
        }
        else
        {
            // User is requesting the very first page
            pageFilter = new OrderListFilter
            {
                Limit = limit
            };
        }

        var service = new OrderService(domain, accessToken);
        var result = await service.ListAsync(pageFilter);

        return View(result);
    }
}

And then in your view you'd do something like this:

@model ShopifySharp.Lists.ListResult<Order>

<div>
    <!-- HTML here -->

    @if (Model.HasPreviousPageLink) {
        <a href="/Orders?page_info=@Model.PreviousLink.PageInfo">
            Previous Page
        </a>
    }

    @if (Model.HasNextPageLink) {
        <a href="/Orders?page_info=@Model.NextLink.PageInfo">
            Next Page
        </a>
    }
</div>

In theory, this new method for pagination should result in faster request execution because it seems like Shopify is caching the results and looking them up by the PageInfo property on each filter. In my limited testing so far, it does seem like this is the case; the first time you access a new page, there's a short delay as (I assume) Shopify populates a cache, but subsequent calls to the same pages return very quickly.

Sidenote: Shopify advises against storing the page_info parameter, but they don't explain why. In my opinion, this is an irrational recommendation and does not reflect the reality of building an app. It's not realistic to walk through dozens of requests every time a user wants to view the next page in a list.

Many list and count filters changed

In version 4.0 of ShopifySharp, I went a little bit overboard with the "Don't Repeat Yourself (DRY)" mantra by trying to extend and inherit filter classes as much as possible. In practice, this actually led to many of the filter classes containing properties that were simply not relevant and not applicable to the things being listed or counted. For example, there were several filter classes in v4 that had properties like PublishedAtMin and PublishedAtMax where were not used and completely ignored when sent to Shopify.

In version 5.0, I've gone in the opposite direction, where you'll find that previous generic filter classes like PublishableListFilter and PublishableCountFilter no longer exist. Each filter class has been handcrafted to contain only the properties that are relevant to the operation it's being used in. The OrderListFilter only has properties relevant to filtering lists of orders; the ProductCountFilter only has properties relevant to filtering counts of products; and so on.

The only exception here is the Fields property, which can be found on all list filters. This property does not apply to all endpoints (although it does apply to most endpoints). It's a sacrifice I had to make to get automatic population of the Fields and Limit values when creating a filter for the next or previous pages with listResult.GetNextPageFilter() and listResult.GetPreviousPageFilter().

Several deprecated properties and methods have been removed

There were a handful of properties that were deprecated in ShopifySharp v4.0, and have been removed in ShopifySharp v5.0. These include (but may not be limited to) the following:

• ProductVariant.InventoryQuantity: deprecated by Shopify; including it when updating a product variant would cause Shopify to throw an error instead of processing the request.
  • UPDATE: it's come to my attention that this property was not actually removed like I had intended. Right now, if the property has any other value than null when you try to update a product, you will encounter an error.
• The CheckoutService class has had several of its methods deprecated. They continue to function, but Shopify no longer documents them and they stop working at any time. These methods will be removed in a future release.
• OrderService.ListForCustomerAsync() was moved to CustomerService.ListOrdersForCustomerAsync().

Better ShopifyException messages!

Since its inception, ShopifySharp has had a special exception class called the ShopifyException which is always thrown when a request to Shopify fails. Unfortunately, the exception message has not always been very helpful. In versions 4 and below, the package would try very hard to parse out a usable message from Shopify's response, but more often than not it would fail and default to a generic error message like "Response did not indicate success, the server returned 406 Not Acceptable".

While that generic error message checked the most basic box of telling the developer something went wrong (as if the exception itself didn't already do that), it was almost entirely worthless when it came to debugging the issue. And as much as I criticize Shopify for being poorly documented and obscure in some areas, this was one was largely an issue with the package and not the API.

Shopify certainly didn't do us any favors here though, thanks to their multiple different undocumented error formats. For almost every request that returns a non-200 status code, Shopify returns some kind of error message trying to tell you what's wrong and what needs to change. The issue was that there are at least five different formats they use, and as far as I can tell there's no consistency or methodology to which format is going to be used.

Each time the Shopify API returns an error, ShopifySharp has to look at the response body and test it to figure out how it can parse out the error message. Again, there are five known error formats, so which of the following is it going to be?

1. Does it look like {"error": "message here"}?
2. Does it look like {"errors": "message here"}?
3. Does it look like {"errors": { "order": "message here" }}?
4. Does it look like {"errors": { "order": ["message here"] }}?
5. Does it look like {"error": "invalid_request", "error_description": "message here"}?

There are also some edge cases, including at least one endpoint where their JSON API will inexplicably return error messages wrapped in HTML instead of JSON.

The package now astutely handles all of these different formats, correctly pulling out the error message and stuffing it into the ShopifyException, where it should help developers quickly debug any issues they discover in production. The only time the package will fall back to a generic exception message is when A) the response does not contain any JSON (such as that HTML edge case); B) when the response does not match one of five known error message formats; or C) the response contains no JSON at all.

Most of these cases should be rare.

Location IDs are now required when creating a fulfillment

This change has been coming for a long time, but it's now finally officially in version 5.0 (or rather, in the version of the Shopify API that 5.0 now targets). When you create a fulfillment, you must pass in a LocationId, or else Shopify will return an error and refuse to process the request.

You can use the locations API and ShopifySharp's corresponding LocationService to get the ID of a location.

Here's a link to Shopify's documentation on managing fulfillments, where they talk a little bit more about locations.

Plans for future releases

We've got some neat stuff in the works for future releases! One thing high up on my radar is to go completely dependency-less in one of the next major versions. In version 4.0, we dropped the dependency on Flurl and RestSharp, meaning there are two dependencies remaining at this time: Microsoft.Extensions.Primitives, and Newtonsoft.Json.

The primitives package is one that isn't really needed. It was introduced at a time when we were transitioning to .NET Standard, and I had mistakenly believed that the StringValues type -- one used largely (perhaps exclusively?) by ASP.NET Core -- was something we'd have to be supporting. To be clear, this type is only used when validating certain Shopify requests, because ASP.NET Core would use the type for request header values. It's trivially simple to turn the type into a plain old string, so there's no need for ShopifySharp to pull in this whole other package to work with it.

Newtonsoft.Json, on the other hand, is practically a household name when it comes to .NET packages. Everyone knows what this is, but just in case you've somehow never encountered it before, Newtonsoft.Json is a JSON serialization and deserialization package. In ShopifySharp, it's responsible for taking those classes you're using and serializing them into a string that Shopify can read; then, when Shopify sends a response, this package takes the response body and deserializes it back into a ShopifySharp class.

It's a tremendously effective and easy to use package, so why do I want to replace it? Simply put, it's just nice to have zero dependencies in a package. This means there's no chance for developers to run into version conflicts, which isn't uncommon thanks to the ubiquity of Newtonsoft.Json in particular.

Recently Microsoft introduced the new System.Text.Json namespace, which, according to the overview, "emphasizes high performance and low memory allocation over an extensive feature set". This new namespace is built in to .NET Core 3.0, which means no external dependencies are needed! For other frameworks, it's available as a nuget package.

What that means for ShopifySharp is we can remove the Newtonsoft.Json dependency entirely, use System.Text.Json when targeting .NET Core 3.0, and bring in the System.Text.Json package for any older targets.

Beyond aiming for a zero-dependency future, I'm looking to introduce a new ShopifySharp.Fluent namespace for updating and creating objects with a fluent style. The goal is to let developers take total control of exactly which properties are being serialized and sent to Shopify. For example, if you wanted to updated a Shopify product's title, you'd do something like this with the new fluent namespace:

var product = new ProductBuilder()
    .SetTitle("My New Product Title");

var updatedProduct = await productService.UpdateAsync(product);

// When serialized, product builder turns into this string:
// {"title":"My New Product Title"}

So as the example shows, once ShopifySharp serializes the product builder, it will only send the value of the Title property to Shopify. Sounds obvious, but that's because the normal method for updating a product (the one we have right now in v5.0 and lower) has some very unobvious behavior. Let's take that fluent example above and convert it into the typical way you'd update a product with ShopifySharp v5 and below:

var product = new Product
{
    Title = "My New Product Title"
};

var updatedProduct = await productService.UpdateAsync(product);

// When serialized, the product turns into this JSON string:
// {"title":"My New Product Title","id":null,"created_at":null,"updated_at":null,"published_at":null, ...}

That JSON is very different from the JSON in the fluent example. Because of the nature of C# and the way JSON serialization works, every property on the Product class gets serialized and shows up in the final result. Even though the example never sets the Id property, you can still see it in the JSON; even though it never sets the CreatedAt property, you can still see it in the JSON; and so on.

Now, in many cases, this isn't a huge deal. If Shopify sees a property that's null or that doesn't belong, it might just ignore it or assign a default value. The real problem arises when null is a valid value.

Case in point: products can be published or unpublished from a Shopify store. When unpublished, the product will show up in the merchant's admin panel, but it will not be available for purchase on the storefront. How do you unpublish a product? You set the PublishedAt value to null. That's where the problem arises: if you're not careful, or more likely, if you don't already know about this behavior, it is extremely easy to unpublisha product when you just want to do something simple like update its title or price.

In the example above using the Product class, the update operation would unpublish the product even though the PublishedAt property was never touched in the first place.

Some knowledgable readers may be ready to point out that there are attributes you can use with Newtonsoft.Json to control when null values are serialized. It is possible to use those attributes and prevent the property from appearing in the JSON string at all if its value was null. However, this then has implications for when you want to set a value to null on purpose. We quickly descend into a miasma of sticky situations and obscure rules, where it's impossible to figure out what's actually going to get serialized without reading the source code.

In my opinion, a fluent builder is the best step forward; it will let developers control exactly which properties and values are going to be sent to Shopify because you'll need to explicitly set them.

Oh yeah, because I'm such an F# fanboy, I've also got a plan to create a pipe-oriented object builder for those lovers of functional programming:

let product = 
    ProductBuilder.create
    |> setTitle "My New Product Title"
    |> setPublished DateTimeOffset.Now 
    |> setTags "cool product"

So those are my plans for ShopifySharp's future! Right now the plan is to target each new version of Shopify's API as they release, so I hope to iterate quickly on these plans as new versions come out. If you've got feedback or questions, please feel free to open an issue on GitHub or reach out to me at joshua@nozzlegear.com.