Shopify in Legacy Systems

Impressionist rendering of the Shopify logo generated by DALL-E 2.

Introduction

A while back, the team at Dome and I were given a task to notify customers when their products were back in stock.

For context, the project we're running is what we would kindly refer to as a legacy project (an old version of Django, using Mezzanine for the Admin interface, Shopify with REST, a semi-supported Mailchimp package called mailchimp3, and 1000s of lines of code).

The Problem

This is what needed to happen:

  1. Customer sees an out-of-stock product
  2. Customer submits the email
  3. Shopify will fire an event when a product is back in stock
  4. That will hit our webhook endpoint, and we'll send an email with Mailchimp

Setbacks

Setting Up the Webhook

Shopify is great. It just happens to be a massive collection of products, APIs, and services. After a fair bit of research, we assumed (wrongly so) that the REST API would be the way to go.

While GraphQL had a specific page for the exact functionality we were looking for, REST did not. So we ended up going with a simple GraphQL query.

mutation webhookSubscriptionCreate($topic: WebhookSubscriptionTopic!, $webhookSubscription: WebhookSubscriptionInput!) {
    webhookSubscriptionCreate(topic: $topic, webhookSubscription: $webhookSubscription) {
        webhookSubscription {
            id
            topic
            format
            apiVersion {
                displayName
                handle
                supported
            }                                   
            endpoint {
                __typename
                ... on WebhookHttpEndpoint {
                    callbackUrl # Our endpoint
                }
            }   
        }
    }
}
json_result = graphql.execute(
    mutation, 
    {
        "topic": "VARIANTS_IN_STOCK",
        "webhookSubscription": {
            "callbackUrl": callback_url,
            "format": "JSON"
        }
    }
)

Shopify Store Types

The webhook wouldn't fire. Why? After countless hours of digging through docs, forums, and issues, we discovered the Shopify store we were using didn't support webhooks.

To get webhook notifications, you needed either a Development Store, or a Plus Store. After transitioning temporarily to a Development Store, we were finally getting hits to our endpoint. Awesome!

It Actually Gets Worse

We actually already had a Development Store previously set up, prior to the functionality of webhooks on Shopify being implemented on Development Stores. This meant that, as far as we could tell, we were doing everything right.

It took us a fair amount of time to find out that Shopify doesn't handle all stores equally, and we needed to transition to a completely new store just to properly get webhooks that worked.

Mailchimp

Mailchimp was quite nice to work with, albeit a bit verbose. First, you create a campaign (yes, even for just one notification), then you add additional content (such as the template to use) with another query, and finally you send the campaign.

client = MailChimp(mc_api=MAILCHIMP_KEY, mc_user=MAILCHIMP_USER)

# Create
campaign = client.campaigns.create({
    'type': 'regular',
    'recipients': {
        'list_id': MAILCHIMP_AUDIENCE_ID,
        'segment_opts': {
            'saved_segment_id': product.mailchimp_segment_id
        }
    },
    'settings': {
        'subject_line': Template(RESTOCKED_SUBJECT_LINE).substitute(name=product.name),
        'reply_to': RESTOCKED_REPLY_TO,
        'from_name': RESTOCKED_FROM_NAME,
    }
})

# Add content
client.campaigns.content.update(
    campaign_id=campaign['id'], 
    data={
        'template': {
            'id': int(RESTOCKED_EMAIL_TEMPLATE),
            'sections': {
                'preheader_leftcol_content': f'<p>{product.name} is in stock!</p>',
                'header_image': image_html,
                'body_content': f'<p>{image_html}</p><p>The product you wanted, <a href="https://'
                                f'{Site.objects.get_current().domain}{product.get_absolute_url()}">{product.name}</a>,'
                                f' is in stock!</p><p>',
            }
        }
    }
)

# Send
client.campaigns.actions.send(campaign_id=campaign['id'])

A couple things to keep in mind here. There are two versions of the Mailchimp API, the Transactional (for single 1-1 emails) and the Marketing (for bulk emails, which is better suited for this purpose). The mailchimp3 package makes use of the Marketing API.

Handling Edge Cases and Duplicates

Shopify, by default, will send one or more notifications per webhook request. While this gives a level of consistency, it can lead to some conflicts.

To combat this and other edge-cases, we deciced to collect notifications in a Django model called ProductInStockNotification.

This solution was comprised of:

  1. Adding ProductInStockNotification models
  2. Adding notifications to the ProductInStockNotification model in our webhook endpoint
  3. Setting up a manage.py command to iterate through and send out the notifications
  4. Starting a cronjob to run the command every 5 minutes
# models.py
class ProductInStockNotification(models.Model):
    shopify_sku = models.CharField(max_length=255, blank=False, null=False)
    campaign_id = models.TextField(max_length=255, null=True, blank=True)
    product = models.ForeignKey(Product, null=True, on_delete=models.CASCADE)

class ProductInStockNotificationError(models.Model):
    product_notification = models.ForeignKey(ProductInStockNotification, related_name="errors")
    # ...
    message = models.TextField()
# views.py (the webhook endpoint)
def product_in_stock_notification(request):
    # The product ID
    sku = str((json.loads(request.body))['sku']).strip()

    # ...

    ProductInStockNotification.objects.create(shopify_sku=sku)
    
    # Returns 200 regardless of status for Shopify
    return HttpResponse(status=200)
# management/commands/send_in_stock_notifications.py
class Command(BaseCommand):
    def handle(self, *args, **options):
        while ProductInStockNotification.objects.filter(status__gte=ProductNotificationStatus.TO_PROCESS).exists():
            with transaction.atomic():
                notification = get_next()

                # ...
                # Handle notification states and errors
                # ...

                # Sends campaign based on the campaign_id in the notification
                send_campaign(notification)

                notification.save()
crontab -e
# 5 * * * * manage.py send_in_stock_notifications

Conclusion

With all of that, we have a notification system that can handle duplicates, errors, and even large numbers of notifications. Lessons learned?

  1. Shopify webhooks MUST return 200 if you received them at all.
  2. Shopify's REST API does not support all features of webhooks like their GraphQL API does.
  3. Shopify webhooks prefer sending multiple times, rather than never sending, so you must handle duplicates.
  4. Mailchimp is still pretty awesome.