Using Salesforce in an Event Driven Architecture
April 23, 2019
Remember the old days, when integrating with a third-party from Salesforce was invariably a job for the humble Apex Trigger?
Well, as it turns out, Salesforce have really made leaps and bounds when it comes to giving developers powerful tools to integrate with third-parties, or to capture and respond to events within Salesforce.
This is a quick survey of the current state of play with streaming and events, in Salesforce.
We will look at Push Topics, CDC, and Platform Events, and Generic Streaming.
A Quick Overview
The streaming event features page includes the following table. It’s a useful at-a-glance table for understanding the differences and similarities between the available options.
Most notably, no matter what you choose, you’ll be able to subscribe to events using CometD. This is because the underlying API is the same.
In many ways, this makes the consumer of your events relatively agnostic about what you choose - as long as they’re able to process the different payloads, they’ll be fine.*
This allows you to focus on the pros and cons for your specific Salesforce implementation, and the use-case you’re solving for.
*That said, we did notice that - as at the time of writing - the popular restforce gem for the Ruby language only supports Push Topics, but that shouldn’t be too hard to fix.
Push Topics are old news by now, of course, but they remain a very powerful tool for quickly creating a subscription feed of changes to data.
Just create a topic, give it a query, and subscribe.
Here is a basic example of a Push Topic that would notify changes to contact emails (inclusive of when the contact is created, in this case)
The flexibility on offer with Push Topics is largely controlled with the combination of the SOQL query, and the NotifyForFields field.
In our example, we used the ‘Reference’ option in the NotifyForFields field, which ensures that when the referenced fields change, a notification is published. For more information about how to use this field, check out the Push Topic Notifications documentation.
So for another example, if we wanted to transfer the state of an SObject as it changes, we could do something like the following:
Just be sure to include the fields you’re concerned with in the SOQL query.
You will want to read about the Limits that are in place for Push Topics. For most use-cases, the one to keep in mind is that you can only have 100 topics in play at a time. For most purposes, this means you shouldn’t dynamically create topics to serve individual users unless you know for sure you’ll be keeping the number low.
Please also consider the supported queries, and unsupported queries. Neither of these pages is exhaustive (for example, the User object is not supported. I’m tempted to speculate that this generalises to Setup objects more generally, but do not quote me), but they are a valuable resource in deciding whether PushTopics are the right tool for you.
CDC - Change Data Capture
CDC is the new kid on the block - You’ll want to read Salesforce’s blog about it.
It’s also now generally available!
I’m thinking about CDC as a more built-for-purpose solution to my latter PushTopic example above. If you want to use events to carry state changes between systems, that’s what this is for. It will require less setup with Salesforce itself, and won’t count as a push topic in your push topic limits.
However, it isn’t merely that simple. CDC, being built for purpose, makes some special affordances, most notably overflow events, and unified diffs for large text fields, which are both optimisations for dealing with large amounts of change at once.
If your primary goal is to sync data, and you’re not interested in more semantic types of events, CDC is looking like a great choice for you.
Just be sure to read the developer guide, and remember that it’s still shiny and new, so start with a prototype and see if it is meeting your needs.
Platform Events are very cool. Unlike Push Topics and CDC, Platform Events are not merely reactions to data changes. You can publish Platform Events from Apex, or from Process Builder and flows. You can define your own payload. You can use Apex triggers to programmatically respond to platform events as they happen. And your other systems can even publish platform events to Salesforce over the API.
Platform Events are a much more complete offering for the various types of events people use in Software. We can see them being used for notifications, event-carried state transfer, and possibly even mixing in some event sourcing on Salesforce.
For example, imagine if your Apex controllers merely created platform events with relevant data. An apex trigger could then take this data and call whatever code it needs to aggregate it into your standard and custom objects. An interesting thought!
Something to consider when using Platform Events for anything closely related to data-integrity is when to trigger them.
In the Summer 19 release, you’ll be able to choose whether to publish immediately, or only after a transaction completes. The latter option will be useful for maintaining data integrity in the case of failed transactions.
Another key point to note with platform events is their scalability. Salesforce provides two volume entitlements: standard-volume and high-volume.
The Standard volume will limit you to 50,000 delivered event notifications per day, while High volume has a limit of 1.5 million delivered event notifications per day.
So, is there a down-side?
Well, for what it’s worth, platform events might require a bit more forethought and effort on the Salesforce side to set up. They’re not just an apex script away from a subscription feed like Push Topics, and as we’ve seen, CDC is ready to go out of the box. With this in mind, definitely consider whether you really need Platform Events.
Finally, there is Generic Streaming. This option is viable when you need to declare an arbitrary payload, you do not feel you need the shiny bells and whistles provided by Platform Events, and importantly, you don’t intend to use Apex.
Generic streaming is essentially the lowest level approach that is available. It works by creating a StreamingChannel, which you will recognise from the Push Topics approach:
To broadcast events, make a POST request to Salesforce, replacing STREAMING_CHANNEL_ID with the ID of your StreamingChannel
If need be, you can constrain the broadcast using an array of userIds:
To wrap up this overview, I think it’s fair to say that Salesforce is providing some much sharper tools for integrations and events than was once the case, and the future looks better than ever.
As for how you use these tools, just be sure to read the documentation thoroughly, and consider implementing a prototype of your solution with each available option to get a feel for how they work.