At Info-Tech we store all of our customer data in Salesforce, this includes things like subscription information which we use to determine what content on our website each client should have access to. This means that we need access to a lot of our Salesforce data from our website, which is written in Ruby on Rails.
Current Integration Method
Our current method of near real-time integration involves tracking changes of data that the website requires in Salesforce, and a Windows service which polls for changes, queries for a subset of the changed records and then sends them to the website via web service calls. It then goes back to Salesforce and marks any successfully integrated records as up-to-date, and leaves any failed records to try again. This has been working relatively well for us for the past year, but it has some issues.
- It requires us to maintain an extra Windows server for our production environment, plus an addition server to support the rest of the development/qa environments.
- It is one of the only applications we have that is written in .NET and there are only a few people in the department who have ever looked at the code.
- Since it polls for records, it uses API calls regardless of whether or not there is any data available.
- It is somewhat prone to “clogging”, since it only pulls a specified number of records at once, if enough records fail to integrate for any reason it will eventually get to a point where it is constantly trying to push these failing records and no other data will make its way through. This results in no data at all moving until someone unclogs the pipes.
Making It Better
Recently, we discovered some data issues on our website. What we realized was that when Contacts were merged in Salesforce this wasn’t being translated properly to the website. Instead of handling this data scenario within our existing integration system, we took this as an opportunity to come up with a new method for integration.
What we settled on was a method that used two features of Salesforce that we hadn’t used together before then. These features are Workflow and Outbound Messaging. Workflow is a configuration based way of triggering some action on Create, Update or Delete of a record, and Outbound Messaging allows you to make a SOAP based web service call as a part of a Workflow rule.
Outbound Messaging has several features which make this an appealing approach. It will queue messages together and send them in batches of up to 100. It will also ensure message delivery, if it doesn’t receive a success response, messages will retry at increasing intervals for up to 24 hours. There is one downside, which is the fact that it won’t make a pure REST based call, you have to send a SOAP message. We settled on a simplistic approach of using a lightweight XML parser to rip out the bits of the SOAP message that we care about.
The New Pattern
What we implemented for this Contact merge scenario was whenever a Contact merge is detected in Salesforce, an Outbound message is triggered via workflow containing the IDs of the two records being merged together. A Rails web service takes this message and pulls out all the pairs of Contact IDs, performs a merge of each pair, and returns either success or failure to Salesforce. This method was simple to implement, had no dependency on external services and was much easier to maintain then the .NET polling method.
Making It Betterer
Once we got this working, we went over it and identified some ways we could make it a little more efficient and robust. We abstracted the processing of the SOAP messages, so that we could process any SOAP outbound message from Salesforce and convert it to a hash. Now when our Rails web service receives the message it parses the SOAP, pulls out the record Id’s and places them in a queue for processing. Once all of the records have been queued for processing, we return a success response to Salesforce, so that the Outbound Message will be removed from the Queue.
From there, a worker process that monitors the integration queue pulls the Salesforce IDs and queries the Salesforce API for the contents of those records. We update the data in the web database, remove the ID from the queue, and the process is complete.
This new process seems to solve all the issues that we had with our old method of integration:
- No more external server, everything is either contained in Salesforce or in our Rails codebase.
- No more .NET, everything is either Salesforce configuration, or Ruby.
- Since Salesforce pushes notifications of data, we don’t have to use API calls to check for modifications.
- Even if there are records that can’t be integrated for any reason, they will never prevent other data from being replicated.
We now have a robust, relatively simple integration method that we can start to roll out for all of our Salesforce -> Rails integration needs.