After working with dozens of customers to integrate hundreds of systems, we want to share some data integration best practices for connecting business systems together.
Focus on the business value before the technical details
In many cases, we see that the primary driver of an integration concept is a person in the IT department. Naturally more technical people gravitate towards thinking about the mechanics and engineering for a solution and the business value can become lost in the details. Always identify what the true impact on the business is going to be first and let that drive the technical details.
Identify project stakeholders and who will drive each portion of the project.
The most successful projects we have seen are the result of having the right leadership driving the project. It is important to identify which areas of the business will be affected by the integration and that a spokesperson is identified from that department that is the most knowledgeable about their area.
For example, if an eCommerce integration will bring in customers from the eCommerce site to an ERP system, then it would be critical that someone from Sales, Customer Service, and IT are all involved in the project. It is also important to have an overall project manager to orchestrate the interactions with other stakeholders and ensure tasks are done on time.
Identify systems of records for each category of data
One of the most important benefits of integrating systems is the elimination of duplicate data entry. To reduce the most duplication, ideally, a single system of record needs to be identified for each unique data source such as Customers, Products, Opportunities, etc.
That doesn’t mean one place the data stored, but one place that data is first created, and then other systems feed from that system. In every business, there is almost always a system that is the first to hold a specific type of data, and leveraging that natural process is critical to getting the most value from your integrations.
Profile data by manually walking through the process to ensure all requirements are identified.
I believe that the most common reason for scope creep and project overruns is that the details of the integration did not end up matching what everyone thought at the start of the project. It is important to profile real data and to not base the entire project on gut feeling or what people think.
Profiling forces you to do a walk through of integration using real data so that the less visible details like field lengths, mappings, naming conventions all come to the surface and can be accounted for in the project plan.
Nail down technical details of how the integration will work and changes to the infrastructure needed to support the integration.
Integration best practices 1-4 are in many ways the minimum pre-requisites needed to be able to nail down the technical details. Once you know for sure which systems will integrate, which direction data needs to flow, and have an understanding of the translations that will be needed….THEN you can nail down exactly how you are going to accomplish the integration from a technical aspect.
Are they going to be database level, web service level, or application level integrations? What programming language will the integration be done in or will you be using a platform like Doc Infusion? What are the security risks and how to mitigate them? Will we need new infrastructure on-premise, or in the cloud? All of these questions must be answered.
Clean data prior to integration
Another major reason for project delays is data quality. At a high level, a project can be oversimplified by assuming that field 1 in System A maps to field 1 in System B. That may be true but what if there is data in System B that does not exist in A? What if field1 in System A is 5 characters long and in System B it’s 10? What if System A has the same record duplicated multiple times, which one is the right record?
Addressing these issues prior to starting the integration is important to keep your project on time and on budget. The worst time to handle them is in the middle of implementing the integration as this will create scope creep and increase the cost of the project.
Budget appropriate time for multiple profiling/testing sessions
It never fails that users want something to work without testing it, but that’s not how it works. Testing is a critical piece of the entire implementation plan; the plan cannot be executed effectively without proper testing. Make sure that data is profiled multiple times, not just once before the project. Make sure that additional tests are done using data that has come through the program automating the integration.
Make sure the users who test are the most knowledgeable in their areas
Not all testing is equal. There is almost no value in an unknowledgeable user reviewing data and doing simple checks like seeing if customer 123 made it from System A to System B. The most important details are the less obvious but crucially important ones that only someone from specific departments would look for.
For example, an IT person would probably not think to check credit limits, terms, bank codes, and other business-critical information so it is important that the user responsible for signing off on data quality is someone who uses that information on a daily basis.
Make sure your integration solution is flexible enough to handle all business requirements.
Selecting the wrong data integration methodology or platform can cause an entire project to have to be scrapped and start over. We wrote an article on how doing partial integration can be worse than no integration at all here.
Separate development, QA, and production environments
Proper promotion of code from development to production is very important to get a project right the first time. However, many companies sacrifice this concept assuming that if the code is deployed directly into production that there will be time saved. Time after time the perceived savings are shown to be false when major mistakes in the integration occur and details that were not seen in advance begin to surface.
It’s important to do development in one place, and to push the code to QA where stakeholders and department personnel can review the data after developer testing. Only once all users and the project manager sign off should the code be promoted to live.