We often require that applications talk to each other seamlessly.
It sounds easy but is a daunting task to achieve a satisfactory solution.
Problems arise, and these cause exceptions Maybe data is badly formed, has incorrect values… the list is endless.
We understand the pitfalls that need to be looked at to ensure a successful delivery between systems.
Sequence Software specialises in solutions for systems integration.
We understand the importance of the sequence of events and stages data must pass in order to obtain the final result
In an ever changing world there are still requirements for multiple packages of software are often adding some information along the chain.
and the market does not stop. SOAP, SAAS and the cloud only add to the complexities of knowing where your data is and where it is coming from.
We can help you to find the best solution at a budget to suit your needs.
All systems rely on a source of data, either being the point of entry from one system or intermediary data being sourced from another.
This can vary from a spreadsheet file, to manually entered data into a screen, to more complex binary data.
The next key element is how data is received from different systems.
This has become more and more complex, for two reasons. Firstly, the number of delivery mechanisms continues to increase, in particular with more and more use of the internet for delivery. FTP,(s), sockets, queuing, and database mechanisms gives just a high level list.
Secondly the integrity of the data is important. Handshaking, acknowledgement, and timely delivery is vital, especially when considering system availability across multiple systems.
Once any data has been obtained, it needs to be parsed. This is the mechanism for understanding what the data stands for.
For example a CSV file has different fields and these need to be mapped to a meaningful piece of data.
Complex formats need to be parsed into a known data format. For example a date may be represented in many ways in a text format.
Each piece of data must correspond to the required type – for example a number, a date, some text with a maximum length.
Once this is passed, there is a second level of validation between fields. A date may need to greater than another date – for example a trade date and a value date, or a date may have to be today or later.
In the case of errors, how is the data to be processed? Is it to be rejected, returned to the sender ....
It is often the case that we find information from different sources and this is further complicated as different sources may use different codes.
For example you may have two sources providing a currency: One uses the identifier “USD” the other uses “US Dollars”.
Normalisation makes sure that once we get the data into our system it will be known by one common value.
Enrichment on the other hand, could be used to provide additional information. One system may provide a customer code, but internally we need to provide the name and address. The enrichment process will allow the transformation from one thing to another.
Calculation is also another form of enrichment. One system provides price and quantity but the next requires the total too.
Enrichment is done by what is commonly known as “static data”. This differentiates it from transaction data, which changes in every transaction, to data which rarely changes. A customer may make regular transactions but it is rare that their name and address will change.
We will look at static data and its problems later…
What is supplied by one system will be different to what is required by another.
The transformation stage allows all of the data elements to be extracted for the requirements for another system, or as is often the case, systems.
For example the source may have been a customer order. This may have three outputs, the billing system, the stock requirements, and the delivery system. All require different information in their own rights. The transformation stage will ensure that all of the data for that system is available
Then we have formatting...
Formatting the data for the destination system.
In a similar fashion as the parsing some of the more complex elements may require being sent in a format that is recognised by the destination. A typical example, once again, is a date. One system may require 2012/10/19, another 19-OCT-12.
There are a variety of common formats, such as fixed width columns, CSV, XML etc.
Finally we arrive at sending the formatted data. Again there are a whole range of mechanisms, and these are equally as complex as receiving the data.
The key issues are the transport mechanism, security, and reliability.
Often system require “handshaking” to check that the data has arrived, has been accepted, and the system is available. Once more, in a similar fashion to the validation stage, we need to identify what we must do if problems occur.
Retrying data, sending data later if unavailable, notification of issues – these are vital to the transmission stage.