Data integration: Getting different BI systems to talk to each other

By Jim Mortleman

Joss Marsh is a managing consultant for Sovereign Business Integration (Gulf), who has been working on systems integration projects in the region for over two years, with clients including a major bank in Qatar. He says: "Organisations here tend to have grown very quickly. As a result systems have been put in for speed of implementation rather than as part of an integrated strategy.

'That has given rise to issues of data quality and integrity. For example, banks generally have a number of independent systems loosely coupled to core banking and there are a lot of inconsistencies in the data among them."

Quocirca principal analyst Clive Longbottom similarly says the issue isn't getting systems to talk to one another, it's ensuring the data they share is consistent. "The vast majority of tools today will talk to any database you have - be it Oracle, IBM, Microsoft, legacy mainframe databases, spreadsheets or whatever. Connectivity is no longer an issue.

'Now the problem most are up against is data quality. For example, a retail organisation might have me on lots of different databases, but on one I might be 'Clive Longbottom', on others 'C.S. Longbottom', 'Clive Longbotham' and so on. So the first thing they need to do is cleanse their data," he says.

Once you've done the initial data clean-up, he says master data management (MDM) tools can ensure new information entering the systems doesn't suffer from the old-style inconsistencies. MDM guides people as they enter data, preventing them from doubling up or mis-entering.

"It ensures companies only ever have one accurate master record that purtains to 'Clive Longbottom', from which they can access all the other data held about my buying habits, credit or whatever else," he says.

Integrating business information

But clean, consistent data involves more than just getting rid of duplicate records and ensuring details are correct. When you start integrating information sources from different departments, you also need agreed company-wide measurements and standards.

Richard Kellett, head of customer marketing and operations at SAS, says: "For example, what does 'turnover' mean? Generally, finance, sales and other departments all have different answers. Getting agreement is hard work because everyone wants to measure things on their terms. But only then can you think about moving on to the next level of predictive modelling, forecasting and statistical analysis - which is way beyond where most organisations are today."

In terms of sourcing BI systems, on the face of it there are a number of options. As long as a company has reached a sufficient level of maturity, it's feasible to opt either for an integrated suite of tools from one of the well-known suppliers or select smaller best-of-breed suppliers for different areas and link them together.

However, Oracle's Jon Ainsworth says: "Its better to go for a pre-built, pre-configured solution that takes data from standard operation systems like ERP and gives you best practice out of the box and pre-built reports. That's a way of de-risking and accelerating the project."

There are even open source integrated BI suites that Longbottom says are "well worth a look", although he concedes they require a higher level of in-house technical skills in terms of implementation and ongoing supports.

But lack or cost of skills can impede implementation of commercial products too. Sovereign's Marsh says: "A Business Objects implementation was taken off a recent project I was involved with, for example, because of the cost of resourcing it. It can be difficult to get the right level of skills to drive these projects in the way required - ie, with ongoing development, very often using a small, dedicated team."

But does the future offer any hope for those put off by the expense of integrated BI? Suppliers hope increasingly to offer BI as a service, often hosted in the cloud, which should make it both more affordable and manageable. And Quocirca's Longbottom believes prices need to fall significantly.

"This shouldn't be costing firms $1000-$1500 a year per person, but more in the region of $100, particularly given the sophistication of the BI tools now available online and within standard packages, essentially free of charge."

However, while the idea of BI as a Service sounds attractive, the whole Software as a Service market is struggling to get off the ground in the Middle East. With little SaaS takeup, it is unlikely that BI as a Service will buck that trend.

'We've not seen it being taken up,' said Ranjit Rajan, Senior Research Manager at analysts IDC. 'We haven't seen any real demand in SaaS in general and BI as a Service in particular [in the Middle East].'