Emerging markets producing more data, only a fraction analysed

Only a small fraction of the world's 'big data' is being made useful, according to an IDC Digital Universe study. Titled "Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East", the report predicts the digital universe will reach 40 zettabytes (ZB) by the year 2020, exceeding previous forecasts by 14% - and a 50-fold growth from 2010.

Around 2.8 ZB of data will be created in by the end of 2012 alone, largely generated by machines rather than individual users. Emerging markets are seen to be driving data growth and will count as the primary producers by 2020.

Investment in IT solutions, telecoms and staff, or the 'infrastructure' of the digital universe could soar by 40% over the next eight years, with investment focused on storage management, security, big data and cloud computing.

The explosion of smartphones, tablets and other portable computers has boosted web access for all markets and the increase in data from other technical equipment, such as surveillance cameras, have led to the doubling of the digital universe over the last two years.

"As the volume and complexity of data barraging businesses from all angles increases, IT organisations have a choice," says Jeremy Burton, Executive Vice President, Product Operations and Marketing, EMC Corporation.

"They can either succumb to information-overload paralysis, or they can take steps to harness the tremendous potential teeming within all of those data streams. This year's study underscores the massive opportunity that exists for businesses that not only identify the potential benefits of the digital universe, but recognise the importance of navigating that universe with the right balance of technology, data security practices and IT skills," he continued.

Useful big data lost

The study, sponsored by EMC, finds that 23% (643 exabytes) of the current digital universe could be useful as big data, if it were tagged and analysed, which only 3% of the potentially useful data currently is. However, the amount of useful data is expanding with the growth of the digital universe. By 2020, 33% of the digital universe (13,000+ exabytes) could have 'big data value' if processed properly.

To try and understand the volume of this exponential surge in data, there are 700,500,000,000,000,000,000 grains of sand on all the beaches on earth (seven quintillion five quadrillion). That means 40 ZB is equal to 57 times the amount of all the grains of sand on all the beaches on earth.

To store all 40 ZB onto Blue-ray discs, the weight of the discs alone, excluding sleeves and cases, would be the same as 424 Nimitz-class aircraft carriers.

This year's study marks the first time IDC was able to capture where the information in the digital universe either originated or was first captured or consumed, revealing some dramatic shifts currently underway. The current global breakdown of the digital universe is: U.S. - 32%, Western Europe - 19%, China - 13%, India - 4%, rest of the world - 32%. By 2020, China alone is expected to generate 22% of the world's data.