The big data explosion may be offering organisations new insights and a wealth of business advantages, but how much control do we have over the storage and handling of this data?
If the answer for your business is not ‘absolute control’, then you are entering a minefield of laws, regulations and possible fines over the mishandling of data, as compliance issues continue to grow.
With this in mind, we spoke to Deidre Mahon, VP of big data software firm Rainstor, who told us about the data storage challenges facing organisations right now, and what compliance managers can do to steer clear of hefty fines.
TechRadar Pro: With respect to the storage of data, which compliance issues are most pertinent right now?
Deirdre Mahon: Rules are particularly stringent for financial services at the moment, particularly investment banks, who face SEC 17a-4(f) and CFTC (commodity futures trade commission) which mandate that transaction and trade data be held for specific time frames in an immutable store. In other words tamper-proof and providing the highest level of protection against malicious damage or breach.
Other regulations for banking include PCI, which is for the protection of sensitive personally identifiable information (PII) following a twelve-step data-related policy.
Additional compliance coverage includes Dodd Frank reform in financial services and Sarbanes Oxley for public corporations, which is mostly related to the retention and storage of structured and semi-structured data.
In other sectors, you have lawful intercept and mass intercept rulings for the retention of communications data from all mobile networks. This, of course, varies by country and region.
TRP: What are some of the potential problem areas that organisations should be aware of when determining their storage and compliance needs?
DM: Most compliance mandates specify how the data should be stored, verified, accessed and disposed of, and unfortunately, most traditional database systems don’t address these requirements out-of-the-box.
Often, when databases and data warehouses reach capacity limits, the data is offloaded to tape which may be WORM compatible and initially lower cost, but it doesn’t satisfy SEC 17a-4 mandates and you simply cannot query it.
Manual intervention and days-to-weeks of effort ensue to bring tape data back to life, which is error-prone and highly risky. By not meeting these compliance mandates, banks face heavy fines and, of course, bad PR which can result in lost revenue and credibility.
IT teams need to work closely with the data governance and compliance officers to determine which data sets need to be retained and for how long.
By institutionalising policies which help you avoid fines and proactively monitor for potential breaches, you lay the foundation and the rest is really about which technology platform you wish to deploy.
The technology solution is by no means trivial, but there are proven and battle-tested solutions that solve this problem and really prevent you from having to build it from scratch.
Built-in security, data disposition, legal hold, and audit trails are all critical elements of what is required in order to stay compliant and within the law.
TRP: Where do you see data storage compliance heading in the next five to ten years?
DM: Standard & Poor’s recently conducted research among financial services chief compliance and risk officers showed who they now collectively spend $50 Billion on compliance mandates alone.
These costs are expected to rise over the coming years. As data continues to grow from a wider variety of sources, proliferates and is shared across global networks, the level of stringency and controls is bound to increase.
Concerns of privacy, identity theft and credit card fraud are everyday occurrences, but once you start to consider other data-related threats around healthcare insurance, pornography, and other criminal activities, you can very quickly see how there must be standards and policies adhered to in order to protect the innocent.
Storing all of this data is a daunting concept. If data growth rates double each year reaching 40 zettabytes by 2020 in the enterprise alone, something different has to be done. Throwing more storage and hardware at the problem does not solve it.
What is really required is an efficient way to keep this data for the time frames required, and data compression technologies have matured enabling you to reduce overall storage footprints by 20 to 40 times.
That translates to keeping that data for 20-40 times longer than before. With this type of compression, there is no re-inflation required for query and performance speeds up. With extreme data growth rates, it is the smartest thing to do.
TRP: What other challenges, issues and trends may be involved in storing data in the future?
DM: Interestingly, big data, Hadoop and analytics have become almost synonymous, which is actually misleading not only for vendors but, most importantly, for buyers.
Organisations need to figure out the business goals and value they are trying to accomplish which will be unique to each organisation.
Some data needs to be stored long-term because the business users deem it valuable and some data has to be kept for various compliance mandates.
Figuring that out is half the battle. In addition to the basic storage question and overall costs, there are a host of other requirements which span aspects such as data latency, performance SLA’s and overall security and availability.
It is important to specify exact requirements at the outset so that the technology solutions selected check all those boxes.
Hadoop is a great platform for low-cost scale and managing multi-structured data sets, but it can also be misleading, and you can very quickly run into operational costs far exceeding what you expected at the outset.
Managing petabytes is a very different data centre challenge. Also, training resources new skills is a costly and time-consuming undertaking. You need to leverage what you have today and use standard tools with SQL being central to that when it comes to managing big data now and in the future.
TRP: Alternatively, how can organisations benefit from these trends? Are many taking advantage?
DM: According to Gartner research analyst Merv Adrian, only 8% of enterprises surveyed are in production on Hadoop today with a further 50% going live in the coming 2 years.
There are many business benefits to be gained by collecting and analysing new data sets, but it’s also important to show incremental value.
Gone are the days where enterprises have the patience to wait 2 years before time-to-value is reached and multi-million dollar investments are committed. Rolling out Hadoop clusters for a very specific use case is the ideal and recommended approach.
TRP: What’s next in big data? Are there any exciting developments we can expect in the near future?
DM: Big data is not going away or abating. We will continue to see more innovation in this area of data management. Hadoop adoption will grow, but it will require more maturity to meet the high expectations which mainstream enterprises have come to expect.
Tolerance levels for security, data governance and high availability will start to dissipate and these capabilities will need to be delivered faster and proven-out.
Real-time query and analytics is already on the rise and there are some very exciting developments in that area around in-memory database solutions, but on the other end of the spectrum organisations will start to be much smarter around "right-tiering".
Not all data is created equal and of the same value. Storing current data in the most expensive systems is a very different approach compared to storing multi-years of corporate memory and petabytes in lower cost storage or cloud environments. Organisations will be much smarter in how they approach that and solve those challenges.
See the article here: