Big Data ? I Object!
Object Based Storage and Big Data – are we confused?
Two things the IT industry and particularly the IT press tend to be guilty of.
1 – Hyping a particular technology or concept out of all proportion
2 – Creating terms that confuse rather than clarify
Big Data and the hype around Big Data is a prime example. The hype is huge and as a result the term itself gets misunderstood as it often gets hijacked by vendors trying re-define the term to fit their own technology. Big Data is proving to be somewhat like with a black whole other with concepts getting sucked into the hype.
Lets take a step back, the occupation of statistician is an old one, and as far as I can see it is not a profession I ever remember the press getting excited about. However today the “Data Scientist” or even “Chief Data Officer” is seeming to be one of the hottest professions you can get into, with press and analysts creating headlines about them on a daily basis. However, how many people can really tell you the difference between what a statistician does and what Data Scientists do?
One term that I have seen sucked into the “Big Data” is “Object Based Storage”. Even in conversations with experienced IT managers, I have heard comments showing that people misunderstand the term, they think Big Data HAS to be built on Object Based Storage.
So lets try to clear this up.
Object Based Storage is not new, but in the last couple of years its application especially in the cloud has accelerated. It dispenses with the need for a hierarchical way of storing data that is linked the confines of the disks on which it resides. Each object stored contains its own meta-data enabling it to be indexed and retrieved very fast, no matter how large the amount of data scales. The fact that it is not limited by the physical structure of the disk on which it resides means that you can scale it much more easily than block level or file level based data. These two points are probably why many people form the opinion that Object Based Storage and Big Data are linked or even the same.
Object Based Storage is great for the following:
Storing Static Data Dealing with whole objects (files) at once, it is not good at breaking down individual files
Retrieving data very quickly
Remote access to data using HTTP (and hence BYOD** access to data)
High Availability (most object based storage solutions automatically store multiple copies of the data you save)
Object Based Storage is Bad for the following:
Block level data
Fast changing data such as in a relational database
OS level data (Object Based Storage is the only type of storage that operates above the OS)
If you look at the above it is clear to see the following. Object Based Storage is perfect for storing huge amount of Static data. But is this what we mean by Big Data. Perhaps to some it is.
However for most, when we see the hype around Big Data. The headlines extol ideas like solving famine in Africa by better understanding crop growth performance or predicting crimes in advance by analysing crime statistics. Large archives of static data may sometimes have a part to play in this, but actually its relational database based data, often running at block level perhaps on Flash based arrays that are far more important technologies to support this definition of Big Data.
Over time and as the hype dies down the terms and definitions around Big Data will become more tightly defined, understanding around exactly what people mean when they speak about big data will become clearer. Big data and the technologies behind the concept, are here to stay even if the term itself dies out.
However, I personally believe that Object Based Storage is something that will become the more important term and concept. Its not just about Big Data, that's one aspect but Object Based data is much more than that. It’s about an efficient flexible, safe, mobile and accessible form of storing data that will become mainstream not just through cloud providers but within on premise IT as well.
**BYOD – another “term” – Bring Your Own Device – something I will tackle in another Blog!)