Gartner defines Big Data as high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making. In Gartner’s hype cycle on emerging technologies (can you find Big Data in under 2 seconds?), Big Data has been moving up the peak of inflated expectations as vendor-marketers, media, consumers and analysts alike float their versions of Big Data.
Figure 1: Gartner 2013 hype cycle for emerging technologies
Data&StorageAsean caught up with Daniel Yuen, Research Director for business intelligence, analytics and performance management at Gartner and asked him for his assessment of Big Data. What transpires is an interesting look at Big Data and how companies in Asia are looking at it (minus “hopefully” from the influence of IT vendors).
Data&StorageAsean: What is Gartner's estimate of the Big Data opportunity in Asia from the perspective of businesses (not IT vendors)?
Daniel Yuen: In Asia, the interest in big data is continuously increasing. From our client discussions in this area, more and more companies are getting started in exploring the “how” to deploy big data projects. Many companies are in the process of developing resource requirements, like best practices and organization, for budgeting purposes.
“Information heavy” industries including manufacturing, telecom, online media providers, banking, financial services and government organizations will particularly benefit from implementing big data solutions. Analyzing big data leads to three categories of business opportunities – making better informed decision making by embracing a widened range of data sources, discovering insights from the ability to process larger data sets to automate of business processing that was not possible before.
Making better-informed decisions — Analysis of big data leads to better decision support. Decisions such as prices, promotions, staffing levels or investments — essentially any business decision — can be improved if big data sources are available for insight.
Discovering hidden insights — Big data analysis can also be used to discover opportunities that are obvious only by looking at large sets of detailed data. For example, telecommunications companies analyze billions of call detail records to find out which customers are the most connected (that is, make or receive the most calls from a wide variety of phone numbers). Promotions can be focused on these individuals to keep them as happy customers, since if they leave, they may "drag" a lot of friends with them to a new carrier. This detailed analysis was not cost-effectively possible before big data tools.
Automating business processes — New technology can also be used to use big data in real time. New technologies allow analysis to be "built into" processes so that automated decision making can occur — even with real-time, high-volume and/or unstructured data. For example, complex-event processing technology is the foundation behind algorithmic trading, which is used by financial securities traders to analyze markets and conduct thousands of trades per second. Used this way, big data and the associated technologies can automate a business process often not possible in the past.
DSA: What are drivers of Big Data among companies in Southeast Asia?
DY: Business pressure may be the most important driver. BI and analytics leaders need to respond to business requests for continuous performance enhancement – for example improve operating efficiency, lower production cost, improve market share – any kind of business improvement through technology. And the term “big data” is now already reaching every level in society around the globe, even TV drama is using “Big Data” as a plot.
Government policy also plays an important role in some countries. Like in Singapore, plans and outline to use information, analytics and big data to improve performance and drive innovation will encourage commercial sectors to adopt big data.
DSA: How is Hadoop involved in Big Data?
DY: Customers across the Asia Pacific region are keen to harness the innovative capabilities of Hadoop. Most big data discussions tend to be mired in the technical capabilities of Hadoop, rather than focusing on the business problem or use case at hand. The market seems to be buying big data solutions from two distinct types of providers: Those that provide infrastructure components, such as Hadoop and object-oriented database management systems; and those that provide business analytics capabilities. Because not many vendors in the region can offer both the technology and the business analytics capabilities, implementation issues are bound to arise.
Customers also view Hadoop as an incremental platform for harnessing new capabilities rather than a replacement for their existing relational DBMS or data warehouses. Hence, integration and coexistence with existing information management infrastructure is paramount to them. Hadoop's popular usage is as a low-cost storage tier for unstructured data that can then be cleansed, transformed and loaded into a data warehouse for advanced analytics. The dearth of technology skills, lack of an established presence from the primary distribution vendors, lack of well-publicized use cases and overall ecosystem immaturity are slowing down-market adoption of Hadoop. Based on discussions with clients, we see the following use cases emerging:
Social media analytics is becoming a large part of customer engagement in Asia/Pacific in industries (such as telecommunications) that have a large volume of customer data and in industries (such as automotive and insurance) that have traditionally serviced customers through dealers and intermediaries and are now looking as social media as a direct engagement channel.
Operational analytics, such as log analysis, risk modeling and fraud detection, customer churn reduction, and clickstream analytics. For example, supply chain use cases include location tracking through route and time, which can be combined with business process tracking.
Life sciences companies generate enormous volumes of data in clinical trials, genomic research and analysis of environmental factors on health conditions. Some of these use cases cannot be handled by existing information management investments and as such are classic "sweet spots" for Hadoop storage and processing.
Most open governments, such as in Australia, promote the publication of great quantities of data in raw format in an effort to use data in meaningful ways for its citizens. Many are also looking at extending the concept of "open data" beyond the traditional boundaries of information technology to encompass operational data, such as that provided by a variety of sensors from transportation, water and electricity networks, which are often under the purview of (local) governments or local utilities. This, along with smart city initiatives being promoted by several governments Hadoop; will be an important part of governments' information management strategy.
Data&StorageAsean recently cooperated with MDEC in Malaysia during the Big Data week. We spoke to a number of local companies looking at their experience working with customers on different aspects of what may – if taken as a whole – represent varying stages of Big Data implementations. Download the Guide to Big Data and read for yourself how Malaysia is taking up (or not) the Big Data hype - warts and all.