Before Big Data became fashionable there was business intelligence and before business intelligence became an everyday phrase among business users, there was data warehouse. Therefore to truly understand Big Data and BI, one must go back to where it all began. Here’s a recap:
One of the benefits of relational databases was the ability to manipulate data to perform basic analysis and reporting. However, these required the assistance of IT to produce reports needed for analysis. Excel spreadsheets and Access databases replicated this capability for the everyday user. But both solutions did not offer the capability to perform ‘what if’ scenarios needed for predicting future outcomes. Data warehousing was created for this purpose but its complexity proved to be its biggest limitation as it required expertise beyond the business users meant to benefit from it. Business Intelligence systems were created to bridge the gap between the Excel and Access duopoly on the one hand and data warehouse solutions on the other. But even as user interfaces have been dumbed down to make it easier for you and me to begin to use the systems, ecommerce created the need to capture, store and manipulate large amounts of data and data types. Cloud computing and self-service accelerated this data explosion. Welcome to big data.
IDC expects the Big Data market in Malaysia to reach US$24.2 million in 2014 and sys that for big data to move into the next stage of maturity in the country (repeatable stage), organizations need to adopt a strategic approach to Big Data adoption as opposed to being siloed as is currently commonly seen. Liew Siew Choon, Market Analyst of Software Research at IDC Malaysia says “The Big Data market will grow bigger and enter into the next stage only when the issue of skillsets and resources are resolved. This in turn will lead to a complete ecosystem whereby channel partners with the right skills will have the capability to deliver end-to-end Big Data solutions, including consultancy and other services.”
data&storageasean (DSA) spoke with Thomas How, managing director of Speedminer, on the state of business intelligence, business analytics, data warehousing and big data in Malaysia.
DSA: Do you agree that there is a persistent misconception that BI and BA are the same and that data warehouse is just an extension of this?
THOMAS HOW: In my discussions with executives and IT professionals from different companies, it is interesting to observe that people have different understanding of what business intelligence, business analytics and data warehouse are. For us at Speedminer, data warehouse is a way to combine data from different sources into a single repository so you can perform some business intelligence and report on it. Business Analytics, for us, is about being able to perform predictive analysis. In a way BA is like data mining - you are trying to make sense of the data, potentially predict an outcome, and so be able to make better business decisions.
Today, people think that by having BI tools, businesses can perform predictive analysis.
DSA: What is the problem with many of today's data warehouse applications, including BI and BA? Where does big data sit in all this?
THOMAS HOW: Most companies collect a lot of data about their operations. This could be from the production floor, from the warehouse, from sales and finance. Sure they can do very quick search, identify areas perhaps in the production process or billing or inventory. They are able to identify peaks and troughs in sales and factory outputs. But this is just one piece of the puzzle.
The reality is that a business relies on partners like suppliers who are outside the system. Customers are usually external to them as well. So to get a whole picture requires collecting data from outside and mixing with the internal data to have a more balance picture of the business overall.
For example, if the Tourism Malaysia wants to understand what campaigns are well received among the thousands of visitors to the country, the might want to look at comments posted on its Facebook page or other popular community forums like Tripadvisor. They need to go to the internet, collect the data, and perhaps perform sentiment analysis, which they cannot do by relying solely on their own data.
It is possible that the enormity of the data and the variety of sources may mean that their current IT infrastructure cannot support the analysis. So they, again, may need to turn to an external party to help them undertake the analysis.
DSA: Comparing your customers in Malaysia versus those in more developed countries, where do you see are the differences in terms of (1) technology adoption; (2) business strategy; and (3) intent to use cutting-edge technologies to further the business?
THOMAS HOW: In Malaysia, only the government has the resources to make the necessary investments in data warehousing. Most private sector organizations sadly do not spend as much. And this may be partly due to old practices and beliefs that continue to this day. Let me explain. In many developed markets, companies recognize that to maximize the use of technology, you need to invest in professional services rather than just hardware and software.
In Malaysia, the prevailing practice is that companies are happy to invest in the tangible assets of hardware and software but not on the services. Sometimes during evaluations of IT solutions, there is lack of clarity on what is needed to get the best ROI. As such, many are unable to realize the full potential of their investments.
DSA: Today, where is your core development team located? Do you find talent acquisition and retention an issue?
THOMAS HOW: Speedminer is a wholly owned Malaysian company. That said, we have nationals from China and India, for example, because finding local talent remains a big challenge for local ICT firms such as us.
As a MSC, we may have nationals from India and China but we are all based in Malaysia. Currently one of our bigger operational challenge is finding local talent so we bring in the skills we need from the region but the reality is that the number of people we want to hire is not enough.
DSA: As cloud computing take roots, are you seeing demand among customers shifting away from on premise to cloud base?
THOMAS HOW: The Malaysian government is probably the single biggest big data customer at this time. They have the resources, including a cloud datacenter in Putrajaya, to make start the big data revolution in Malaysia. Quite a lot of government departments are moving their servers to the datacenter.
Within the private sector, infrastructure cost is a key inhibitor towards building systems in-house. Cloud computing may be just what the private sector needs – unlimited infrastructure on a pay as you go model.
DSA: In Malaysia, what needs to be done to accelerate adoption of big data solutions? Who should spearhead this adoption?
THOMAS HOW: I see customers in Malaysia that want to do analytics. The lack of in-house expertise is a problem. When we do analytics, it is not the normal BI people to do this. Sometimes they need statisticians to take full use of the system especially in data mining. This expertise is what is lacking in the private sector. I believe that the impetus is on Government to spearhead the development on local experts to help accelerate adoption.
DSA: What should the Government do to accelerate adoption of big data in Malaysia?
THOMAS HOW: Local skills are clearly the missing ingredient. The Government could consider partnering with educational institutions to encourage students to take up the computing courses geared towards business intelligence and big data. I believe that when these graduates out to join the labor force, they will tell their employers the types of tools needed to get the job done.
DSA: Tell us a little bit about speedminer.
THOMAS HOW: We started in 1997 with about 10 people. Currently we have about 40-plus technical staff mainly around development. We originally focused on the healthcare industry; back in 1997 business intelligence and data warehouse were not common in Malaysia.
At the time the solutions were not mature; there were no significant products in the market. We saw the need and the opportunity; we were fortunate to have customers willing to explore the future with us. Armed with our experience and success in healthcare, we slowly move to other industries with solutions specific to the industry requirements.
From business intelligence, we have branched to Big Data. Many of our customers have huge amount of data. They recognize that information is valuable. Together with our customers, we are progressing towards big data. We are building a development platform in support of that strategic direction. We firmly believe in that future.
DSA: Any last thoughts to share with our readers?
THOMAS HOW: Today there is a disconnect between senior management (Board) and operations. Top management recognise the potential of the technology. Middle management and IT however see the operational challenges of in-house skills and processes that are not aligned to what the technology has to offer.
We believe that to truly reap the benefits of any technology, the organization needs to identify the problem, determine what they want to achieve, and from there to look for the solution and expertise to harness the potential.