According to Gartner, the future of the database market lies in the cloud, as the global database management system (DBMS) revenue grew 18.4% to $46 billion back in 2018. The analyst firm also predicts that 75% of all databases will be deployed or migrated to a cloud platform by 2022.
To understand what’s driving this significant growth and why organisations are quickly turning to the cloud, DSA interviewed Feifei Li, VP of Alibaba Group, President and Senior Fellow of Database, Alibaba Cloud Intelligence, when he was in town as one of the invited speakers of the Alibaba Cloud Summit in Kuala Lumpur.
Without a doubt, Li is a certified expert on the topic, being an ACM Distinguished Scientist, Director of the Database Products Business Unit of Alibaba Cloud and the Database and Storage Lab of DAMO academy. Before joining Alibaba, he was a tenured full professor at the School of Computing, University of Utah. Li has also won multiple awards from NSF, ACM, IEEE, Visa, Google, HP, and Huawei.
He started off by explaining that traditional database systems typically use single node architectures where computational resources (CPU and memory) are tightly coupled with storage to bring ease of development and deployment. Unfortunately, Li stated that there are several key limitations to this, especially when it comes to scalability and elasticity.
When a business wants to scale out its storage, for example, it has to purchase more single nodes, which also includes additional computation resources that it may not necessarily need. Moreover, deploying those new nodes can end up being very costly for the organisation, both in terms of time it takes and the manpower it requires.
Li explained that the cloud native database systems concept, which was introduced sometime in 2016 or 2017, has enabled companies to overcome such limitations by “decoupling storage and compute so that they can scale out independently from each other.” By leveraging a technology called distributed shared storage, Li said businesses that use cloud native databases such as PolarDB are able to scale out their database systems quickly and easily, without much intervention from the user end.
Another critical feature of cloud native database that Li mentioned is related to the use of serverless technologies. Li said, “In a traditional database setup, you have to deploy a fixed amount of resources ahead of time, and each node is featured with a given set of resources. Serverless means that you don’t have to purchase or deploy resources ahead of time, rather, you ask for resources on demand.”
Therefore, he said with cloud native databases, organisations now have the option of just upgrading what they need, be it storage or compute resources, on demand, and they can scale out these resources within minutes. “As your concurrency goes high, you scale out more nodes, but when your concurrency goes down, you can scale back down. This is very important for businesses that require elasticity.”
He continued with an example with regards to e-commerce. “Let’s take an e-commerce application as example. Let’s say you have a sales event going on today, and because of that, your application workload goes to the peak for today only. Using cloud native technology, you can allocate resources on demand for today. When it finishes, you can quickly release those resources. So, this cuts down the waste of resources, which, in turn, will reduce your cost of ownership.”
Overcoming Connectivity Issues When Migrating to Cloud
We asked Li whether connectivity was an issue for businesses in the ASEAN region, where the network connection in many areas, away from the major cities, are spotty at best. Li mentioned that this was not a unique challenge for ASEAN countries, but one that is also facing businesses in remote parts of China.
The fact is that “there’s no silver bullet in addressing this challenge” said Li. To help businesses, Alibaba has been working on upgrading their infrastructure and setting up local data centres at various locations, such as in Kuala Lumpur, Jakarta, Singapore, Hong Kong and Taiwan, in order to bring their infrastructure closer to where the clients are.
However, there’s still a problem for larger countries like Indonesia or Malaysia. “We have a data centre in KL, but how about clients in another city that’s far away from KL? When the network connection isn’t good, either we have a stability challenge, where the network isn’t so stable, or a speed challenge, where the connection may be stable, but is very slow. How do you cope with those issues?”.
Alibaba Cloud’s first response, Li said, was to choose the locations of the local data centres carefully so that they form a triangulation coverage of the areas. This is to enable Alibaba Cloud to direct customers’ traffic to one of the data centres that is not so far away from where they are.
“On top of that, we have built networking technologies to ensure that, for example, we set up a secure private network like VPC to ensure that the network connection is not only secure, but also stable, at least,” Li explained.
From a database point of view, Li said that the engineers at Alibaba Cloud have also designed a technology called the database gateway – essentially a client that customers can install in their on-premises data centre. “So whenever they’re ready to move to our cloud, the database gateway will ensure that a stable, reliable and secure network connection is set up, bypassing the local firewall or whatever proxy you have,” said Li.
In other words, the database gateway acts as a proxy between the client’s data centre and Alibaba Cloud, ensuring that the connection is stable for a smooth migration process.
In terms of applications, he added that once the database is migrated to the cloud, businesses would typically also migrate their applications with it. Li explained, “For example, if you run your application on our virtual machine service, ECS, we have smart proxy protocols to ensure that the client application running on our ECS is geo-colocated with where the back end database is running.”
Li said those are some of the things that Alibaba Cloud has done to ensure minimal latency issues and that the network connection is not a concern when organisations move to cloud.
Most Common Cloud Misconceptions
In terms of moving from a traditional database solution to a cloud native one, Li stated that there still exist several major concerns, or “misconceptions”, about cloud databases, especially around security and high availability.
Organisations often feel that having a database that’s installed and running in their own data centre may be better than if the data is somewhere they can’t physically see or access. While he understands the notion, Li suggests that moving the database to the cloud is oftentimes more secure than running it in your own local data centre, as counterintuitive as it sounds.
“If you run your database in a local data centre, then you have to maintain a large and responsive security team to make sure you have the latest patches are applied to the database kernel,” he said, adding that you also need to have a proper firewall setting that blocks adversary access to the network, and a secure proxy running on top of it that safeguards against SQL injection and access control, to prevent any potential attacks from the application layer.
For a typical company, where database is not their major strength, he said this could be a serious challenge and a difficult one to handle.
Speaking from Alibaba Cloud’s perspective, Li said that the cloud provider has professional security teams at their disposal to take care of these security challenges for customers. Moreover, when it comes to database security in particular, “we have addressed these challenges for so many different clients, so the adversary cases we have seen are definitely way more than what a single client have experienced.”
Thus, Li’s philosophy behind this seemingly “counter-intuitive” argument is that for major cloud vendors like Alibaba Cloud, their collective intelligence will always be stronger and better than what a single client can achieve.
Similarly, having access to a physical data centre where the machines or servers are running cannot guarantee better reliability or availability. When it comes to addressing the high availability issue, for example, from a software system aspect, Alibaba Cloud can equip their customers with the latest high availability technologies, as well as a team of engineers that are well trained in addressing such failures and emergency situations.
Li added, “We also use a lot of machine learning and artificial intelligence technologies so that we can monitor all the different database instances running on our platform, 24/7. So if anything goes wrong, we’ll be alerted right away and typically, our response times are even quicker than a lot of our clients.”
Why Businesses Are Moving Away From Traditional Databases
Getting back to the meat of why people are seemingly moving away from the traditional commercial model where database solution licenses are sold to clients, Li mentioned several factors that are contributing to this. Among them, many of the benefits of cloud native database systems Li has already mentioned, such as better scalability, elasticity, security and high availability.
He explained that those can be that can be achieved on-prem, but it will cost a lot of time, manpower and money, whereas in the cloud, everything is on demand and very flexible.
More importantly, he said, “If you look across the board, for most cloud database providers, their database systems are 100% compatible with open source database systems such as MySQL and PostgreSQL. This brings up two major benefits; One is you have a very healthy and large ecosystem, and you are not locked into a closed system.”
What that means is that organisations that are using such open source cloud native database systems can migrate away very easily to MySQL or PostgreSQL or to another cloud vendor of their choice.
When competing with the traditional database vendors, cloud native database players like Alibaba cloud have the added advantage of not having to rely just on their own engineering team to provide all the bells and whistles that customers may require out of their database systems. Li said that the open ecosystem allows Alibaba to leverage the collective intelligence of the entire community to help them build and develop a rich set of tools, utilities and support that customers can benefit, often for free.
As he further explained, “If you use an open source or open source-compatible database system, chances are, you’ll be able to find a tool that can support what you need. Or even if you don’t, you can post a request to the open source community and somebody from somewhere will pick up on that, if not by us at Alibaba Cloud. So that’s the benefit of being open and being able to migrate and run your business application in a bigger, larger open source ecosystem than doing so in a closed system setup.”
While the total database market share is still dominated by traditional, on-prem database vendors, Li was quick to point out that the growth of the cloud database market is accelerating very fast – doubling every year in the last two to three years. In that time, Alibaba Cloud has been doing well for itself in this space, generating the third largest cloud database management system (DBMS) revenue among global players in 2018, just behind AWS and Microsoft Azure.
At the rate that the market is growing plus the benefits that cloud database systems currently have over their more traditional, on-prem and closed system equivalent, as mentioned by Li, we can surmise that Gartner’s prediction that the majority of businesses will migrate to a cloud database platform by 2022, does not look like a distant possibility.