“I believe that “one size does not fit all”… in every vertical market I can think of, there is a way to beat legacy relational DBMSs by 1-2 orders of magnitude.”

– Michael Stonebraker, Database Guru, Founder Ingres, Vertica, VoltDB and former CTO of Informix”

Things are really heating up in the database industry; the last ten years have seen an explosion in the number of products and vendors (analyst firm 451 Research now tracks 386 different products on its popular ‘tube-map’ of the database landscape). The rapid evolution of the internet and the move towards data driven decisions across all industries has created a rich environment for database innovation. The worldwide adoption of services from Google and Facebook has accelerated this trend and created a need for backend database technology to operate at webscale, allowing millions or even billions of users to access different services when they want them, through a mobile app which relies on real-time decision making and instant responses.

For over thirty years things were pretty simple; relational database technology was ‘good enough’ for almost everything and became the dominant storage model for transactional platforms and enterprise applications. You could use the same standard database technology for all applications from managing customer data, to transaction processing, to tracking usage records. These applications could be deployed and run across businesses without too many surprises as volumes grew at a reasonable pace and were fairly predictable. The core technology was typically main-frame or client-server based, and sized for known transaction rates and number of users, and no one ever got fired for choosing a relational database.

The internet changed everything for databases as suddenly companies were able to sell direct to consumers across the globe. Traditional relational databases were not equipped to handle the technology shift to rapid unplanned adoption rates, flexible data sets to handle new use cases and new types of users as business models evolved. The pressure for increased performance aligned with the falling cost of memory opened the door to optimize using in-memory database technology. But ‘falling’ does not mean ‘cheap’, and in-memory systems remained a luxury for key platforms that could justify the cost.

The launch of the iPhone and the global take-up of social media and other Apps created a new problem in the late 2000’s. Facebook and other internet giants were trying to figure out how they could handle billions of users and still give rapid responses to queries. They developed their own technology, Cassandra, to power these ‘web-scale’ requirements for large scale data storage, high availability and a cost efficient hardware model that could scale to unprecedented volumes. This is one of the earliest high profile examples of where a company’s unique vertical requirements could not be met by existing generic technologies and they were forced to build a specialized solution. The switch to on-demand services across many industries following the success of Uber, Netflix and others has driven the latest wave of database innovation. To transition to on-demand a company must be able to extract insights and make decisions in real-time despite massive volumes of dynamic data. This is creating opportunities for vertical databases in mobile, financial services, advertising and other industries to deliver specialized requirements around performance, scale and consistency.

In my next blog in this series I will discuss what’s so different about the needs of a Telco, and why other technologies struggle to meet these needs. Then I’ll dive into the database technology we have developed at MATRIXX.

Blog 2/3: Telco databases are tricky