Zomato Ingests Massive Data Streams with Cost-Effective Logging Platform

Zomato, the popular Indian food delivery platform, has unveiled its innovative approach to handling the immense volume of logs generated by its operations. This novel logging system, designed for petabyte-scale data ingestion, empowers Zomato to analyze critical information and optimize various aspects of its business.

Previously, Zomato relied on a self-hosted Elasticsearch cluster to store logs. With a daily maximum of 150 million logs per minute, this translated to a staggering 50 terabytes of uncompressed data. However, as user traffic and the platform itself grew exponentially, managing these ever-expanding Elasticsearch clusters became a significant operational burden.

To address this challenge, Zomato's engineers strategically migrated to ClickHouse, an open-source columnar database management system. ClickHouse's inherent strengths, including its exceptional scalability and optimized architecture for real-time analytics, made it an ideal choice for Zomato's logging needs.

One of the key advantages of ClickHouse is its distributed architecture. Unlike traditional relational databases with a single point of failure, ClickHouse seamlessly distributes data across multiple nodes. This ensures robustness and reliability – if a particular node experiences unavailability or high load, data ingestion automatically redirects to a functional node. This redundancy safeguards against potential disruptions in the logging process.

Another crucial aspect of Zomato's logging system is its schema design. Zomato's logs are inherently semi-structured, meaning they possess a flexible data format that can evolve over time. ClickHouse's schema readily accommodates this by allowing for the creation of semi-structured tables. In this design, common data points across all logs occupy the top level of the table, while any unique or unmatched columns are stored within a dedicated map data type. This map can be seamlessly expanded at the API level, granting users unhindered access to the data.

Implementing ClickHouse has resulted in significant benefits for Zomato. The cost-effectiveness of the solution is a major advantage – ClickHouse's open-source nature eliminates licensing fees associated with proprietary solutions. Additionally, ClickHouse's efficient storage and retrieval capabilities streamline data management, reducing operational overhead.

Zomato's innovative logging system serves as a prime example of leveraging technology to address large-scale data challenges. By adopting ClickHouse, Zomato has secured a robust, scalable, and cost-effective platform to manage its ever-growing stream of logs, empowering them to gain valuable insights and continuously optimize their food delivery operations.