Table of Contents
- Executive Summary
- Streaming Data in the Cloud
- Ease-of-Use Test Setup
- About William McKnight
- About Jake Dolezal
This report focuses on real-time data and how autonomous systems can be fed at scale reliably. To shed light on this challenge, we assess the ease of use of a fully managed Kafka platform—Confluent Cloud—and a self-managed open-source Apache Kafka solution.
The most popular tool for streaming data is the Apache Kafka project. Created by LinkedIn, Kafka was open sourced and graduated from the Apache Incubator in late 2012. Kafka is a distributed publish-subscribe messaging system that maintains feeds in groups, known as topics. Publishers write data to topics where subscribers can read them. Kafka is a distributed system, where topics are partitioned and replicated across multiple nodes in the cluster.
Within Kafka, messages are key/value pairs that can store objects in any format. Messages with the same key are ordered and stored in the same partition so they can be consumed by the same instance of a subscriber.
In our test, we utilized all steps of a use case for a distributed event store and stream-processing platform. The categories, or components of ease of use, included in our calculations were across setup, development, operations, and scale categories.
Using story points, we assessed the comparative ease-of-use value realization between Confluent Cloud and Kafka across setup, development, and operations. We found that the value realization of fully managed Confluent Cloud was about three times that of open-source Kafka in setup, nearly double in development, and more than double in operations.
Scalability is a significant component of why fully managed Confluent Cloud is easier to use than open-source Kafka. It’s easy to get started and can grow up to 5 GBps ingress with the click of a button. This requires hours or days of manual effort with open-source Kafka
Our team found that fully managed Confluent Cloud is much easier to use than open-source Kafka. While Confluent Cloud accelerates the setup, development, and operations, the most impressive feature is the seamless scale out for when the application grows.