Debezium Blog

When people think about Debezium integrations, the first idea is usually a production pipeline: capture changes from a database, deliver them to Kafka, and route them to downstream services or analytical systems. That is, of course, a very common and very good use case. But there is another side to change data capture that deserves attention too: exploration.

Sometimes you do not want to start with a full deployment and a chain of consumers. Sometimes you want to inspect the events, validate a connector configuration, build a quick proof of concept, or show colleagues what the stream actually looks like. In these situations, an interactive environment can be much more effective than a traditional application.

This is where Jupyter notebooks fit surprisingly well.

In this post, we will look at a simple Debezium and Jupyter integration based on the jupyter example in the debezium-examples repository. The demo runs Debezium Engine from a notebook by using pydbzengine, captures changes from PostgreSQL, stores the resulting records in a pandas data frame, and lets you inspect and aggregate them interactively.

When it comes to replicating operational data for analytics, Change Data Capture (CDC) is the gold standard. It offers scalability, near real-time performance, and captures all data modifications, ensuring your analytical datasets are always up-to-date. Debezium is a leading tool in this space, connecting to a wide range of databases and exporting CDC events in various formats like JSON and Avro, making integration with diverse systems a breeze.

While Debezium itself is a Java-based project, the data engineering world increasingly relies on Python. This blog post demonstrates how to leverage Debezium within a Python environment, using pydbzengine. We’ll explore how to use these technologies to build a robust and scalable CDC solution.