> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-fix-nav-issues.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> Introduction to Kafka with ClickHouse

# Integrating Kafka with ClickHouse

[Apache Kafka](https://kafka.apache.org/) is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. ClickHouse provides multiple options to **read from** and **write to** Kafka and other Kafka API-compatible brokers (e.g., Redpanda, Amazon MSK).

<h2 id="available-options">
  Available options
</h2>

Choosing the right option for your use case depends on multiple factors, including your ClickHouse deployment type, data flow direction and operational requirements.

| Option                                                                                            | Deployment type                | Fully managed | Kafka to ClickHouse | ClickHouse to Kafka |
| ------------------------------------------------------------------------------------------------- | ------------------------------ | :-----------: | :-----------------: | :-----------------: |
| [ClickPipes for Kafka](/integrations/clickpipes/kafka/index)                                      | [Cloud], [BYOC] (coming soon!) |       ✅       |          ✅          |                     |
| [Kafka Connect Sink](/integrations/connectors/data-ingestion/kafka/kafka-clickhouse-connect-sink) | [Cloud], [BYOC], [Self-hosted] |               |          ✅          |                     |
| [Kafka table engine](/integrations/connectors/data-ingestion/kafka/kafka-table-engine)            | [Cloud], [BYOC], [Self-hosted] |               |          ✅          |          ✅          |

For a more detailed comparison between these options, see [Choosing an option](#choosing-an-option).

<h3 id="clickpipes-for-kafka">
  ClickPipes for Kafka
</h3>

[ClickPipes](/integrations/clickpipes/home) is a managed integration platform that makes ingesting data from a diverse set of sources as simple as clicking a few buttons. Because it is fully managed and purpose-built for production workloads, ClickPipes significantly lowers infrastructure and operational costs, removing the need for external data streaming and ETL tools.

<Tip>
  This is the recommended option if you're a ClickHouse Cloud user. ClickPipes is **fully managed** and purpose-built to deliver the **best performance** in Cloud environments.
</Tip>

<h4 id="clickpipes-for-kafka-main-features">
  Main features
</h4>

* Optimized for ClickHouse Cloud, delivering blazing-fast performance
* Horizontal and vertical scalability for high-throughput workloads
* Built-in fault tolerance with configurable replicas and automatic retries
* Deployment and management via ClickHouse Cloud UI, [Open API](/integrations/clickpipes/programmatic-access/openapi), or [Terraform](/integrations/clickpipes/programmatic-access/terraform)
* Enterprise-grade security with support for cloud-native authorization (IAM) and private connectivity (PrivateLink)
* Supports a wide range of [data sources](/integrations/clickpipes/kafka/reference), including Confluent Cloud, Amazon MSK, Redpanda Cloud, and Azure Event Hubs
* Supports most common serialization formats (JSON, Avro, Protobuf)

<h4 id="clickpipes-for-kafka-getting-started">
  Getting started
</h4>

To get started using ClickPipes for Kafka, see the [reference documentation](/integrations/clickpipes/kafka/reference) or navigate to the `Data Sources` tab in the ClickHouse Cloud UI.

<h3 id="kafka-connect-sink">
  Kafka Connect Sink
</h3>

Kafka Connect is an open-source framework that works as a centralized data hub for simple data integration between Kafka and other data systems. The [ClickHouse Kafka Connect Sink](https://github.com/ClickHouse/clickhouse-kafka-connect) connector provides a scalable and highly-configurable option to read data from Apache Kafka and other Kafka API-compatible brokers.

<Tip>
  This is the recommended option if you prefer **high configurability** or are already a Kafka Connect user.
</Tip>

<h4 id="kafka-connect-sink-main-features">
  Main features
</h4>

* Can be configured to support exactly-once semantics
* Supports most common serialization formats (JSON, Avro, Protobuf)
* Tested continuously against ClickHouse Cloud

<h4 id="kafka-connect-sink-getting-started">
  Getting started
</h4>

To get started using the ClickHouse Kafka Connect Sink, see the [reference documentation](/integrations/connectors/data-ingestion/kafka/kafka-clickhouse-connect-sink).

<h3 id="kafka-table-engine">
  Kafka table engine
</h3>

The [Kafka table engine](/integrations/connectors/data-ingestion/kafka/kafka-table-engine) can be used to read data from and write data to Apache Kafka and other Kafka API-compatible brokers. This option is bundled with open-source ClickHouse and is available across all deployment types.

<Tip>
  This is the recommended option if you're self-hosting ClickHouse and need a **low entry barrier** option, or if you need to **write** data to Kafka.
</Tip>

<h4 id="kafka-table-engine-main-features">
  Main features
</h4>

* Can be used for [reading](/integrations/connectors/data-ingestion/kafka/kafka-table-engine#kafka-to-clickhouse) and [writing](/integrations/connectors/data-ingestion/kafka/kafka-table-engine#clickhouse-to-kafka) data
* Bundled with open-source ClickHouse
* Supports most common serialization formats (JSON, Avro, Protobuf)

<h4 id="kafka-table-engine-getting-started">
  Getting started
</h4>

To get started using the Kafka table engine, see the [reference documentation](/integrations/connectors/data-ingestion/kafka/kafka-table-engine).

<h3 id="choosing-an-option">
  Choosing an option
</h3>

| Product                  | Strengths                                                                                                                                                                                                                                                                                               | Weaknesses                                                                                                                                                                                                              |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **ClickPipes for Kafka** | • Scalable architecture for high throughput and low latency<br />• Built-in monitoring and schema management<br />• Private networking connections (via PrivateLink)<br />• Supports SSL/TLS authentication and IAM authorization<br />• Supports programmatic configuration (Terraform, API endpoints) | • Doesn't support pushing data to Kafka<br />• At-least-once semantics                                                                                                                                                  |
| **Kafka Connect Sink**   | • Exactly-once semantics<br />• Allows granular control over data transformation, batching and error handling<br />• Can be deployed in private networks<br />• Allows real-time replication from databases not yet supported in ClickPipes via Debezium                                                | • Doesn't support pushing data to Kafka<br />• Operationally complex to set up and maintain<br />• Requires Kafka and Kafka Connect expertise                                                                           |
| **Kafka table engine**   | • Supports [pushing data to Kafka](/integrations/connectors/data-ingestion/kafka/kafka-table-engine#clickhouse-to-kafka)<br />• Operationally simple to set up                                                                                                                                          | • At-least-once semantics<br />• Limited horizontal scaling for consumers. Can't be scaled independently from the ClickHouse server<br />• Limited error handling and debugging options<br />• Requires Kafka expertise |

<h3 id="other-options">
  Other options
</h3>

* [**Confluent Cloud**](/integrations/connectors/data-ingestion/kafka/confluent/index) - Confluent Platform provides an option to upload and [run ClickHouse Connector Sink on Confluent Cloud](/integrations/connectors/data-ingestion/kafka/confluent/custom-connector) or use [HTTP Sink Connector for Confluent Platform](/integrations/connectors/data-ingestion/kafka/confluent/kafka-connect-http) that integrates Apache Kafka with an API via HTTP or HTTPS.

* [**Vector**](/integrations/connectors/data-ingestion/kafka/kafka-vector) - Vector is a vendor-agnostic data pipeline. With the ability to read from Kafka, and send events to ClickHouse, this represents a robust integration option.

* [**JDBC Connect Sink**](/integrations/connectors/data-ingestion/kafka/kafka-connect-jdbc) - The Kafka Connect JDBC Sink connector allows you to export data from Kafka topics to any relational database with a JDBC driver.

* **Custom code** - Custom code using Kafka and ClickHouse [client libraries](/integrations/language-clients/index) may be appropriate in cases where custom processing of events is required.

[BYOC]: /products/cloud/guides/infrastructure/deployment-options/byoc/overview

[Cloud]: /products/cloud/getting-started/cloud-get-started

[Self-hosted]: /get-started/about/intro