Self-host GlassFlow ClickHouse ETL
v3.2.0Open-source stream engine for ClickHouse pipelines
- Stars
- 503
- Ports
- 8080 (Web UI), 8081 (API)
- Requires
- ClickHouse
What is GlassFlow ClickHouse ETL?
This project provides a stream processing engine that moves data into ClickHouse from sources like Kafka and OpenTelemetry. It applies transformations with an expression engine, handles deduplication using an embedded state store, and supports temporal joins across time windows. A web interface and Python SDK let operators manage pipelines. Deployment targets Kubernetes via Helm. The open-source edition covers core sources and sinks, while a commercial tier adds more connectors and operational features. The codebase uses the Apache-2.0 license.
Should you self-host GlassFlow ClickHouse ETL?
GlassFlow ClickHouse ETL scores 53/100 on selfhost.directory (how scoring works) and ranks #6 of 8 in Data Pipeline. It has 503 GitHub stars.
It shipped 10 stable releases in the past year. The latest, v3.2.0, came out on 22 May 2026. New versions land roughly every 6 days.
Setup is advanced: plan for several services and some ongoing maintenance. It is Apache-2.0-licensed and written mostly in Typescript.
Its closest alternative on selfhost.directory is apache-nifi (73/100, 6 277 stars). It scores higher overall, so compare the two before you pick. See GlassFlow ClickHouse ETL vs apache-nifi side by side.
Features
- Dead Letter Queue
- Deduplication
- Filtering
- Ingest only mode
- Kafka source
- Metrics & OpenTelemetry
- OTLP source
- Prometheus Metrics
- Python SDK
- Stateful transformations
- Stateless transformations
- Temporal joins
- Web UI
GlassFlow ClickHouse ETL README
FAQ
What happens during failures? Can you lose or duplicate data?
GlassFlow uses NATS JetStream as an internal buffer. The source is only acknowledged after messages are successfully published to NATS. If the system crashes after acknowledging the source but before inserting into ClickHouse, that batch is lost. Recovery guarantees are being actively improved.
Why do duplicates happen in data pipelines at all?
Duplicates can occur from producer retries, network issues, consumer reprocessing after failures in Kafka, or collector retries and overlapping export intervals in OTLP pipelines.
How does GlassFlow's deduplication work?
GlassFlow uses BadgerDB for deduplication in disk and memory.
How is GlassFlow's deduplication different from ClickHouse's ReplacingMergeTree?
ReplacingMergeTree performs deduplication via background merges which can delay accurate query results unless you force merges with FINAL, impacting read performance. GlassFlow moves deduplication upstream before data is written to ClickHouse, ensuring real-time correctness and reducing load on ClickHouse.
More in ETL
KestraSelf-hosted, open-source (Apache-2.0) engine that schedules and chains data, AI and infrastructure jobs from YAML or a visual editor, with 2,200+ plugins.
llm-appOpen-source, self-hosted template collection that streams your files, Google Drive or S3 into live RAG, vector-store and enterprise-search APIs.
logstashAn open-source server-side data pipeline that ingests, transforms and routes events with 200+ plugins; self-hosted via Docker, deb/rpm packages or tar/zip downloads.
multiwovenSelf-hosted, open-source Reverse ETL platform syncing data from Snowflake, BigQuery, Redshift to 50+ tools like Salesforce, HubSpot, Slack. AGPL-3.0 licensed.
PeerDBSelf-hosted, open-source (AGPL-3.0) ETL tool streaming Postgres to ClickHouse and other warehouses via CDC, with a web UI and a Postgres-compatible SQL interface.
rudderstackAn open-source, Segment-alternative customer data platform: run the self-hosted Go/React event pipeline and route data to 90+ warehouse and tool destinations.
superglueAI-agent integration platform: agents build connectors, migrate data and sync ERPs, CRMs, databases. Source-available, self-hosted with Docker Compose or on Kubernetes.
airbyteAirbyte is a self-hosted data movement platform with hundreds of connectors for ELT pipelines and AI agents, deployable via Docker, Kubernetes, and abctl CLI.
Apache HopApache Hop is a self-hosted, open-source data integration platform for visual pipelines that run on local, Hop Server, Spark, Flink, and Dataflow without rewriting.
apache-nifiApache NiFi is an open-source, self-hosted data flow automation platform with browser-based UI, provenance tracking, and secure configuration.
dagsterOpen-source, self-hosted data orchestration platform whose pipelines are Python-defined data assets with lineage, schedules and a web UI; runs via pip or Docker Compose.
DuckleDuckle is a free, open-source, self-hosted ETL platform: build pipelines visually or in Python/SQL, compile to DuckDB SQL, run on your own infra.
0 comments