Six Years Running Petabyte ClickHouse: Where the Open-Source Version Falls Short

A Tinybird engineer who has run ClickHouse in production since version 18.4 argues that standing up a cluster is trivial—keeping petabyte-scale clusters healthy is the real work. His team deliberately avoided sharding for years, scaling replicas vertically and horizontally instead, because re-sharding is painful and careful schema design lets you defer it. The setup leans on an HTTP load balancer (not the native TCP protocol) that routes queries by type, load, and consistent hashing, plus dedicated replicas for writes and for latency-sensitive workloads that must be kept under ~40% load to hold p99 targets. The catch is cost: replicating full datasets across every machine means a 300TB table serving heavy traffic can balloon to thousands of terabytes of SSD.

The sharper critique is aimed at storage. ClickHouse’s open-source build still treats local disk as a first-class citizen and lags competitors on cloud storage—Snowflake normalized compute/storage separation over a decade ago, and StarRocks is further along. The S3 path exists mainly through zero-copy replication, but that feature was contributed from outside ClickHouse, Inc., which appears reluctant to support it: it is buggy, can lose data during certain part operations, and litters S3 with garbage. ClickHouse Cloud’s own storage layer is closed-source. Because the engine was built assuming near-free local writes, it hammers S3 with excessive write operations and file counts.

Tinybird works around this with a modified zero-copy fork, local SSD caching, and a hot/cold tiering scheme to get latency S3 alone can’t deliver. The takeaway for operators: compute/storage separation via zero-copy is viable and worthwhile, but only if you understand its failure modes, monitor closely for data loss, and accept that ClickHouse, Inc. could drop the feature at any time.