DRYCODEWORKS

Data engineering & ClickHouse: database consulting

Slow dashboards, numbers that don't match the source, a database bill that grows every month, a migration nobody wants to start: these usually come down to how the data is moved and laid out. I fix it, often in ClickHouse, with pipelines from PostgreSQL and SQL Server.

Book a callSee client work

What I build for

Migrations without downtime

A migration shouldn't be a bet on the live database. I load and check the new copy alongside the old one, by change capture, dual writes or a resumable backfill, then switch over in one step and keep the old one as the way back.

Data you can trust

Analytics only helps if it matches the source. I deduplicate on the way in and check the copy against the source, so a stale or doubled row is caught before anyone sees it on a dashboard.

Speed without a bigger cluster

Slow queries, out-of-memory crashes and a growing bill usually trace back to the table design. I read your query logs first, then shape the sort keys, materialized views and dictionaries around the queries you actually run, and batch inserts so writes keep up, before anyone buys a bigger cluster.

Schema changes

Schema changes deserve the same review as code. ClickHouse can't roll back a failed migration, so each statement has to be one you can run again without damage. ch-migrate, the open-source tool I maintain, checks that for you.
Your appwrites as people use itthe old path, keptas the way backDashboards3 readers query ClickHouseeach query reads 1 tablefast, on the same clusterRelational databasePostgreSQL or SQL ServerClickHouseone wide, sorted table121 · Migrations without downtime2 · Speed without a bigger cluster

From client work

All case studies →
Moving IoT analytics from SQL Server to ClickHouse1.3 billion sensor rows moved to ClickHouse Cloud with no downtime, for $2,200 a month less.
Dashboard query time45 s → under 2 sIoT sensor platform · In production
→
Cutting ClickHouse materialized view memory with a dictionaryMoving a materialized view's lookup from an ASOF JOIN to a dictionary cut its per-insert memory, while the cluster's resident memory stayed flat.
Per-insert view memorybaseline → 95% lessClickHouse tuning · Measured in nonprod
→

Questions

We're looking at ClickHouse because it's fast. Is it right for us?

It's a strong fit for time-series data and queries over time ranges; observability, meaning logs, metrics and traces; high write volumes; and analytics combining multiple sources. Relational databases (RDBMS) such as PostgreSQL and SQL Server are built for the application itself: reading and changing individual records inside transactions. In database terms, ClickHouse is built for analytics (OLAP), and a relational database for transactions (OLTP). The insert-batching worked example explores write shape without claiming a client result.

Do we have to choose between ClickHouse and our relational database?

Often the answer is both. The relational database keeps running the application, and a pipeline copies its data into ClickHouse, which holds the history and serves dashboards and reports. Each database does the work it was built for. The resumable-backfill worked example illustrates one way to build that copy.

How hard is it to switch to ClickHouse?

Most of the work is in the data model. A relational database splits data across many tables and joins them when you query. ClickHouse is fastest with fewer, wider tables that already hold everything a query needs, sorted the way you query them. Plan on remodeling the data first, then moving it.

We take in a lot of data, and our database bill keeps climbing. Would ClickHouse cost less?

Often, yes. Adding rows is cheap in ClickHouse. It writes them in large sorted blocks and merges those in the background, instead of updating indexes row by row, so the same hardware takes in far more data than a relational database would. It also compresses each column, so the data takes a fraction of the disk. One client’s database bill went from $4,000 a month to $1,800, including the relational database they kept for the application. Small inserts can create too many parts; the batching worked example explains why.

Can't we just make our current database faster?

Sometimes, yes. Indexes that match your queries, partitioning, or tables of pre-computed totals can make a relational database fast enough, and one database is simpler to run than two. One IoT platform in my case studies did both. First it stayed on SQL Server, with response times cut from 12 seconds to 150 ms. Later, its analytics moved to ClickHouse as its sensor history grew.

Tell me what you're building

A free 30-minute call with me, not a sales team. Bring the diagram or the prototype. My engagements run long and personal, so I'd rather know we fit first. Either way, you'll get a Slack Connect invite for later questions.

Book a call

How an engagement runs

First callFree, 30 minutes. You tell me what's in the way, and I'll tell you whether it's a problem I'd take on. It's a fit only if we both say so.
ShapeEngagements flex to the problem: an assessment, a build, a rescue, or working inside your team. We pick the shape after the call, not before.
DeliveryEverything ships through your repos and your review process, with the tests and docs your team needs to run it without me.