Design for Schema Evolution, Not Just Throughput

Throughput gets the attention in a big data build. Stakeholders ask how many events per second, how many terabytes per day, how the numbers compare to the old batch job. Those are fair questions, and the platform I built for a US media corporation answered them: real-time analytics over event streams that used to take a batch window to query, running on Apache Beam and BigQuery.

What stakeholders don't ask about, and what actually determined whether the project succeeded, was what happens when the event schema changes. On this build it changed six times during development. A new field here, a renamed key there, a type that widened from an integer to a float once someone started measuring in decimals. None of those changes were mistakes. They were the product teams doing their job while the platform was being built underneath them.

A pipeline that assumes a fixed schema treats every one of those six changes as an incident: a broken job, a Slack thread, a hotfix deployed under pressure. A pipeline designed for evolution treats them as Tuesday. The difference isn't a clever library. It's a decision made on day one to validate incoming events against a versioned schema, route anything that fails validation to a dead-letter path instead of crashing the job, and let downstream consumers declare which schema version they're reading rather than assuming there's only one.

That decision costs something upfront. It means writing the schema registry and the validation layer before you've proven the pipeline works at all, which feels like solving a problem you don't have yet. By the third schema change, it's the reason the platform kept running while the roadmap kept moving.

If I rebuilt this platform today, throughput would still matter, and BigQuery and Beam would still be the right tools for the volume. But the first design review wouldn't start with a load test. It would start with a fake schema change, run through the pipeline, to see what breaks.