Self-Serve BigQuery Is an Interface Problem, Not a Data Problem

When we built the analytics platform for a US mass-media corporation at CI&T (Apache Beam pipelines on Dataflow feeding a BigQuery warehouse, with App Engine services and a React front end on top), the pipelines were the part that got the attention. Ingesting massive event streams reliably, keeping schemas from breaking downstream consumers: that was the visible engineering problem. What took longer to get right, and what I think about more now as a FinOps consultant, was something quieter: getting the warehouse into a state where a product manager or an editor could open it and ask their own question, without filing a ticket and waiting for an engineer to write the SQL.

Self-serve sounds like a permissions checkbox. Grant read access to the dataset, point people at a query editor, done. In practice that's the fastest way to make a warehouse unusable to the people you built it for. Raw event tables in a system like this are denormalized, partitioned for ingestion efficiency rather than readability, and full of columns that only make sense if you know how the pipeline populated them. A non-engineer given direct access to that layer doesn't get self-serve, they get a wall of tables named after internal pipeline stages, and they go back to asking an engineer to write the query. The access was never the bottleneck. The interface was.

What actually made self-serve work was a curated layer sitting between the raw tables and the people using them: views built specifically to be read by someone who doesn't know the ingestion internals, named after the business concepts editors and product managers already used in conversation, not after the tables they were derived from. BigQuery's authorized views were the mechanism for this: they let you expose a query result to a group of users without granting them access to the underlying dataset at all, so the raw layer could stay locked down while the curated layer stayed open. That separation did two things at once. It gave analysts a stable, comprehensible surface to query against, and it gave engineers room to change the raw schema underneath without breaking anyone's dashboard, since the view's contract to its consumers was independent of how the source tables were structured. Getting the naming and grouping of those views right took more iteration than the query logic inside them: you're designing for someone who thinks in 'articles read by region last week', not in join keys.

The other half of the problem was cost, and it's the half that gets skipped when self-serve is treated as a solved problem once the views exist. BigQuery bills on-demand queries by bytes scanned, and a self-serve user exploring a warehouse of event-level media data can trigger a query that scans terabytes without realizing it, especially against a table that isn't partitioned or clustered the way they'd expect. An engineer writing a query knows to check the bytes-processed estimate first. A product manager clicking through a query editor generally doesn't, and shouldn't have to. The fix wasn't to trust people to be careful, it was to build the guardrails into the platform: custom per-user and per-project query quotas so a single runaway query couldn't consume a disproportionate share of the daily budget, and views scoped tightly enough that an accidental full-table scan hit a curated subset instead of the entire event stream. Cost control, in a self-serve context, is a design constraint on the interface, not a monitoring dashboard you check after the fact.

The lesson I carried forward into FinOps work is that self-serve analytics fails or succeeds on decisions that look unrelated to data modeling: what you name a view, who gets access to which layer, what happens by default when someone runs an expensive query without knowing it's expensive. None of that shows up in a pipeline diagram. All of it determines whether a warehouse actually gets used by the people it was built for, or quietly reverts to asking an engineer, which is the state you were trying to escape in the first place. The engineering effort to build the warehouse and the design effort to make it approachable are different disciplines, and treating the second as an afterthought to the first is the most common way these platforms underdeliver on their promise.