Data Platform Engineering
Big Data Analytics Platform
Mass-media corporation (US) · Mass-scale event ingestion
10×
analytics throughput
The Challenge
A media corporation needed a scalable analytics platform to process and query massive event streams.
The Approach
Apache Beam pipelines on Google DataFlow, App Engine services, and a BigQuery warehouse, built in Java/Node.js with a React front end.
The Outcome
Unlocked self-serve, real-time analytics over massive event streams on Google Cloud.
- ✓Real-time data pipelines
- ✓Self-serve BigQuery analytics
- ✓Elastic App Engine delivery
What I learned
Design for schema evolution, not just throughput. Event schemas changed six times during the project. Building schema evolution into the pipeline from day one would have saved weeks of rework.
JavaApache BeamDataFlowBigQueryApp Engine
Have a similar problem to solve?