Batching was Yesterday! Real-Time Tracking For 100+ Million Visitors

November 30, 2020

Fast analytics are critical for business success, because you cannot act quickly when your dashboards are stale. But how do you keep latency down when your growth is through the roof? This is a war story about scaling our analytics stack from 5 to 100 million users in a single month.

As a web performance startup, accelerating websites is our day-to-day business and tracking user performance is part of the job: We collect traces for every single page load that happens on our infrastructure to monitor the critical business KPIs in real-time. But with our user base doubling every other month, performing heavy-weight analytics with sub-minute end-to-end latency turned from "difficult" to "seemingly impossible". Together with the AWS Prototyping Team, we tackled the problem head-on and reengineered our analytics stack from scratch: We replaced our ETL-heavy Athena warehouse with a continuously running Kinesis Data Analytics application and devised a blithesome dashboard based on Elasticsearch and Kibana. In this talk, we will share our most exciting learnings of that journey.

Previous Video
From Zero to BSI C5: How a small company achieved BSI C5 attestation and how you can do it, too.
From Zero to BSI C5: How a small company achieved BSI C5 attestation and how you can do it, too.

We describe what C5 is in detail, how to leverage AWS and how C5 transforms a startup into a better, strong...

Next Video
Content of the month: How to get started on AWS
Content of the month: How to get started on AWS