Fast analytics are critical for business success, because you cannot act quickly when your dashboards are stale. But how do you keep latency down when your growth is through the roof? This is a war story about scaling our analytics stack from 5 to 100 million users in a single month.
As a web performance startup, accelerating websites is our day-to-day business and tracking user performance is part of the job: We collect traces for every single page load that happens on our infrastructure to monitor the critical business KPIs in real-time. But with our user base doubling every other month, performing heavy-weight analytics with sub-minute end-to-end latency turned from "difficult" to "seemingly impossible". Together with the AWS Prototyping Team, we tackled the problem head-on and reengineered our analytics stack from scratch: We replaced our ETL-heavy Athena warehouse with a continuously running Kinesis Data Analytics application and devised a blithesome dashboard based on Elasticsearch and Kibana. In this talk, we will share our most exciting learnings of that journey.
