Scaling a real-time streaming warehouse with Apache Flink, Parquet and Kubernetes--Aditi Verma (Branch Metrics) & Ramesh Shanmugam (Branch Metrics)
对应的现场视频已上传至B站，地址为 https://www.bilibili.com/video/av53226221/
At Branch, we process more than 12 billions events per day, and store and aggregate terabytes of data daily. We use Apache Flink for processing, transforming and aggregating events, and parquet as the data storage format. This talk covers our challenges with scaling our warehouse, namely:
- How did we scale our Flink-Parquet warehouse to handle 3x increase in traffic?
- How do we ensure exactly once, event-time based, fault tolerant processing of events?
In this talk, we also provide an overview on deploying and scaling our streaming warehouse. We give an overview on:
- How we scaled our Parquet warehouse by tuning memory
- Running on Kubernetes cluster for resource management
- How we migrated our streaming jobs with no disruption from Mesos to Kubernetes
- Our challenges and learnings along the way 、 使用Apache Flink、Parquet和Kubernetes扩展实时流式仓库
在Branch，我们每天处理超过120亿个事件，并每天存储和聚合万亿字节的数据。我们使用ApacheFlink来处理、转换和聚合事件，并使用拼花作为数据存储格式。本次讨论涵盖了我们在扩展仓库方面面临的挑战，即：
我们如何扩大我们的Flink镶木地板仓库来处理3倍的流量增长？
我们如何确保事件的一次性、基于事件时间的容错处理？
在本文中，我们还概述了如何部署和扩展流仓库。我们概述了：
我们如何通过调整内存来扩展我们的镶木地板仓库
在Kubernetes集群上运行以进行资源管理
我们如何在不中断从Meos到Kubernetes的情况下迁移流媒体工作
一路走来的挑战和学习
