Towards Decentralized Auto-Scaling Policies for Data Stream Processing Applications

G. Russo Russo

Proc. of 10th ZEUS Workshop (ZEUS 2018), Dresden, Germany, February 2018.

[pdf]

Data Stream Processing applications can process large data volumes in near real-time. In order to face varying workloads in a scalable and cost-effective manner, it is critical to adjust the application parallelism at run-time. We formulate the elasticity problem as a Markov Decision Process (MDP). As the MDP resolution requires full knowledge of the system dynamics, which is rarely available, we rely on model based Reinforcement Learning to improve the scaling policy at run-time. We show promising results even for a decentralized approach, compared tothe optimal MDP solution.