Benchmarking Tool for Modern Distributed Stream Processing Engines

Abstract

There is an upsurge in the usage and adaptation of streaming applications in the recent years by both industry and academia. At the core of these applications is streaming data processing engines that perform resource management and allocation in order to support continuous track of queries over distributed data streams. Several stream processing engines exists to handle these distributed streaming applications. In this paper, we present different challenges of the stream processing systems, in particular to stateful operators and implement Linear Road benchmark to examine the characteristic and performance metrics of the streaming system, in particular Apache Flink. Furthermore, we examine that Apache Flink can be used as a core for an efficient Linear Road application implementation for distributed environments without breaching the SLA requirements of the application.