Logo des Repositoriums
 

On-the-fly Reconfiguration of Query Plans for Stateful Stream Processing Engines

Kurzanzeige
dc.contributor.authorBartnik, Adrian
dc.contributor.authorDel Monte, Bonaventura
dc.contributor.authorRabl, Tilmann
dc.contributor.authorMarkl, Volker
dc.contributor.editorGrust, Torsten
dc.contributor.editorNaumann, Felix
dc.contributor.editorBöhm, Alexander
dc.contributor.editorLehner, Wolfgang
dc.contributor.editorHärder, Theo
dc.contributor.editorRahm, Erhard
dc.contributor.editorHeuer, Andreas
dc.contributor.editorKlettke, Meike
dc.contributor.editorMeyer, Holger
dc.date.accessioned2019-04-11T07:21:40Z
dc.date.available2019-04-11T07:21:40Z
dc.date.issued2019
dc.description.abstractStream Processing Engines (SPEs) must tolerate the dynamic nature of unbounded data streams and provide means to quickly adapt to fluctuations in the data rate. Many major SPEs however provide very little functionality to adjust the execution of a potentially infinite streaming query at runtime. Each modification requires a complete query restart, which involves an expensive redistribution of the state of a query and may require external systems in order to guarantee correct processing semantics. This results in significant downtime, which increase the operational cost of those SPEs. We present a modification protocol that enables modifying specific operators as well as the data flow of a running query while ensuring exactly-once processing semantics. We provide an implementation for Apache Flink, which enables stateful operator migration across machines, the introduction of new operators into a running query, and changes to a specific operator based on external triggers. Our results on two benchmarks show that migrating operators for queries with small state is as fast as using the savepoint mechanism of Flink. Migrating operators in the presence of large state even outperforms the savepoint mechanism by a factor of more than 2.3. Introducing and replacing operators at runtime is performed in less than 10 s. Our modification protocol demonstrates the general feasibility of runtime modifications and opens the door for many other modification use cases, such as online algorithm tweaking and up-or downscaling operator instances.en
dc.identifier.doi10.18420/btw2019-09
dc.identifier.isbn978-3-88579-683-1
dc.identifier.pissn1617-5468
dc.identifier.urihttps://dl.gi.de/handle/20.500.12116/21739
dc.language.isoen
dc.publisherGesellschaft für Informatik, Bonn
dc.relation.ispartofBTW 2019
dc.relation.ispartofseriesLecture Notes in Informatics (LNI) – Proceedings, Volume P-289
dc.subjectData Stream Processing
dc.subjectResource Elasticity
dc.subjectQuery Plan Maintenance
dc.subjectFault Tolerance
dc.titleOn-the-fly Reconfiguration of Query Plans for Stateful Stream Processing Enginesen
gi.citation.endPage146
gi.citation.startPage127
gi.conference.date4.-8. März 2019
gi.conference.locationRostock
gi.conference.sessiontitleWissenschaftliche Beiträge

Dateien

Originalbündel
1 - 1 von 1
Vorschaubild
Name:
B2-5.pdf
Größe:
1.75 MB
Format:
Adobe Portable Document Format
Herunterladen

Sammlungen

P289 - BTW2019 - Datenbanksysteme für Business, Technologie und Web