GI LogoGI Logo
  • Login
Digital Library
    • All of DSpace

      • Communities & Collections
      • Titles
      • Authors
      • By Issue Date
      • Subjects
    • This Collection

      • Titles
      • Authors
      • By Issue Date
      • Subjects
Digital Library Gesellschaft für Informatik e.V.
GI-DL
    • English
    • Deutsch
  • English 
    • English
    • Deutsch
View Item 
  •   DSpace Home
  • Lecture Notes in Informatics
  • Proceedings
  • BTW - Datenbanksysteme für Business, Technologie und Web
  • P289 - BTW2019 - Datenbanksysteme für Business, Technologie und Web
  • View Item
JavaScript is disabled for your browser. Some features of this site may not work without it.
  •   DSpace Home
  • Lecture Notes in Informatics
  • Proceedings
  • BTW - Datenbanksysteme für Business, Technologie und Web
  • P289 - BTW2019 - Datenbanksysteme für Business, Technologie und Web
  • View Item

On-the-fly Reconfiguration of Query Plans for Stateful Stream Processing Engines

Author:
Bartnik, Adrian [DBLP] ;
Del Monte, Bonaventura [DBLP] ;
Rabl, Tilmann [DBLP] ;
Markl, Volker [DBLP]
Abstract
Stream Processing Engines (SPEs) must tolerate the dynamic nature of unbounded data streams and provide means to quickly adapt to fluctuations in the data rate. Many major SPEs however provide very little functionality to adjust the execution of a potentially infinite streaming query at runtime. Each modification requires a complete query restart, which involves an expensive redistribution of the state of a query and may require external systems in order to guarantee correct processing semantics. This results in significant downtime, which increase the operational cost of those SPEs. We present a modification protocol that enables modifying specific operators as well as the data flow of a running query while ensuring exactly-once processing semantics. We provide an implementation for Apache Flink, which enables stateful operator migration across machines, the introduction of new operators into a running query, and changes to a specific operator based on external triggers. Our results on two benchmarks show that migrating operators for queries with small state is as fast as using the savepoint mechanism of Flink. Migrating operators in the presence of large state even outperforms the savepoint mechanism by a factor of more than 2.3. Introducing and replacing operators at runtime is performed in less than 10 s. Our modification protocol demonstrates the general feasibility of runtime modifications and opens the door for many other modification use cases, such as online algorithm tweaking and up-or downscaling operator instances.
  • Citation
  • BibTeX
Bartnik, A., Del Monte, B., Rabl, T. & Markl, V., (2019). On-the-fly Reconfiguration of Query Plans for Stateful Stream Processing Engines. In: Grust, T., Naumann, F., Böhm, A., Lehner, W., Härder, T., Rahm, E., Heuer, A., Klettke, M. & Meyer, H. (Hrsg.), BTW 2019. Gesellschaft für Informatik, Bonn. (S. 127-146). DOI: 10.18420/btw2019-09
@inproceedings{mci/Bartnik2019,
author = {Bartnik, Adrian AND Del Monte, Bonaventura AND Rabl, Tilmann AND Markl, Volker},
title = {On-the-fly Reconfiguration of Query Plans for Stateful Stream Processing Engines},
booktitle = {BTW 2019},
year = {2019},
editor = {Grust, Torsten AND Naumann, Felix AND Böhm, Alexander AND Lehner, Wolfgang AND Härder, Theo AND Rahm, Erhard AND Heuer, Andreas AND Klettke, Meike AND Meyer, Holger} ,
pages = { 127-146 } ,
doi = { 10.18420/btw2019-09 },
publisher = {Gesellschaft für Informatik, Bonn},
address = {}
}
DateienGroesseFormatAnzeige
B2-5.pdf1.745Mb PDF View/Open

Sollte hier kein Volltext (PDF) verlinkt sein, dann kann es sein, dass dieser aus verschiedenen Gruenden (z.B. Lizenzen oder Copyright) nur in einer anderen Digital Library verfuegbar ist. Versuchen Sie in diesem Fall einen Zugriff ueber die verlinkte DOI: 10.18420/btw2019-09

Haben Sie fehlerhafte Angaben entdeckt? Sagen Sie uns Bescheid: Send Feedback

More Info

DOI: 10.18420/btw2019-09
ISBN: 978-3-88579-683-1
ISSN: 1617-5468
xmlui.MetaDataDisplay.field.date: 2019
Language: en (en)

Keywords

  • Data Stream Processing
  • Resource Elasticity
  • Query Plan Maintenance
  • Fault Tolerance
Collections
  • P289 - BTW2019 - Datenbanksysteme für Business, Technologie und Web [47]

Show full item record


About uns | FAQ | Help | Imprint | Datenschutz

Gesellschaft für Informatik e.V. (GI), Kontakt: Geschäftsstelle der GI
Diese Digital Library basiert auf DSpace.

 

 


About uns | FAQ | Help | Imprint | Datenschutz

Gesellschaft für Informatik e.V. (GI), Kontakt: Geschäftsstelle der GI
Diese Digital Library basiert auf DSpace.