Logo des Repositoriums

Auflistung nach Schlagwort "benchmark"

1 - 10 von 11
  • Konferenzbeitrag
    Benchmarking Function Hook Latency in Cloud-Native Environments
    (Softwaretechnik-Trends Band 43, Heft 4, 2023) Kahlhofer, Mario; Kern, Patrick; Henning, Sören; Rass, Stefan
    Researchers and engineers are increasingly adopting cloud-native technologies for application development and performance evaluation. While this has improved the reproducibility of benchmarks in the cloud, the complexity of cloud-native environments makes it difficult to run benchmarks reliably. Cloud-native applications are often instrumented or altered at runtime, by dynamically patching or hooking them, which introduces a significant performance overhead. Our work discusses the benchmarking-related pitfalls of the dominant cloud-native technology, Kubernetes, and how they affect performance measurements of dy namically patched or hooked applications. We present recommendations to mitigate these risks and demonstrate how an improper experimental setup can negatively impact latency measurements.
  • Konferenzbeitrag
    Benchmarking Stream Processing Frameworks for Large Scale Data Shuffling
    (Softwaretechnik-Trends Band 43, Heft 4, 2023) Henning, Sören; Vogel, Adriano; Leichtfried, Michael; Ertl, Otmar; Rabiser, Rick
    Distributed stream processing frameworks help building scalable and reliable applications that perform transformations and aggregations on continuous data streams. We outline our ongoing research on designing a new benchmark for distributed stream processing frameworks. In contrast to other benchmarks, it focuses on use cases where stream processing frameworks are mainly used for redistributing data records to perform state-local aggregations, while the actual aggregation logic is considered as black-box software components. We describe our benchmark architecture based on a real-world use case, show how we imple mented it with four state-of-the-art frameworks, and give an overview of initial experimental results.
  • Konferenzbeitrag
    Better a Microbenchmark on a Cluster than a User at the Office: Flink Cluster Benchmarking
    (Softwaretechnik-Trends Band 39, Heft 3, 2019) Reichelt, David Georg; Meyer, Lars-Peter; Kühne, Stefan
    When operating an Apache Flink cluster, performance problems may occur on all components of its setup. Reproducing those problems in different software or hardware components and on different nodes requires systematic experiments. We present an Apache Flink cluster benchmark set for server operators which is able to measure the performance of an Apache Flink cluster. This enables spotlighting irregularities in software or hardware behaviour.
  • Konferenzbeitrag
    Better Feedback Times Using Test Case Prioritization? Mining Data of Past Build Failures in an Automated Benchmark
    (Softwaretechnik-Trends Band 40, Heft 2, 2020) Rott, Jakob; Niedermayr, Rainer; Jürgens, Elmar
    In software projects with growing functionality, the number of tests increases fast which results in long execution times for the whole test suite. As a consequence, it is not possible to always execute the whole test suite after each commit so that feedback time to developers increases. With long test feedback times, the effort for an early fix rises and developers can be hindered in productive work. One solution to reduce feedback times is test case prioritization. Although test prioritization strategies have been extensively studied, they are rarely used in practice and their benefits are widely unknown. In this paper, we present a benchmark framework to evaluate the benefits of different test prioritization algorithms on open source projects and primarily use the time until the first failure (TUFF) as relevant metric. We conducted an empirical study with 31 open-source projects hosted on GitHub, using data of 437 builds that failed on the CI server. In 75% of the builds, the first test will fail within the first 18% of the total test suite’s duration.
  • Konferenzbeitrag
    Comparing the Performance of Data Processing Implementations
    (Softwaretechnik-Trends Band 43, Heft 4, 2023) Beierlieb, Lukas; Iffländer, Lukas; Prantl, Thomas; Kounev, Samuel
    This paper compares the execution speed of R, Python, and Rust implementations in the context of data processing. A real-world data processing task in the form of an aggregation of benchmark measure ment results was implemented in each language, and the execution times were measured. Rust and Python showed significantly superior performance compared to the R implementation. Further, we compared the results of different Python interpreters (the most recent versions of CPython and PyPy), also resulting in measurable variations. Finally, a study of the effectiveness of multithreading was performed.
  • Zeitschriftenartikel
    Cooperative Android App Analysis
    (Softwaretechnik-Trends Band 44, Heft 2, 2024) Pauck, Felix
    In this summary, the three main contributions of the thesis ”Cooperative Android App Analysis” are presented. The first contribution proposes the cooperative analysis approach. The centerpiece of this approach is the AQL (Android App Analysis Query Language) – a domain specific query language. It allows formulating (AQL-)queries in order to interact with arbitrary analysis tools. As counterpart AQL-Answer come into play, which are able to universally but well structured embody any kind of analysis result. The second contribution uses the AQL to define reproducible benchmarks that can be used to automatically evaluate analysis tools on such. Various benchmarks are then used in the third contribution to conduct a thorough evaluation of 13 Android taint analysis tools. Please note, in the context of the thesis, the cooperative analysis implementation is tailored to Android taint analysis, however, the concept can be applied to any kind of analysis.
  • Konferenzbeitrag
    Experiences from Building the Open Database Performance Ranking with benchANT
    (Softwaretechnik-Trends Band 43, Heft 1, 2023) Seybold, Daniel; Domaschka, Jörg
    Benchmarking is an important method to advance database management systems (DBMS) from the industry and research perspective. Ensuring transparent and reproducible results is a key requirement to ensure the acceptance and credibility of benchmarking. To advance the research towards transparent and reproducible benchmark data, we report on building an open DBMS performance ranking with 130 benchmark configurations and ensuring comparability, transparency and reproducibility. We derive the required data on cloud, resource, DBMS and benchmark level to enable transparency and reproducibility and demonstrate the generation of such data sets with benchANT. Building upon such data, we outline future research directions for DBMS performance modelling, DBMS auto-tuning and decision support.
  • Konferenzbeitrag
    On the Validity of Performance Antipatterns at Code Level
    (Softwaretechnik-Trends Band 39, Heft 4, 2019) Reichelt, David Georg; Kühne, Stefan; Hasselbring, Wilhelm
    Performance antipatterns at code level should be avoided to assure good application performance. Performance antipatterns avoidance is hard, since it requires up-to-date knowledge of these antipatterns. Common lists of antipatterns, like the performance rules of the static code checker PMD, only contain limited information about versions and circumstances where the performance antipatterns are valid. We close this gap by prodiving a suite of 30 performance benchmarks. Each of this benchmarks checks whether the performance antipattern is measurable in Java 6, 7, 8, 11 and 12. We find that two of the 30 performance checks are not valid in the current OpenJDK 12.
  • Zeitschriftenartikel
    Scalability Benchmarking of Cloud-Native Applications: The Theodolite Approach
    (Softwaretechnik-Trends Band 44, Heft 2, 2024) Henning, Sören; Hasselbring, Wilhelm
    Scalability is a driving requirement for many software systems, especially for those designed as cloud-native applications and event-driven microservices. Empirically evaluating and comparing the scalability of such applications is therefore of great interest for software engineers, architects, and researchers. In this summary, we outline the thesis Scalability Benchmarking of Cloud-Native Applications Applied to Event Driven Microservices, which presents and evaluates Theodolite, our scalability benchmarking method for cloud-native applications along with specific scalability benchmarks for event-driven microservices.
  • Konferenzbeitrag
    Toward Efficient Scalability Benchmarking of Event-Driven Microservice Architectures at Large Scale
    (Softwaretechnik-Trends Band 40, Heft 3, 2020) Henning, Sören; Hasselbring, Wilhelm
    Over the past years, an increase in software architectures containing microservices, which process data streams of a messaging system, can be observed. We present Theodolite, a method accompanied by an open source implementation for benchmarking the scalability of such microservices as well as their employed stream processing frameworks and deployment options. According to common scalability definitions, Theodolite provides detailed insights into how resource demands evolve with increasing load intensity. However, accurate and statistically rigorous insights come at the cost of long execution times, making it impracticable to execute benchmarks for large sets of systems under test. To overcome this limitation, we raise three research questions and propose a research agenda for executing scalability benchmarks more time-efficiently and, thus, for running scalability benchmarks at large scale.