Towards Measuring and Understanding Performance in Infrastructure- and Function-as-a-Service Clouds


Context. Cloud computing has become the de facto standard for deploying modern software systems, which makes its performance crucial to the efficient functioning of many applications. However, the unabated growth of established cloud services, such as Infrastructure-as-a-Service (IaaS), and the emergence of new services, such as Function-as-a-Service (FaaS), has led to an unprecedented diversity of cloud services with different performance characteristics.
Objective. The goal of this licentiate thesis is to measure and understand performance in IaaS and FaaS clouds. My PhD thesis will extend and leverage this understanding to propose solutions for building performance-optimized FaaS cloud applications.
Method. To achieve this goal, quantitative and qualitative research methods are used, including experimental research, artifact analysis, and literature review.
Findings. The thesis proposes a cloud benchmarking methodology to estimate application performance in IaaS clouds, characterizes typical FaaS applications, identifies gaps in literature on FaaS performance evaluations, and examines the reproducibility of reported FaaS performance experiments. The evaluation of the benchmarking methodology yielded promising results for benchmark-based application performance estimation under selected conditions. Characterizing 89 FaaS applications revealed that they are most commonly used for short-running tasks with low data volume and bursty workloads. The review of 112 FaaS performance studies from academic and industrial sources found a strong focus on a single cloud platform using artificial micro-benchmarks and discovered that the majority of studies do not follow reproducibility principles on cloud experimentation.
Future Work. Future work will propose a suite of application performance benchmarks for FaaS, which is instrumental for evaluating candidate solutions towards building performance-optimized FaaS applications.

Chalmers University of Technology

Thesis Publication

This licentiate thesis was published by the Chalmers University of Technology at Chalmers Research.

Note: The published PDF only contains the synopsis. The full-texts of the included papers are linked below and at Chalmers Research. Feel free to contact me for a personal copy of the full thesis containing all papers. Unfortunately, I cannot publish the full thesis due to potential copyright issues.

Included Papers

[𝛂] A Cloud Benchmark Suite Combining Micro and Applications Benchmarks

[𝛃] Estimating Cloud Application Performance Based on Micro-Benchmark Profiling

[𝛄] Serverless Applications: Why, When, and How?

[𝛅] Function-as-a-Service Performance Evaluation: A Multivocal Literature Review

[𝛆] Transpiling Applications into Optimized Serverless Orchestrations