DSMM: A Dynamic Setting for Memory Management in Apache Spark - Chae, Suk Joo; Chung, Tae Sun (Proceedings - 2019 IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2019, 2019)
Apache Spark (Spark) is a unified analytics engine for large-scale data processing. Unlike traditional data processing engines like Hadoop, Spark is a framework that caches data in memory. Therefore, memory management in Spark is importance. However, there are several factors that interfere with memory management. First, if users want to cache data in memory, they need to choose their own storage level. In this case, if they do not select the optimal storage level, Spark will be put a heavy burden on memory. Next, users need to select the ratio for spark memory directly within Spark. If they do not choose optimal ratio for spark memory, garbage collection overheads will be incurred. In this poster, we propose DSMM that dynamically select the above factors on the system for memory management. Our experimental result shows 13% execution time improvement as compared to standard Spark.
Keywords: Cache data; Dynamic settings; Garbage collection; Large-scale data processing; Memory management; Processing engine; Storage level; Apache Spark; Configuration Setting; Spark memory; Storage Levels