Ajou University repository

DSMM: A Dynamic Setting for Memory Management in Apache Spark
Citations

SCOPUS

0

Citation Export

Publication Year
2019-04-22
Journal
Proceedings - 2019 IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2019
Publisher
Institute of Electrical and Electronics Engineers Inc.
Citation
Proceedings - 2019 IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2019, pp.143-144
Keyword
Apache SparkConfiguration SettingSpark memoryStorage Levels
Mesh Keyword
Cache dataDynamic settingsGarbage collectionLarge-scale data processingMemory managementProcessing engineStorage level
All Science Classification Codes (ASJC)
Hardware and ArchitectureSoftwareSafety, Risk, Reliability and Quality
Abstract
Apache Spark (Spark) is a unified analytics engine for large-scale data processing. Unlike traditional data processing engines like Hadoop, Spark is a framework that caches data in memory. Therefore, memory management in Spark is importance. However, there are several factors that interfere with memory management. First, if users want to cache data in memory, they need to choose their own storage level. In this case, if they do not select the optimal storage level, Spark will be put a heavy burden on memory. Next, users need to select the ratio for spark memory directly within Spark. If they do not choose optimal ratio for spark memory, garbage collection overheads will be incurred. In this poster, we propose DSMM that dynamically select the above factors on the system for memory management. Our experimental result shows 13% execution time improvement as compared to standard Spark.
Language
eng
URI
https://aurora.ajou.ac.kr/handle/2018.oak/36459
https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85065394150&origin=inward
DOI
https://doi.org/10.1109/ispass.2019.00024
Journal URL
http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=8686044
Type
Conference
Show full item record

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

Related Researcher

Chung, Tae-Sun Image
Chung, Tae-Sun정태선
Department of Software and Computer Engineering
Read More

Total Views & Downloads

File Download

  • There are no files associated with this item.