Show simple item record

AuthorZhang, Fan
AuthorMalluhi, Qutaibah M.
AuthorElsayed, Tamer
AuthorKhan, Samee U.
AuthorLi, Keqin
AuthorZomaya, Albert Y.
Available date2015-12-29T12:54:37Z
Publication Date2015-10
Publication NameFuture Generation Computer Systems
CitationZhang F., Malluhi Q.M., Elsayed T., Khan S.U., Li K., Zomaya A.Y., CloudFlow: A data-aware programming model for cloud workflow applications on modern HPC systems, (2015) Future Generation Computer Systems, 51, pp. 98-110.
AbstractTraditional High-Performance Computing (HPC) based big-data applications are usually constrained by having to move large amount of data to compute facilities for real-time processing purpose. Modern HPC systems, represented by High-Throughput Computing (HTC) and Many-Task Computing (MTC) platforms, on the other hand, intend to achieve the long-held dream of moving compute to data instead. This kind of data-aware scheduling, typically represented by Hadoop MapReduce, has been successfully implemented in its Map Phase, whereby each Map Task is sent out to the compute node where the corresponding input data chunk is located. However, Hadoop MapReduce limits itself to a one-map-to-one-reduce framework, leading to difficulties for handling complex logics, such as pipelines or workflows. Meanwhile, it lacks built-in support and optimization when the input datasets are shared among multiple applications and/or jobs. The performance can be improved significantly when the knowledge of the shared and frequently accessed data is taken into scheduling decisions. To enhance the capability of managing workflow in modern HPC system, this paper presents CloudFlow, a Hadoop MapReduce based programming model for cloud workflow applications. CloudFlow is built on top of MapReduce, which is proposed not only being data aware, but also shared-data aware. It identifies the most frequently shared data, from both task-level and job-level, replicates them to each compute node for data locality purposes. It also supports user-defined multiple Map- and Reduce functions, allowing users to orchestrate the required data-flow logic. Mathematically, we prove the correctness of the whole scheduling framework by performing theoretical analysis. Further more, experimental evaluation also shows that the execution runtime speedup exceeds 4X compared to traditional MapReduce implementation with a manageable time overhead.
SponsorNPRP grant # 09-1116-1-172 from the Qatar National Research Fund (a member of Qatar Foundation). Ministry of Science and Technology of China under National 973 Basic Research Program (Grant No. 2013CB228206), National Natural Science Foundation of China (Grant Nos. 61472200 and 61233016).
SubjectData aware
SubjectProgramming model
TitleCloudFlow: A data-aware programming model for cloud workflow applications on modern HPC systems
Volume Number51

Files in this item


There are no files associated with this item.

This item appears in the following Collection(s)

Show simple item record