ePrints

Orchestrating big data analysis workflows in the cloud: Research challenges, survey, and future directions

Lookup NU author(s): Professor Aad van Moorsel, Professor Raj Ranjan

Downloads

Full text for this publication is not currently held within this repository. Alternative links are provided below where available.

Abstract

© 2019 Association for Computing Machinery.Interest in processing big data has increased rapidly to gain insights that can transform businesses, government policies, and research outcomes. This has led to advancement in communication, programming, and processing technologies, including cloud computing services and technologies such as Hadoop, Spark, and Storm. This trend also affects the needs of analytical applications, which are no longer monolithic but composed of several individual analytical steps running in the form of a workflow. These big data workflows are vastly different in nature from traditional workflows. Researchers are currently facing the challenge of how to orchestrate and manage the execution of such workflows. In this article, we discuss in detail orchestration requirements of these workflows as well as the challenges in achieving these requirements. We also survey current trends and research that supports orchestration of big data workflows and identify open research challenges to guide future developments in this area.

Publication metadata

Author(s): Barika M, Garg S, Zomaya AY, Wang L, Moorsel AVAN, Ranjan R

Publication type: Article

Publication status: Published

Journal: ACM Computing Surveys

Year: 2019

Volume: 52

Issue: 5

Pages: ePub ahead of print

Online publication date: 01/09/2019

Acceptance date: 01/05/2019

ISSN (print): 0360-0300

ISSN (electronic): 1557-7341

Publisher: Association for Computing Machinery

URL: https://doi.org/10.1145/3332301

DOI: 10.1145/3332301

Altmetrics

Actions

Share