Abstract
Big data pipelines are developed to process data characterized by one or more of the
three big data features, commonly known as the three Vs (volume, velocity, and variety), through a
series of steps (e.g., extract, transform, and move), making the ground work for the use of advanced
analytics and ML/AI techniques. Computing continuum (i.e., cloud/fog/edge) allows access to
virtually infinite amount of resources, where data pipelines could be executed at scale; however, the
implementation of data pipelines on the continuum is a complex task that needs to take computing
resources, data transmission channels, triggers, data transfer methods, integration of message queues,
etc., into account. The task becomes even more challenging when data storage is considered as part
of the data pipelines. Local storage is expensive, hard to maintain, and comes with several challenges
(e.g., data availability, data security, and backup). The use of cloud storage, i.e., storage-as-a-service
(StaaS), instead of local storage has the potential of providing more flexibility in terms of scalability,
fault tolerance, and availability. In this article, we propose a generic approach to integrate StaaS with
data pipelines, i.e., computation on an on-premise server or on a specific cloud, but integration with
StaaS, and develop a ranking method for available storage options based on five key parameters:
cost, proximity, network performance, server-side encryption, and user weights/preferences. The
evaluation carried out demonstrates the effectiveness of the proposed approach in terms of data
transfer performance, utility of the individual parameters, and feasibility of dynamic selection of a
storage option based on four primary user scenarios.
three big data features, commonly known as the three Vs (volume, velocity, and variety), through a
series of steps (e.g., extract, transform, and move), making the ground work for the use of advanced
analytics and ML/AI techniques. Computing continuum (i.e., cloud/fog/edge) allows access to
virtually infinite amount of resources, where data pipelines could be executed at scale; however, the
implementation of data pipelines on the continuum is a complex task that needs to take computing
resources, data transmission channels, triggers, data transfer methods, integration of message queues,
etc., into account. The task becomes even more challenging when data storage is considered as part
of the data pipelines. Local storage is expensive, hard to maintain, and comes with several challenges
(e.g., data availability, data security, and backup). The use of cloud storage, i.e., storage-as-a-service
(StaaS), instead of local storage has the potential of providing more flexibility in terms of scalability,
fault tolerance, and availability. In this article, we propose a generic approach to integrate StaaS with
data pipelines, i.e., computation on an on-premise server or on a specific cloud, but integration with
StaaS, and develop a ranking method for available storage options based on five key parameters:
cost, proximity, network performance, server-side encryption, and user weights/preferences. The
evaluation carried out demonstrates the effectiveness of the proposed approach in terms of data
transfer performance, utility of the individual parameters, and feasibility of dynamic selection of a
storage option based on four primary user scenarios.