Abstract
The development of the Edge computing paradigm shifts data processing from centralised infrastructures to heterogeneous and geographically distributed infrastructure. Such a paradigm requires
data processing solutions that consider data locality in order to reduce the performance penalties from data transfers between remote
(in network terms) data centres. However, existing Big Data processing solutions have limited support for handling data locality and
are inefficient in processing small and frequent events specific to
Edge environments. This paper proposes a novel architecture and a
proof-of-concept implementation for software container-centric Big
Data workflow orchestration that puts data locality at the forefront.
Our solution considers any available data locality information by
default, leverages long-lived containers to execute workflow steps,
and handles the interaction with different data sources through
containers. We compare our system with Argo workflow and show
significant performance improvements in terms of speed of execution for processing units of data using our data locality aware Big
Data workflow approach.
data processing solutions that consider data locality in order to reduce the performance penalties from data transfers between remote
(in network terms) data centres. However, existing Big Data processing solutions have limited support for handling data locality and
are inefficient in processing small and frequent events specific to
Edge environments. This paper proposes a novel architecture and a
proof-of-concept implementation for software container-centric Big
Data workflow orchestration that puts data locality at the forefront.
Our solution considers any available data locality information by
default, leverages long-lived containers to execute workflow steps,
and handles the interaction with different data sources through
containers. We compare our system with Argo workflow and show
significant performance improvements in terms of speed of execution for processing units of data using our data locality aware Big
Data workflow approach.