Development of Experimental Data Processing Workflows Based on Kubernetes Infrastructure and REANA Workflow Management Systemстатья
Информация о цитировании статьи получена из
Scopus
Статья опубликована в журнале из списка Web of Science и/или Scopus
Дата последнего поиска статьи во внешних источниках: 15 февраля 2024 г.
Аннотация:In this paper we present the design of data processing workflow for scientific experiments, which require complicated multi-step analysis procedure. We test it on datasets from Single Particle Imaging (SPI) experiments. The workflow is based on microservice architecture, Docker containers and Kubernetes platform. For workflow setup and management we use REANA software which is compatible with Kubernetes ochestrator and supports standard Common Workflow Language (CWL) to describe complex computing jobs. Our approach allows easy construction of workflows of diverse architecture for a wide range of applications. It allows integration of heterogeneous software in a uniform way as well as easy modification or replacement of workflow components. In the same time it allows easy scaling of computations in a cloud infrastructure. We show the applicability of the designed scheme and estimate the overhead of the platform middleware.