|
DataCloud Toolbox |
The DataCloud Toolbox provides 6 main tools (DIS-PIPE, DEF-PIPE, SIM-PIPE, ADA-PIPE, R-MARKET and DEP-PIPE) as shown in the figure below.
Each tool has its own GitHub repository and may consist of different tool components having their separate component repository on GitHub.
DIS-PIPE (discovery pipeline tool)
DIS-PIPE provides scalable integration of process mining techniques and artificial intelligence algorithms to learn the structure of Big Data pipelines by extracting, processing and interpreting vast amounts of event data collected from several data sources. Furthermore, DIS-PIPE supports a variety of analytics techniques for visualising the discovered pipelines together with detailed diagnostics information about their execution.
Provides scalable integration of process mining techniques and AI algorithms to learn the structure of Big Data pipelines.
DEF-PIPE (definition pipeline tool)
DEF-PIPE provides a visual design for domain experts to implement Big Data pipelines based on a DSL, including means to store and load the pipeline definitions. Furthermore, it enables data scientists to define the pipelines by configuring each step, injecting code, or customising predefined generic templates.
Graphical tool for designing Big Data pipelines according to the DSL.
Components
Domain-Specific Language (DSL) for defining Big Data pipeplines.
SIM-PIPE (simulation pipeline tool)
SIM-PIPE simulates the pipeline execution and provides final deployment configurations that conform to the hardware requirements. SIM-PIPE also provides testing functionalities, such as a sandbox for evaluating the performance of individual pipeline steps and statistical analysis of the performance of the overall pipeline.
Main repository that provides deployment instructions for the SIM-PIPE tool and its sub-components.
ADA-PIPE (adaption pipeline tool)
ADA-PIPE provides a data-aware and adaptive scheduling algorithm for allocating the data pipeline step to the Computing Continuum with infrastructure drift adaptation capability. ADA-PIPE allows dynamic resource reconfiguration for improved performance and SLO fulfillment.
Provides a data-aware algorithm for smart and adaptable provisioning of resources and services across the Computing Continuum.
R-MARKET (resource marketplace tool)
R-MARKET deploys a decentralised hybrid permissioned and permissionless blockchain network that federates a vast set of heterogeneous resources from various providers spread across the Computing Continuum. R-MARKET creates a democratic marketplace of trustworthy resources and enables transparent provisioning over multiple control and network domains for external use.
Deploys a decentralized backbone resource network.
Components
UI app for the R-MARKET resource marketplace.
It is a CLI and a JS library that allows easy interactions with the decentralized Marketplace.
Entrypoint of the R-MARKET tool. The Node.js server provides interfaces which enables the connection between the R-MARKET tool and other pipeline tools, also it helps users to access the R-MARKET tool.
Allow offchain order sharing.
R-MARKET Scheduler is responsible for organizing the work distribution and order publication of a worker pool.
Worker(s) is (are) in charge of running computing tasks, send by the requesters through the R-MARKET.
Along with R-MARKET Worker and Scheduler, this component forms the R-MARKET tool's middleware.
Explorer is a web application which allows to explore a chain.
Provides the blockchain-based decentralised marketplace for the resources.
DEP-PIPE (deployment pipeline tool)
DEP-PIPE enables flexible and scalable deployment and orchestration of Big Data pipelines over the Computing Continuum resources. DEP-PIPE monitors the pipeline execution and provides online SLO metrics to the other tools.
Main repository with instructions for the setup of DEP-PIPE.
Components
Main conponent of DEP-PIPE, being the mediator between DataCloud tools ( ADA-PIPE, DEF-PIPE, R-MARKET) and the MAESTRO orchestator
Tools and libraries for data monitoring and data drift.
Tools and libraries for protecting the data pipelines.