Integration with DAG framework Prefect

[tchaton]

🚀 Feature

Motivation

The library Prefect https://github.com/PrefectHQ/prefect is pretty great at building DAGs and orchestrating then.

In the future, LitData will enables to build a DAG of map, optimize, etc.. operators to perform large distributed processing.

The ideal goal would be the reverse engineer Prefect Orion Engine and monitoring and have an adapted version for the Lightning.AI platform.

Pitch

Alternatives

Additional context

[deependujha]

Orion engine is deprecated in Prefect - Release 2.13.0. So, I couldn't find much information.

And, I guess, we can already use litdata and prefect.

So, are we interested in Prefect workpools specific to lightning studio, create a deployment and run the workflows on lightning nodes (studio/jobs)?

Or, is it something else that we want?