Transformation Support for ES 6.8 to OS 2.0

[sumobrian]

Is your feature request related to a problem?

The current transformation framework has several limitations. Customers face challenges applying transformations to tuples, RFS documents, and metadata, leading to incomplete or inconsistent migrations. Additionally, all transformations are applied to every request, making it difficult to debug complex transformation behaviors. Furthermore, there is no local developer workflow to easily test transformations, and the absence of comments in transformation specs hinders documentation.

What solution would you like?

We seek to extend the transformation framework by:

• Adding transformation support for Tuples, RFS documents, and Metadata Migrations.
• Incorporating JMESPath Predicates to selectively apply transformations.
• Allowing comments in transformation specs to improve readability and documentation.
• Providing local tools for easier testing of transformations, including cluster modification utilities.
• Creating a Python-based transformation system to complement the current Java implementation.
• Improving the developer iteration cycle for transformation creation and testing, potentially using GenAI.

What alternatives have you considered?

• Keeping the current system and providing only documentation and examples for more common transformations.
• Relying on manual post-migration processes to handle limitations, such as RFS documents and metadata transformations, instead of automating them within the framework.
• Using other templating languages or providing pre-built transformation libraries, such as Jinja or specialized JSON manipulation tools.

Do you have any additional context?

The transformation framework should be robust enough to handle all phases of the migration process, including tuples, RFS, and metadata. The limitations currently hinder seamless migrations, particularly for more complex use cases like data analysis or direct capture and replay of requests. Addressing these limitations will significantly reduce the manual overhead and improve migration efficiency for OpenSearch customers.

Child Issue:

Allow Renaming of an Index from Source to Target with Transformations
#1082