Ingestion Module/App

[MANTENN]

Is your feature request related to a problem? Please describe.
Importing a large datasets means you need a queue on the plugin end of the module and this would involve duplicate work across a variety of platforms: Strapi, WordPress, Drupal, etc. Instead a "global" app that would connect with databases and import documents from the database with SQL introspection when ingestion is necessary. In addition, can be added later, this UI can edit the UI sort of how ElasticSearch has, facet setup, etc.

This would: centralize the work of an ingestion pipeline and simplify development for multiple platforms; and offload work from the backend of an api onto an ingestion engine/pipeline. I am aware MeiliSearch queues records when a large dataset is imported through the API.

This is a continuation of issue 88 on meilisearch/strapi-plugin-meilisearch#88 (comment).
Describe the solution you'd like
There could be an instance of an ingestion engine running alongside MeiliSearch that would handle imports.

Describe alternatives you've considered
Creating an ingestion pipeline for all platforms(multiple codebases to manage, multiple apis, etc.) and with an ingestion engine you get a single source of truth.

Additional context
N/A

[curquiza]

Hello @MANTENN!

Thanks for the suggestion, this is not planned for the moment, but this would be a great feature to add in the future 🙂
Please, if you have time could you open a card in our roadmap? https://roadmap.meilisearch.com/tabs/5-ideas
It would be really helpful!