Leverage camel-k to materialize syndesis-meta

[lburgazzoli]

This is a...

[X] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Documentation issue or request

Description

The syndesis-meta service is s spring-boot based application (fat-jar) that provides metadata validation, retrieval and manipulation facilities for all the connectors known to syndesis at build time.

This closed world assumption as some limitations:

• extension developer, can't provide functionalities such as parameter validations as there's no way to have component/action extensions taken into account by syndesis-meta
• libraries extensions are made available to the syndesis-meta through a persistent volume shared between the syndesis-server and syndesis-meta PODs which adds additional maintenance costs and breaks immutability
• the more connector will be developed and added to syndesis the higher the chance to have incompatible libraries on the classpath will be
• the service can't take advantage as platform such as Knative and scale to zero when there's no requests because of a long startup time

As syndesis is heading to use camel-k as engine to deploy integrations, we can leverage it also to materialize the syndesis-meta service which would provide the following benefits:

• meta-service can be re-built when the connector are added through the extension mechanism or when there is the need to provide additional dependencies such as jdbc driver
• meta-service can be deployed as a monolith as it is today but on Knative the meta-servicew can be split into multiple service each one providing metadata facilities for a single connector or a subset of the known ones

This of course does not come for free as there some related changes:

• store extensions in a s3 compatible object storage( instead of a database as it is today) with a structure compatible with maven repositories so later camel-k can retrieve it as any other dependency Let dependencies be defined using file URLs apache/camel-k#1227 (detecting it is an extension is a camel-k enhancement support for dependencies archive apache/camel-k#334)
• enhance the syndesis-operator to generate camel-k integration to properly materialize syndesis-meta service(s)
• enhance the syndesis-server to generate a CRs the syndesis-operator can leverage to materialize syndesis-meta service(s)
  
• extension can be deployed straight to the s3 storage and discovered by syndesis via CRs