[Enhancement]: Merge RootCoord, DataCoord And QueryCoord into one component

[jaime0815]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

1. Eliminate RPC among coordinator nodes
2. Implement a unified meta cache manager
3. Establish a unified task scheduler manager for coordinators

Why is this needed?

• Simplify the Milvus architecture by reducing component count for a lighter design.
• Eliminate redundant metadata manager implementations between coordinators to facilitate easier maintenance.
• Decrease RPC usage in each coordinator.
  • Address corner cases where some metadata operations lack transaction guarantees.
  • Enhance the efficiency of metadata operations.

Anything else?

No response

[jaime0815]

related to #37757

[xiaofan-luan]

what's the target release for this feature?
should we target it at 3.0?

[chyezh]

what's the target release for this feature? should we target it at 3.0?

We will achieve the enhancement with following step.
The step one will merge in 2.5 openning in standalone or mixcoord without break change.
Other break changes will merge in 3.0.
After step one, we can design new feature without considering distribution issues between coords.

• Remove the rpc layer of the communication between one coordinator to another. enhance: remove the rpc layer of coordinator when enabling standalone or mixcoord #37815
• Remove the cache layer of one coordinator.
• Make Session into one.
• Refactor the recovery and startup logic of whole coordinator.

[jaime0815]

what's the target release for this feature? should we target it at 3.0?

The task will progress through three stages:

• In version 2.5.x, we will eliminate the RPC layer among coordinators, allowing mixcoord deployment to rely solely on internal process communication, without affecting rolling upgrades.
• In version 3.0, only one coordinator component will be permitted to start, requiring the cluster to be stopped before upgrading from 2.x.x.
• In version 3.0.x, we will merge and streamline the coordinators, including the meta manager and task scheduler, into a single unit, while maintaining compatibility for rolling upgrades from 3.0 to 3.x.

Additionally, the next task will focus on enhancing support for streaming queries in streaming services, targeting a release with version 3.0.