[Enhancement]:When migrating, the metadata IDs are automatically generated. However, it is expected that in the new data, instead of using automatically generated IDs, the old metadata IDs should be retained.

[mmahao]

What would you like to be added?

When migrating, the metadata IDs are automatically generated. However, it is expected that in the new data, instead of using automatically generated IDs, the old metadata IDs should be retained.

Why is this needed?

We are currently using Milvus version 2.2.16 and want to upgrade to 2.4.8. I have strongly associated the primary key ID of Milvus with MySQL, but when using the migration tool, I noticed that the IDs are not the same as the old metadata primary key IDs, which could cause problems. The current plan is to set up a new 2.2.16 instance, migrate the data from the current 2.2.16 instance to the new one using the migration tool, and then follow the steps in the official documentation (‘https://milvus.io/docs/upgrade_milvus_standalone-operator.md#Conduct-a-rolling-upgrade’) to perform a rolling upgrade. This approach aims to ensure a smooth upgrade from 2.2.16 to 2.4.8 in the production environment.

Anything else?

I hope the indexes and descriptions can be synchronized as well.

[mmahao]

Also, it would be great if a batch migration tool could be implemented.

[wenhuiZilliz]

@mmahao

1. about ID(primary key) , when your target Milvus AutoId property is open, then source Milvus collection's ID will not be target collection ID, it's determined by Milvus' characteristics. If your target collection not exists then the migration tool will create it and the collection structure same with source collection, so if source collection open AutoId feature then target collection as well. Now the fastest solution is manually create target collection and close AutoID feature. (if target collection already exists migration tool will skip create it) that will let source collection ID as the target collection's ID.

2. are you means is want migration multi-collections(batch migration) ? if yes, you can write a bash script like below: (Loop passes the collection to be migrated to parameter - t)

#!/bin/bash

collections=("collection1" "collection2" "collection3")

for collection in "${collections[@]}"; do
echo "BatchMigration==> $collection"
./milvus-migration start -t="$collection" -c=/{YourConfigPath}/migration.yml
done

https://github.com/zilliztech/milvus-migration/blob/main/testfiles/milvus2x/batch_collection_migration.sh

[wenhuiZilliz]

@mmahao use latest code it supported migration collection description, but we don't want support create index (create index is relatively independent time-consuming operations, hope to do it alone )

[xiaofan-luan]

@wenhuiZilliz how long does it take to use VTS to support this?

[wenhuiZilliz]

@xiaofan-luan VTS internal migration tool currently supports batch collections and whether AutoId features are enabled

[breadmakerQQQ]

Same situation occurred... When migrating, the old data's ids are unretained, however, if I handly create a new collection and make it auto_id disabled as suggested, the new following data cannot be insert since I didn't specify a id by myself in program... And it seems Milvus does not support modification on schema include auto_id property, so if I want to make a collection auto_id enabled from disabled, I need to migrate again from a none auto_id collection to auto_id one, which make ids unretained still, what a mess...

[wenhuiZilliz]

@breadmakerQQQ however I don't know your scene，but if you want source collection(enable autoId) to target collection(enable autoId), and copy the ids of the existing source collection to the target that this tool cannot achieve it(that need target collection disable autoId), but you can use our another tool the milvus-backup that use backup/restore data ability to achieve your requirement. see: https://github.com/zilliztech/milvus-backup