Delete is not working, Delete entities not working

[geminigeek]

Delete is not working, Delete entities not working, on flip side all these functions works in python api, i am using a custom primary int id , it seems this can be the issue, my current total records are around 65K in the collection

Steps to reproduce:

1. add a custom primary key
2. add records try deleting from node response is success but no records are deleted

Milvus-node-sdk version:
2.2.20
Milvus version:
2.2.11

[shanghaikid]

Hi there,

Since attu is using the same node sdk, I tested deletion on attu, it works well.

One question: What's the consistency level of your collection？

[geminigeek]

hi,

yes attu was doing deletes on my test data, when i started to add real data and it became around 65k , i figured some meta fields were added as "undefined" , when i tried to delete the records with data as undefined ,records were not deleting , i realized that these records were around 18K , with python i could delete 8k in one go , by finding the ids first then deleting , but i guess as the field value was "undefined" JavaScript might be interpreting it differently while using delete entities?, i will test further and let you know , thanks for your reply.

consistency level is default i am just starting to use milvus!

[sadwsh]

Hi,

I've started using Milvus recently and stumbled upon the same issue that deleting entities don't seem to work. After some research I also tried a recommended approach of creating the collection with a consistency level of Strong, but it still didn't solve the issue.
All other operations (i.e. creating collections, querying data) could be successfully carried out except for delete so far.
I tried both the delete as well as the deleteEntities method as described in the Milvus docs but neither one yielded a result.
In fact the operation runs without errors and I also get an appropriate response including the delete_cnt which is set to the number of entities that should have been removed. But upon querying the collection after some time, the entities still exist.
Am I missing some configurations?

Steps to reproduce:

1. create a collection
2. add data (using LangChain JS and OpenAI Embeddings API)
3. query data and obtain primaryIds
4. delete entities for given primaryIds (succeeds without error)
5. query again and same data is displayed as if they weren't deleted

Milvus Node SDK version: 2.3.1
Milvus version (standalone docker): 2.3.0

[sadwsh]

@shanghaikid could you please explain this behavior?

[shanghaikid]

Could you try this file https://github.com/milvus-io/milvus-sdk-node/blob/main/examples/milvus/DataQuery.ts

On my testing, If I set the consistency_level to Strong, I can get the correct result.

[shanghaikid]

@sadwsh

[sadwsh]

@shanghaikid thank you for providing the example file.

I've done some tests and also changed the code to rather simulate the conditions I use in my own project.
Generally the code works fine and the selected entities are also deleted from the collection. But as soon as I use the autoID feature on the id field, then the delete doesn't seem to work anymore and the subsequent query shows all 6 entities.

Here's my modified version of the code:

import { MilvusClient, InsertReq, DataType } from '@zilliz/milvus2-sdk-node';

const COLLECTION_NAME = 'data_query_example';

(async () => {
  // build client
  const milvusClient = new MilvusClient({
    address: 'localhost:19530',
    username: 'username',
    password: 'Aa12345!!',
  });

  console.log('Node client is initialized.');
  // create collection
  const create = await milvusClient.createCollection({
    collection_name: COLLECTION_NAME,
    consistency_level: 'Strong',
    fields: [
      {
        name: 'id',
        description: 'ID field',
        data_type: DataType.Int64,
        is_primary_key: true,
        autoID: true,
      },
      {
        name: 'vector',
        description: 'Vector field',
        data_type: DataType.FloatVector,
        dim: 8,
      },
      { name: 'height', description: 'int64 field', data_type: DataType.Int64 },
      {
        name: 'source',
        description: 'VarChar field',
        data_type: DataType.VarChar,
        max_length: 128,
      },
    ],
  });
  console.log('Create collection is finished.', create);

  // build example data
  const vectorsData = [
    {
      vector: [
        0.11878310581111173, 0.9694947902934701, 0.16443679307243175,
        0.5484226189097237, 0.9839246709011924, 0.5178387104937776,
        0.8716926129208069, 0.5616972243831446,
      ],
      height: 20405,
      source: 'page1',
      // id: 1,
    },
    {
      vector: [
        0.9992090731236536, 0.8248790611809487, 0.8660083940881405,
        0.09946359318481224, 0.6790698063908669, 0.5013786801063624,
        0.795311915725105, 0.9183033261617566,
      ],
      height: 93773,
      source: 'page1',
      // id: 2,
    },
    {
      vector: [
        0.8761291569818763, 0.07127366044153227, 0.775648976160332,
        0.5619757601304878, 0.6076543120476996, 0.8373907516027586,
        0.8556140171597648, 0.4043893119391049,
      ],
      height: 85122,
      source: 'page2',
      // id: 3,
    },
    {
      vector: [
        0.5849602436079879, 0.5108258101682586, 0.8250884731578105,
        0.7996354835509332, 0.8207766774911736, 0.38133662902290566,
        0.7576720055508186, 0.4393152967662368,
      ],
      height: 92037,
      source: 'page2',
      // id: 4,
    },
    {
      vector: [
        0.3768133716738886, 0.3823259261020866, 0.7906232829855262,
        0.31693696726284193, 0.3731715403499176, 0.3300751870649885,
        0.22353556137796238, 0.38062799545615444,
      ],
      height: 31400,
      source: 'page3',
      // id: 5,
    },
    {
      vector: [
        0.0007531778212483964, 0.12941566118774994, 0.9340164428788116,
        0.3795768837758642, 0.4532443258064389, 0.596455163143,
        0.9529469158782906, 0.7692465408044873,
      ],
      height: 1778,
      source: 'page3',
      // id: 6,
    },
  ];
  const params: InsertReq = {
    collection_name: COLLECTION_NAME,
    fields_data: vectorsData,
  };
  // insert data into collection
  await milvusClient.insert(params);
  console.log('Data is inserted.');

  // create index
  const indexCreateParams = {
    index_type: 'HNSW',
    metric_type: 'L2',
    params: JSON.stringify({ M: 8, efConstruction: 64 }),
  };

  const createIndex = await milvusClient.createIndex({
    collection_name: COLLECTION_NAME,
    field_name: 'vector',
    extra_params: indexCreateParams,
  });

  console.log('Index is created', createIndex);

  // need load collection before search
  const load = await milvusClient.loadCollectionSync({
    collection_name: COLLECTION_NAME,
  });
  console.log('Collection is loaded.', load);

  // do the query
  console.time('Query time');
  const query = await milvusClient.query({
    collection_name: COLLECTION_NAME,
    filter: 'source == "page1"',
    output_fields: ['id', 'source', 'vector'],
    limit: 100,
  });
  console.timeEnd('Query time');
  console.log('query result', query);

  // delete data
  const { data } = query;
  const ids: number[] = data.map((entity) => +entity['id']);
  const del = await milvusClient.delete({
    collection_name: COLLECTION_NAME,
    ids,
  });

  console.log('del', del);
  // do the query
  console.time('Query after del');
  const queryAfterDel = await milvusClient.query({
    collection_name: COLLECTION_NAME,
    filter: 'id > 0',
    output_fields: ['id', 'source', 'vector'],
    limit: 100,
  });
  console.timeEnd('Query after del');
  console.log('query after del', queryAfterDel);

  // drop collection
  await milvusClient.dropCollection({
    collection_name: COLLECTION_NAME,
  });
})();

Setting autoID to false and uncommenting the ids to use fixed integer values will work as I said.
I had the assumption, that maybe the autoID feature affects the way the generated ids behave, but the type is still set to Int64, so I have no clues why this shouldn't work!

Would you please look into this issue?

[shanghaikid]

@sadwsh the type of id is int64, which is not supported in node.

 const ids: number[] = data.map((entity) => +entity['id']);

You can use string type for ids array. I tested your code after change this part, everything works.

 const ids: string[] = data.map((entity) => entity['id']);

[shanghaikid]

otherwise the id will be changed by node system. I have an issue for that, but I don't have time to give a better solution. so, please use string id if autoID is enabled.