Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][cluster] hybrid_search raises error fail to search on QueryNode 2: channel not found in concurrent dql & ddl scene #37830

Open
1 task done
wangting0128 opened this issue Nov 20, 2024 · 2 comments
Open
1 task done

[Bug]: [benchmark][cluster] hybrid_search raises error fail to search on QueryNode 2: channel not found in concurrent dql & ddl scene #37830

wangting0128 opened this issue Nov 20, 2024 · 2 comments
Assignees
@liliu-z
Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@wangting0128
Copy link
Contributor

wangting0128 commented Nov 20, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:master-20241119-b3fc5301-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc106
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: memory-opt-scenes-m2jwl
test case name: test_hybrid_search_locust_multi_ddl_dql_hybrid_search_cluster

server:

NAME                                                              READY   STATUS      RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
memory-opt-scenes-m2jwl-4-etcd-0                                  1/1     Running     0                14h     10.104.21.71    4am-node24   <none>           <none>
memory-opt-scenes-m2jwl-4-etcd-1                                  1/1     Running     0                14h     10.104.23.39    4am-node27   <none>           <none>
memory-opt-scenes-m2jwl-4-etcd-2                                  1/1     Running     0                14h     10.104.18.213   4am-node25   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-datanode-566c74cb8b-j44f4        1/1     Running     5 (14h ago)      14h     10.104.14.116   4am-node18   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-indexnode-869b8fd5cc-lflmx       1/1     Running     0                14h     10.104.20.51    4am-node22   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-indexnode-869b8fd5cc-q87kz       1/1     Running     2 (14h ago)      14h     10.104.33.244   4am-node36   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-indexnode-869b8fd5cc-vpxfp       1/1     Running     2 (14h ago)      14h     10.104.19.11    4am-node28   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-indexnode-869b8fd5cc-vxs6m       1/1     Running     2 (14h ago)      14h     10.104.5.217    4am-node12   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-mixcoord-85f66f55d6-9444s        1/1     Running     0                14h     10.104.6.186    4am-node13   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-proxy-86ddbc987d-xtml2           1/1     Running     0                14h     10.104.6.187    4am-node13   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-querynode-6fd58c9669-74754       1/1     Running     0                14h     10.104.6.188    4am-node13   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-querynode-6fd58c9669-dmff7       1/1     Running     2 (14h ago)      14h     10.104.16.57    4am-node21   <none>           <none>
memory-opt-scenes-m2jwl-4-minio-0                                 1/1     Running     0                14h     10.104.17.84    4am-node23   <none>           <none>
memory-opt-scenes-m2jwl-4-minio-1                                 1/1     Running     0                14h     10.104.15.175   4am-node20   <none>           <none>
memory-opt-scenes-m2jwl-4-minio-2                                 1/1     Running     0                14h     10.104.21.72    4am-node24   <none>           <none>
memory-opt-scenes-m2jwl-4-minio-3                                 1/1     Running     0                14h     10.104.23.42    4am-node27   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-bookie-0                         1/1     Running     0                14h     10.104.15.176   4am-node20   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-bookie-1                         1/1     Running     0                14h     10.104.24.24    4am-node29   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-bookie-2                         1/1     Running     0                14h     10.104.23.43    4am-node27   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-bookie-init-s7x8m                0/1     Completed   0                14h     10.104.21.61    4am-node24   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-broker-0                         1/1     Running     0                14h     10.104.24.16    4am-node29   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-proxy-0                          1/1     Running     0                14h     10.104.17.81    4am-node23   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-pulsar-init-c2sgn                0/1     Completed   0                14h     10.104.20.50    4am-node22   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-recovery-0                       1/1     Running     0                14h     10.104.21.62    4am-node24   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-zookeeper-0                      1/1     Running     0                14h     10.104.23.36    4am-node27   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-zookeeper-1                      1/1     Running     0                14h     10.104.17.88    4am-node23   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-zookeeper-2                      1/1     Running     0                14h     10.104.21.84    4am-node24   <none>           <none>

image

client log:

hybrid search on collection scene_hybrid_search_test_FUdbgF8A failed

[2024-11-19 16:58:44,500 - ERROR - fouram]: RPC error: [hybrid_search], <MilvusException: (code=500, message=fail to search on QueryNode 2: channel not found[channel=by-dev-rootcoord-dml_0_454038175298478930v0])>, <Time:{'RPC start': '2024-11-19 16:58:40.889907', 'RPC error': '2024-11-19 16:58:44.500434'}> (decorators.py:140)

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `DDL & DQL`
            verify DDL & DQL scenario,
            which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 128dim,
                'float_vector_3': 128dim,
                scalar field: int64_1, varchar_1
            2. build indexes:
                IVF_FLAT: 'float_vector'
                HNSW: 'float_vector_1',
                DISKANN: 'float_vector_2'
                IVF_SQ8: 'float_vector_3'
                INVERTED: 'int64_1', 'varchar_1'
                default scalar index: 'id'
            3. insert 1 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
                replica: 1
            7. concurrent request:
                - scene_hybrid_search_test: 4 vector fields, 3 scalar fields
                    (collection: create->insert->flush->index->load->hybrid_search->drop) <- hybrid_search raises error
                - search
                - hybrid_search
                - query

Milvus Log

No response

Anything else?

test result:

[2024-11-20 00:19:16,684 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-11-20 00:19:16,685 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: grpc     hybrid_search                                                                   9092     0(0.00%) |   3013     114   17500   2400 |    0.21        0.00 (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: grpc     query                                                                           9060     0(0.00%) |    855      75   29817    590 |    0.21        0.00 (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: grpc     scene_hybrid_search_test                                                        9007     1(0.01%) |  83431   11993  479722  52000 |    0.21        0.00 (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: grpc     search                                                                          8990     0(0.00%) |   8373    2773   49407   7200 |    0.21        0.00 (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]:          Aggregated                                                                     36149     1(0.00%) |  23842      75  479722   4700 |    0.84        0.00 (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]:  (stats.py:790)
[2024-11-20 00:19:16,689 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c8m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '32.0', 'memory': '32Gi'}, 'requests': {'cpu': '17.0', 'memory': '17Gi'}}, 'replicas': 2},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0', 'memory': '8Gi'}, 'requests': {'cpu': '5.0', 'memory': '5Gi'}}, 'replicas': 4},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0', 'memory': '8Gi'}, 'requests': {'cpu': '2.0', 'memory': '5Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'extraConfigFiles': {'user.yaml': 'queryNode:\n  segcore:\n    multipleChunkedEnable: true\n'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': 'master-20241119-b3fc5301-amd64'}}},
            'host': 'memory-opt-scenes-m2jwl-4-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_multi_ddl_dql_hybrid_search_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {}, 'int64_1': {'index_type': 'INVERTED'}, 'varchar_1': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8, 'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'DISKANN', 'index_param': {}, 'metric_type': 'IP'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 2048},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_2': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_3': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 1000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1', 'float_vector_2', 'float_vector_3', 'int64_1', 'varchar_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20, 'during_time': '12h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'scene_hybrid_search_test',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 1,
                                                                  'reqs': [{'search_param': {'nprobe': 128}, 'anns_field': 'float_vector', 'top_k': 100},
                                                                           {'search_param': {'nprobe': 32},
                                                                            'anns_field': 'float_vector_scene_hybrid_search_test_1',
                                                                            'top_k': 10},
                                                                           {'search_param': {'ef': 32},
                                                                            'anns_field': 'float_vector_scene_hybrid_search_test_2',
                                                                            'top_k': 5},
                                                                           {'search_param': {'search_list': 20},
                                                                            'anns_field': 'float_vector_scene_hybrid_search_test_3',
                                                                            'top_k': 10}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'dataset': 'local',
                                                                  'dim': 128,
                                                                  'shards_num': 2,
                                                                  'data_size': 3000,
                                                                  'nb': 3000,
                                                                  'index_type': 'IVF_SQ8',
                                                                  'index_param': {'nlist': 2048},
                                                                  'metric_type': 'L2',
                                                                  'other_fields': ['float_vector_scene_hybrid_search_test_1',
                                                                                   'float_vector_scene_hybrid_search_test_2',
                                                                                   'float_vector_scene_hybrid_search_test_3', 'int64_1', 'bool_1',
                                                                                   'varchar_1'],
                                                                  'replica_number': 1,
                                                                  'scalars_params': {'float_vector_scene_hybrid_search_test_1': {'params': {'dim': 128},
                                                                                                                                 'other_params': {'dataset': 'sift'}},
                                                                                     'float_vector_scene_hybrid_search_test_2': {'params': {'dim': 128},
                                                                                                                                 'other_params': {'dataset': 'sift'}},
                                                                                     'float_vector_scene_hybrid_search_test_3': {'params': {'dim': 128},
                                                                                                                                 'other_params': {'dataset': 'sift'}}},
                                                                  'scalars_index': {'int64_1': {},
                                                                                    'bool_1': {'index_type': 'INVERTED'},
                                                                                    'varchar_1': {'index_type': 'INVERTED'}},
                                                                  'vectors_index': {'float_vector_scene_hybrid_search_test_1': {'index_type': 'IVF_FLAT',
                                                                                                                                'index_param': {'nlist': 1024},
                                                                                                                                'metric_type': 'L2'},
                                                                                    'float_vector_scene_hybrid_search_test_2': {'index_type': 'HNSW',
                                                                                                                                'index_param': {'M': 8,
                                                                                                                                                'efConstruction': 200},
                                                                                                                                'metric_type': 'L2'},
                                                                                    'float_vector_scene_hybrid_search_test_3': {'index_type': 'DISKANN',
                                                                                                                                'index_param': {},
                                                                                                                                'metric_type': 'IP'}},
                                                                  'prepare_before_insert': False,
                                                                  'hybrid_search_counts': 10,
                                                                  'new_connect': False,
                                                                  'new_user': False}},
                                                      {'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 1,
                                                                  'search_param': {'nprobe': 1000},
                                                                  'expr': 'int64_1 >= 0',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 100,
                                                                  'reqs': [{'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': 'int64_1 > 100000',
                                                                            'top_k': 100},
                                                                           {'search_param': {'ef': 64},
                                                                            'anns_field': 'float_vector_1',
                                                                            'expr': 'id < 900000',
                                                                            'top_k': 10},
                                                                           {'search_param': {'search_list': 32},
                                                                            'anns_field': 'float_vector_2',
                                                                            'expr': 'varchar_1 > "1"',
                                                                            'top_k': 30},
                                                                           {'search_param': {'nprobe': 16}, 'anns_field': 'float_vector_3', 'top_k': 400}],
                                                                  'rerank': {'WeightedRanker': [0.85, 0.95, 0.51, 0.32]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 > -1 && ',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'consistency_level': None,
                                                                  'random_data': True,
                                                                  'random_count': 20,
                                                                  'random_range': [0, 100000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64',
                                                                  'custom_expr': None,
                                                                  'custom_range': [0, 1],
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}}]},
            'run_id': 2024111972065950,
            'datetime': '2024-11-19 11:53:26.410333',
            'client_version': '2.5.0'},
 'result': {'test_result': {'index': {'RT': 62.9087,
                                      'float_vector_1': {'RT': 9.0731},
                                      'float_vector_2': {'RT': 142.9301},
                                      'float_vector_3': {'RT': 0.5176},
                                      'id': {'RT': 0.5732},
                                      'int64_1': {'RT': 0.5161},
                                      'varchar_1': {'RT': 0.5152}},
                            'insert': {'total_time': 153.3464, 'VPS': 6521.1834, 'batch_time': 1.5335, 'batch': 10000},
                            'flush': {'RT': 3.0494},
                            'load': {'RT': 4.6697},
                            'Locust': {'Aggregated': {'Requests': 36149,
                                                      'Fails': 1,
                                                      'RPS': 0.84,
                                                      'fail_s': 0.0,
                                                      'RT_max': 479722.31,
                                                      'RT_avg': 23842.91,
                                                      'TP50': 4700.0,
                                                      'TP99': 252000.0},
                                       'hybrid_search': {'Requests': 9092,
                                                         'Fails': 0,
                                                         'RPS': 0.21,
                                                         'fail_s': 0.0,
                                                         'RT_max': 17500.05,
                                                         'RT_avg': 3013.76,
                                                         'TP50': 2400.0,
                                                         'TP99': 10000.0},
                                       'query': {'Requests': 9060,
                                                 'Fails': 0,
                                                 'RPS': 0.21,
                                                 'fail_s': 0.0,
                                                 'RT_max': 29817.09,
                                                 'RT_avg': 855.73,
                                                 'TP50': 590.0,
                                                 'TP99': 5900.0},
                                       'scene_hybrid_search_test': {'Requests': 9007,
                                                                    'Fails': 1,
                                                                    'RPS': 0.21,
                                                                    'fail_s': 0.0,
                                                                    'RT_max': 479722.31,
                                                                    'RT_avg': 83431.53,
                                                                    'TP50': 52000.0,
                                                                    'TP99': 314000.0},
                                       'search': {'Requests': 8990,
                                                  'Fails': 0,
                                                  'RPS': 0.21,
                                                  'fail_s': 0.0,
                                                  'RT_max': 49407.92,
                                                  'RT_avg': 8373.27,
                                                  'TP50': 7200.0,
                                                  'TP99': 30000.0}}}}}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Nov 20, 2024
@wangting0128 wangting0128 added this to the 2.5.0 milestone Nov 20, 2024
@yanliang567
Copy link
Contributor

/assign @liliu-z
/unassign

@sre-ci-robot sre-ci-robot assigned liliu-z and unassigned yanliang567 Nov 20, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 20, 2024
@liliu-z
Copy link
Member

liliu-z commented Nov 20, 2024 •
edited
Loading

similar to #33550

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liliu-z liliu-z

Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
3 participants
@wangting0128 @yanliang567 @liliu-z