[Bug]: [benchmark][standalone] dql & upsert request RT burst and upsert timeout(30s) in concurrent dql & upsert scene #37801

wangting0128 · 2024-11-19T04:13:17Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version:master-20241118-0fc0d1a8-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc97
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-bitmap-scenes-kzz5l
test case name: test_bitmap_locust_dql_dml_upsert_standalone

server:

NAME                                                              READY   STATUS      RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-bitmap-scenes-kzz5l-3-etcd-0                              1/1     Running     0                3h13m   10.104.19.232   4am-node28   <none>           <none>
fouramf-bitmap-scenes-kzz5l-3-milvus-standalone-c557649bc-58kw4   1/1     Running     1 (3h12m ago)    3h13m   10.104.24.234   4am-node29   <none>           <none>
fouramf-bitmap-scenes-kzz5l-3-minio-78f678f7c5-7xlzv              1/1     Running     0                3h13m   10.104.18.4     4am-node25   <none>           <none>

dql latency and mutation latency increase

CPU usage drops

queryNode growing segment increase

The dql reporting metric is interrupted and the dql requests are queued

client log:

[2024-11-18 11:43:10,108 - ERROR - fouram]: grpc RpcError: [upsert], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-11-18 11:42:40.104889', 'gRPC error': '2024-11-18 11:43:10.108482'}> (decorators.py:151)
[2024-11-18 11:43:14,604 - ERROR - fouram]: grpc RpcError: [upsert], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-11-18 11:42:44.601758', 'gRPC error': '2024-11-18 11:43:14.604947'}> (decorators.py:151)
[2024-11-18 11:43:15,103 - ERROR - fouram]: grpc RpcError: [upsert], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-11-18 11:42:45.099702', 'gRPC error': '2024-11-18 11:43:15.103004'}> (decorators.py:151)
[2024-11-18 11:43:16,200 - ERROR - fouram]: grpc RpcError: [upsert], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-11-18 11:42:46.197698', 'gRPC error': '2024-11-18 11:43:16.200882'}> (decorators.py:151)
[2024-11-18 11:43:16,204 - ERROR - fouram]: grpc RpcError: [upsert], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-11-18 11:42:46.201354', 'gRPC error': '2024-11-18 11:43:16.204871'}> (decorators.py:151)

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `primary key: INT64`, shards_num=16
            1. building `BITMAP` index on all supported 12 scalar fields
            2. 2 fields of different vector types
            3. verify DQL & DML(upsert) requests

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim
                'sparse_float_vector': sparse_range=[1, 100] <- the range of non-zero values of a sparse vector
                'id': primary key type is INT64

                all scalar fields: varchar max_length=100, array max_capacity=13
            2. build indexes:
                IVF_SQ8: 'float_vector'
                SPARSE_WAND: 'sparse_float_vector'

                BITMAP: all scalar fields
            3. insert 500k data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
            7. concurrent request:
                - search
                - query
                - hybrid_search
                - load
                - upsert: batch=10
                - flush: ignore RateLimiter

Milvus Log

No response

Anything else?

test result:

[2024-11-18 12:07:37,351 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-11-18 12:07:37,351 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-11-18 12:07:37,351 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-11-18 12:07:37,351 -  INFO - fouram]: grpc     flush                                                                           1160     0(0.00%) | 179493     506  707979 109000 |    0.11        0.00 (stats.py:789)
[2024-11-18 12:07:37,351 -  INFO - fouram]: grpc     hybrid_search                                                                   1190     0(0.00%) |    968      53   59487     94 |    0.11        0.00 (stats.py:789)
[2024-11-18 12:07:37,351 -  INFO - fouram]: grpc     load                                                                            1200     0(0.00%) |    122       4    7893     27 |    0.11        0.00 (stats.py:789)
[2024-11-18 12:07:37,351 -  INFO - fouram]: grpc     query                                                                           1182     0(0.00%) |    441       9   54596     14 |    0.11        0.00 (stats.py:789)
[2024-11-18 12:07:37,351 -  INFO - fouram]: grpc     search                                                                          1208     0(0.00%) |    865       7   59891     13 |    0.11        0.00 (stats.py:789)
[2024-11-18 12:07:37,352 -  INFO - fouram]: grpc     upsert                                                                          1152     5(0.43%) |    529       7   30004     16 |    0.11        0.00 (stats.py:789)
[2024-11-18 12:07:37,352 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-11-18 12:07:37,352 -  INFO - fouram]:          Aggregated                                                                      7092     5(0.07%) |  29848       4  707979     31 |    0.66        0.00 (stats.py:789)
[2024-11-18 12:07:37,352 -  INFO - fouram]:  (stats.py:790)
[2024-11-18 12:07:37,356 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'standalone',
            'config_name': 'standalone_16c16m',
            'config': {'standalone': {'resources': {'limits': {'cpu': '16.0', 'memory': '16Gi'}, 'requests': {'cpu': '9.0', 'memory': '9Gi'}}},
                       'cluster': {'enabled': False},
                       'etcd': {'replicaCount': 1, 'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
                       'minio': {'mode': 'standalone', 'metrics': {'podMonitor': {'enabled': True}}},
                       'pulsar': {'enabled': False},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': 'master-20241118-0fc0d1a8-amd64'}}},
            'host': 'fouramf-bitmap-scenes-kzz5l-3-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_bitmap_locust_dql_dml_upsert_standalone',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'max_length': 100,
                                                    'scalars_index': {'int8_1': {'index_type': 'BITMAP'},
                                                                      'int16_1': {'index_type': 'BITMAP'},
                                                                      'int32_1': {'index_type': 'BITMAP'},
                                                                      'int64_1': {'index_type': 'BITMAP'},
                                                                      'varchar_1': {'index_type': 'BITMAP'},
                                                                      'bool_1': {'index_type': 'BITMAP'},
                                                                      'array_int8_1': {'index_type': 'BITMAP'},
                                                                      'array_int16_1': {'index_type': 'BITMAP'},
                                                                      'array_int32_1': {'index_type': 'BITMAP'},
                                                                      'array_int64_1': {'index_type': 'BITMAP'},
                                                                      'array_varchar_1': {'index_type': 'BITMAP'},
                                                                      'array_bool_1': {'index_type': 'BITMAP'}},
                                                    'vectors_index': {'sparse_float_vector': {'index_type': 'SPARSE_INVERTED_INDEX',
                                                                                              'index_param': {'drop_ratio_build': 0.2},
                                                                                              'metric_type': 'IP'}},
                                                    'scalars_params': {'array_int8_1': {'params': {'max_capacity': 13},
                                                                                        'other_params': {'dataset': 'random_algorithm',
                                                                                                         'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                              'specify_range': [-128, 128],
                                                                                                                              'max_capacity': 13}}},
                                                                       'array_int16_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                               'specify_range': [-200, 200],
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_int32_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'specify_scope',
                                                                                                                               'specify_range': [-300, 300],
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_int64_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'fixed_value_range',
                                                                                                                               'specify_range': [-400, 432],
                                                                                                                               'batch': 50,
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_varchar_1': {'params': {'max_capacity': 13},
                                                                                           'other_params': {'dataset': 'random_algorithm',
                                                                                                            'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                                 'specify_range': [-1500, 1500],
                                                                                                                                 'max_capacity': 13}}},
                                                                       'array_bool_1': {'params': {'max_capacity': 13}},
                                                                       'int8_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                   'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                        'specify_range': [-128, 128],
                                                                                                                        'max_capacity': 13}}},
                                                                       'int16_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                         'specify_range': [-200, 200],
                                                                                                                         'max_capacity': 13}}},
                                                                       'int32_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'specify_scope',
                                                                                                                         'specify_range': [-300, 300],
                                                                                                                         'max_capacity': 13}}},
                                                                       'int64_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'fixed_value_range',
                                                                                                                         'specify_range': [-400, 432],
                                                                                                                         'batch': 50,
                                                                                                                         'max_capacity': 13}}},
                                                                       'varchar_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                      'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                           'specify_range': [-1500, 1500],
                                                                                                                           'max_capacity': 13}}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 500000,
                                                    'ni_per': 5000},
                                 'collection_params': {'other_fields': ['sparse_float_vector', 'int8_1', 'int16_1', 'int32_1', 'int64_1', 'varchar_1', 'bool_1',
                                                                        'array_int8_1', 'array_int16_1', 'array_int32_1', 'array_int64_1', 'array_varchar_1',
                                                                        'array_bool_1'],
                                                       'shards_num': 16},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_SQ8', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20, 'during_time': '3h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 8,
                                                                  'top_k': 10,
                                                                  'search_param': {'nprobe': 16},
                                                                  'expr': 'int8_1 == 100',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': ['id', 'float_vector', 'int64_1'],
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 180,
                                                                  'random_data': True,
                                                                  'check_task': 'check_search_output',
                                                                  'check_items': {'nq': 8}}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 > -1',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': 10,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': 60,
                                                                  'consistency_level': None,
                                                                  'random_data': False,
                                                                  'random_count': 0,
                                                                  'random_range': [0, 1],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64',
                                                                  'check_task': 'check_query_output',
                                                                  'check_items': {'expect_length': 10}}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 3,
                                                                  'top_k': 5,
                                                                  'reqs': [{'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': '(array_contains_any(array_int32_1, [0]) || array_contains(array_int64_1, '
                                                                                    '1)) || ((varchar_1 like "1%") and (bool_1 == True))',
                                                                            'top_k': 100},
                                                                           {'search_param': {'drop_ratio_search': 0.1},
                                                                            'anns_field': 'sparse_float_vector',
                                                                            'expr': 'not (int16_1 == int8_1) && ARRAY_CONTAINS_ANY(array_int64_1, [-1, 0, '
                                                                                    '1])'}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 180,
                                                                  'random_data': True,
                                                                  'check_task': 'check_search_output',
                                                                  'check_items': {'output_fields': ['sparse_float_vector', 'int8_1', 'int16_1', 'int32_1',
                                                                                                    'int64_1', 'varchar_1', 'bool_1', 'array_int8_1',
                                                                                                    'array_int16_1', 'array_int32_1', 'array_int64_1',
                                                                                                    'array_varchar_1', 'array_bool_1', 'id', 'float_vector'],
                                                                                  'nq': 3}}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1, 'timeout': 180, 'check_task': 'check_response', 'check_items': None}},
                                                      {'type': 'upsert',
                                                       'weight': 1,
                                                       'params': {'nb': 10,
                                                                  'timeout': 30,
                                                                  'random_id': False,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 500000,
                                                                  'shuffle_id': False,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'flush',
                                                       'weight': 1,
                                                       'params': {'timeout': 600,
                                                                  'check_task': 'check_ignore_expected_errors',
                                                                  'check_items': [{'message': 'request is rejected by grpc RateLimiter middleware, please '
                                                                                              'retry later'},
                                                                                  {'message': 'wait for flush timeout'}]}}]},
            'run_id': 2024111801204280,
            'datetime': '2024-11-18 08:55:20.537468',
            'client_version': '2.5.0'},
 'result': {'test_result': {'index': {'RT': 489.4149,
                                      'sparse_float_vector': {'RT': 0.5123},
                                      'int8_1': {'RT': 0.5178},
                                      'int16_1': {'RT': 0.5181},
                                      'int32_1': {'RT': 0.5107},
                                      'int64_1': {'RT': 0.5108},
                                      'varchar_1': {'RT': 0.5124},
                                      'bool_1': {'RT': 0.5125},
                                      'array_int8_1': {'RT': 0.5112},
                                      'array_int16_1': {'RT': 0.5187},
                                      'array_int32_1': {'RT': 0.5136},
                                      'array_int64_1': {'RT': 0.5129},
                                      'array_varchar_1': {'RT': 0.5123},
                                      'array_bool_1': {'RT': 0.5129}},
                            'insert': {'total_time': 38.9207, 'VPS': 12846.6343, 'batch_time': 0.3892, 'batch': 5000},
                            'flush': {'RT': 3.1105},
                            'load': {'RT': 1.6411},
                            'Locust': {'Aggregated': {'Requests': 7092,
                                                      'Fails': 5,
                                                      'RPS': 0.66,
                                                      'fail_s': 0.0,
                                                      'RT_max': 707979.46,
                                                      'RT_avg': 29848.98,
                                                      'TP50': 31,
                                                      'TP99': 601000.0},
                                       'flush': {'Requests': 1160,
                                                 'Fails': 0,
                                                 'RPS': 0.11,
                                                 'fail_s': 0.0,
                                                 'RT_max': 707979.46,
                                                 'RT_avg': 179493.61,
                                                 'TP50': 109000.0,
                                                 'TP99': 601000.0},
                                       'hybrid_search': {'Requests': 1190,
                                                         'Fails': 0,
                                                         'RPS': 0.11,
                                                         'fail_s': 0.0,
                                                         'RT_max': 59487.16,
                                                         'RT_avg': 968.27,
                                                         'TP50': 94,
                                                         'TP99': 32000.0},
                                       'load': {'Requests': 1200,
                                                'Fails': 0,
                                                'RPS': 0.11,
                                                'fail_s': 0.0,
                                                'RT_max': 7893.68,
                                                'RT_avg': 122.53,
                                                'TP50': 27,
                                                'TP99': 3200.0},
                                       'query': {'Requests': 1182,
                                                 'Fails': 0,
                                                 'RPS': 0.11,
                                                 'fail_s': 0.0,
                                                 'RT_max': 54596.98,
                                                 'RT_avg': 441.74,
                                                 'TP50': 14,
                                                 'TP99': 15000.0},
                                       'search': {'Requests': 1208,
                                                  'Fails': 0,
                                                  'RPS': 0.11,
                                                  'fail_s': 0.0,
                                                  'RT_max': 59891.24,
                                                  'RT_avg': 865.25,
                                                  'TP50': 13,
                                                  'TP99': 30000.0},
                                       'upsert': {'Requests': 1152,
                                                  'Fails': 5,
                                                  'RPS': 0.11,
                                                  'fail_s': 0.0,
                                                  'RT_max': 30004.69,
                                                  'RT_avg': 529.26,
                                                  'TP50': 16,
                                                  'TP99': 15000.0}}}}}

The text was updated successfully, but these errors were encountered:

yanliang567 · 2024-11-19T06:27:11Z

this is a regular test that passed before 11/11

wangting0128 · 2024-11-19T07:19:19Z

Test on November 13th was successful

argo task：fouramf-bitmap-scenes-rg47r
image： master-20241113-3389a6b5-amd64

server:

NAME                                                              READY   STATUS             RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-bitmap-scenes-rg47r-3-etcd-0                              1/1     Running            0               3h13m   10.104.34.16    4am-node37   <none>           <none>
fouramf-bitmap-scenes-rg47r-3-milvus-standalone-7c586bccffzlfhd   1/1     Running            3 (3h12m ago)   3h13m   10.104.26.30    4am-node32   <none>           <none>
fouramf-bitmap-scenes-rg47r-3-minio-64dfb4d5c8-pkzpj              1/1     Running            0               3h13m   10.104.34.19    4am-node37   <none>           <none>

liliu-z · 2024-11-19T08:19:52Z

/assign @tedxu
plz help redirect
/unassign @zhagnlu

yanliang567 · 2024-11-19T09:53:21Z

there are 500+ threads in querynode

liliu-z · 2024-11-19T10:45:44Z

CPU of host Node was booming

While CPU usage of this pod was dropping

Super long latency from Knowhere

Looks like resource racing to me

liliu-z · 2024-11-19T11:47:38Z

/assign @wangting0128 @yanliang567
/unassign @tedxu

wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Nov 19, 2024

wangting0128 added this to the 2.5.0 milestone Nov 19, 2024

wangting0128 assigned zhagnlu and yanliang567 Nov 19, 2024

wangting0128 added the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Nov 19, 2024

yanliang567 removed the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Nov 19, 2024

yanliang567 removed their assignment Nov 19, 2024

sre-ci-robot assigned tedxu and unassigned zhagnlu Nov 19, 2024

sre-ci-robot assigned wangting0128 and yanliang567 and unassigned tedxu Nov 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: [benchmark][standalone] dql & upsert request RT burst and upsert timeout(30s) in concurrent dql & upsert scene #37801

[Bug]: [benchmark][standalone] dql & upsert request RT burst and upsert timeout(30s) in concurrent dql & upsert scene #37801

wangting0128 commented Nov 19, 2024

yanliang567 commented Nov 19, 2024

wangting0128 commented Nov 19, 2024

liliu-z commented Nov 19, 2024

yanliang567 commented Nov 19, 2024

liliu-z commented Nov 19, 2024

liliu-z commented Nov 19, 2024

[Bug]: [benchmark][standalone] dql & upsert request RT burst and upsert timeout(30s) in concurrent dql & upsert scene #37801

[Bug]: [benchmark][standalone] dql & upsert request RT burst and upsert timeout(30s) in concurrent dql & upsert scene #37801

Comments

wangting0128 commented Nov 19, 2024

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

yanliang567 commented Nov 19, 2024

wangting0128 commented Nov 19, 2024

Test on November 13th was successful

liliu-z commented Nov 19, 2024

yanliang567 commented Nov 19, 2024

liliu-z commented Nov 19, 2024

liliu-z commented Nov 19, 2024