[CDCSDK] Data integrity (data correctness on select) issue on YB between 2024.2.0.0-b120 and 2024.2.0.0-b116 in CDC runs

[shamanthchandra-yb]

Jira Link: DB-14016

Description

Suspecting mostly https://phorge.dev.yugabyte.com/D39806 based on data. Please refer slack thread in JIRA.

Source row count was less than sink row count. Even after load is completed, source count just keeps juggling between the values when we query multiple times.

Table name: test_cdc_d6e254

Last load completed at: 19:13:27

2024-11-12 19:52:05,014: INFO: waiting replication test_cdc_d6e254 yb=170926 sink=90383 kafka_topic_size=1061982
2024-11-12 20:10:06,834: INFO: waiting replication test_cdc_d6e254 yb=170919 sink=130312 kafka_topic_size=1079131
2024-11-12 20:27:07,628: INFO: waiting replication test_cdc_d6e254 yb=170914 sink=170927 kafka_topic_size=1079131
2024-11-12 20:28:43,254: INFO: waiting replication test_cdc_d6e254 yb=170874 sink=170927 kafka_topic_size=1079131
2024-11-12 21:01:33,221: INFO: waiting replication test_cdc_d6e254 yb=170880 sink=170927 kafka_topic_size=1093692

Source connector version

quay.io/yugabyte/debezium-connector:dz.1.9.5.yb.grpc.2024.1

Connector configuration

add connector connector_name='ybconnector_cdc_6bb0ca_test_cdc_5c956c' stream_id='a01f21ad52ec0890554da9de33c1a8d1' db_name='cdc_6bb0ca' connector_host='172.151.23.169' table_list=['test_cdc_5c956c'] {'name': 'ybconnector_cdc_6bb0ca_test_cdc_5c956c', 'config': {'database.master.addresses': '172.151.25.20:7100,172.151.26.192:7100,172.151.28.227:7100', 'database.hostname': '172.151.25.20:5433,172.151.26.192:5433,172.151.28.227:5433', 'database.port': 5433, 'database.masterhost': '172.151.25.20', 'database.masterport': '7100', 'database.user': 'yugabyte', 'database.password': 'yugabyte', 'database.dbname': 'cdc_6bb0ca', 'snapshot.mode': 'initial', 'admin.operation.timeout.ms': 600000, 'socket.read.timeout.ms': 300000, 'max.connector.retries': '10', 'operation.timeout.ms': 600000, 'topic.creation.default.compression.type': 'lz4', 'topic.creation.default.cleanup.policy': 'delete', 'topic.creation.default.partitions': 2, 'topic.creation.default.replication.factor': '1', 'tasks.max': '5', 'connector.class': 'io.debezium.connector.yugabytedb.YugabyteDBgRPCConnector', 'database.server.name': 'ybconnector_cdc_6bb0ca_test_cdc_5c956c', 'database.streamid': 'a01f21ad52ec0890554da9de33c1a8d1', 'table.include.list': 'public.test_cdc_5c956c', 'database.sslrootcert': '/kafka/ca.crt'}}

YugabyteDB version

2024.2.0.0-b120

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.