Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

After upgrade to v2.60.10 in Sepolia Nodes can´t sync #12722

Open
diegopazosrego opened this issue Nov 14, 2024 · 2 comments
Open

After upgrade to v2.60.10 in Sepolia Nodes can´t sync #12722

diegopazosrego opened this issue Nov 14, 2024 · 2 comments

Comments

@diegopazosrego
Copy link

System information

Erigon version: ./erigon --version v2.60.10

OS & Version: Linux

Commit hash:

Erigon Command (with flags/config):

       "--chain=sepolia"
            "--db.size.limit=8TB",
            "--metrics",
            "--metrics.addr=0.0.0.0",
            "--metrics.port=6060",
            "--private.api.addr=0.0.0.0:9089",
            "--pprof",
            "--pprof.addr=0.0.0.0",
            "--pprof.port=6061",
            "--snapshots=true",
            "--torrent.upload.rate=500mb",
            "--torrent.download.rate=1200mb", 
            "--http",
            "--http.api=engine,net,eth,debug,trace,txpool,web3",
            "--http.addr=0.0.0.0",
            "--http.port=8545",
            "--http.vhosts=*",
            "--http.corsdomain=*",
            "--authrpc.addr=0.0.0.0",
            "--authrpc.jwtsecret=/mnt/jwtsecret/jwtsecret",
            "--authrpc.port=9545",
            "--authrpc.vhosts=*",
            "--prune=",

Consensus Layer: Lighthouse v5.1.3

Consensus Layer Command (with flags/config):

            "--chain=sepolia",
            "--db.size.limit=8TB", 
            "--metrics",
            "--metrics.addr=0.0.0.0",
            "--metrics.port=6060",
            "--private.api.addr=0.0.0.0:9089",
            "--pprof",
            "--pprof.addr=0.0.0.0",
            "--pprof.port=6061",
            "--snapshots=true",
            "--torrent.upload.rate=500mb",
            "--torrent.download.rate=1200mb", 
            "--http",
            "--http.api=engine,net,eth,debug,trace,txpool,web3",
            "--http.addr=0.0.0.0",
            "--http.port=8545",
            "--http.vhosts=*",
            "--http.corsdomain=*",
            "--authrpc.addr=0.0.0.0",
            "--authrpc.jwtsecret=/mnt/jwtsecret/jwtsecret",
            "--authrpc.port=9545",
            "--authrpc.vhosts=*",
            "--prune=",

Chain/Network: Sepolia

Actual behaviour

After upgrade to v2.60.10 from v2.60.8 in Sepolia, the nodes can´t sync fully. If we request to the node the latest_block we get 0:

curl -vv -H "Origin: localhost" -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"eth_blockNumber","id":1}'  http://localhost:8545
Note: Unnecessary use of -X or --request, POST is already inferred.
* Host localhost:8545 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8545...
* Connected to localhost (::1) port 8545
> POST / HTTP/1.1
> Host: localhost:8545
> User-Agent: curl/8.7.1
> Accept: */*
> Origin: localhost
> Content-Type: application/json
> Content-Length: 51
> 
* upload completely sent off: 51 bytes
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Content-Type: application/json
< Vary: Origin
< Date: Thu, 14 Nov 2024 13:43:20 GMT
< Content-Length: 40
< 
{"jsonrpc":"2.0","id":1,"result":"0x0"}

Also, we can see in our volumes a weird behavior in the node since the upgrade.

image

After 10 hours, this version is always showing this kind of logs:

[INFO] [11-14|13:46:25.977] [EngineBlockDownloader] Downloading block bodies block_num=3783615 delivery/sec=12.0MB wasted/sec=0B remaining=3291652 delivered=2006743 cache=2.0MB alloc=5.3GB sys=7.9GB
[INFO] [11-14|13:46:36.258] [NewPayload] Handling new payload        height=7075601 hash=0x6985977e95eca50ad2fe8443fccfba4ca3053b2439cd7fc0a466d724243f3e43
[INFO] [11-14|13:46:42.749] [p2p] GoodPeers                          eth67=5
[INFO] [11-14|13:46:42.766] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=6.4GB sys=7.9GB
[INFO] [11-14|13:46:42.905] [mem] memory stats                       Rss=13.0GB Size=0B Pss=13.0GB SharedClean=4.0KB SharedDirty=0B PrivateClean=4.7GB PrivateDirty=8.3GB Referenced=13.0GB Anonymous=8.0GB Swap=0B alloc=6.4GB sys=7.9GB
[INFO] [11-14|13:46:48.760] [NewPayload] Handling new payload        height=7075602 hash=0xc6958be319f49b0046e6845845e5b1afa9a054b6c0ea16328db8528ef0cfeea4
[INFO] [11-14|13:46:55.977] [EngineBlockDownloader] Downloading block bodies block_num=3785939 delivery/sec=12.0MB wasted/sec=74.6KB remaining=3289328 delivered=2009069 cache=8.9MB alloc=3.9GB sys=7.9GB
[INFO] [11-14|13:47:00.755] [NewPayload] Handling new payload        height=7075603 hash=0x83dd2d69666529f75bef7133ad52504cdb950dc624cbc89e62f9b02cd1074566
[INFO] [11-14|13:47:13.044] [NewPayload] Handling new payload        height=7075604 hash=0xe898cae21a03d5633b908269597401dcbf6af57249d1508b5b471a14d39d0231
[INFO] [11-14|13:47:24.496] [NewPayload] Handling new payload        height=7075605 hash=0xd8e9c0765c5690ceda415c53cfc6574e496a3072c9f51e235782e0daed0fdf8d
[INFO] [11-14|13:47:25.978] [EngineBlockDownloader] Downloading block bodies block_num=3788740 delivery/sec=12.2MB wasted/sec=0B remaining=3286554 delivered=2011852 cache=129.5KB alloc=5.6GB sys=7.9GB
[INFO] [11-14|13:47:36.522] [NewPayload] Handling new payload        height=7075606 hash=0x07331bf6c440f38134e56f805f97420622fe55275fd5e6c2ded32cba891a6c60
[INFO] [11-14|13:47:49.352] [NewPayload] Handling new payload        height=7075607 hash=0xc3bfdd521ef39221679920478373e32f3df5a69e44dfceec979b0af49dd8eff0
[INFO] [11-14|13:47:55.982] [EngineBlockDownloader] Downloading block bodies block_num=3790985 delivery/sec=11.1MB wasted/sec=0B remaining=3284300 delivered=2014096 cache=0B alloc=4.4GB sys=7.9GB
[INFO] [11-14|13:48:00.778] [NewPayload] Handling new payload        height=7075608 hash=0xeabd0481b650dfe9c4b3115698fa5090fb8575521a83d22be85ef5e23f376f63
[INFO] [11-14|13:48:12.770] [NewPayload] Handling new payload        height=7075609 hash=0xe78bb79be08de7342a4b2553da3ea5483bcab56853fbbe01adeed17df4f3a648
[INFO] [11-14|13:48:25.978] [EngineBlockDownloader] Downloading block bodies block_num=3793754 delivery/sec=12.0MB wasted/sec=0B remaining=3281530 delivered=2016870 cache=558.9KB alloc=6.1GB sys=7.9GB
[INFO] [11-14|13:48:36.504] [NewPayload] Handling new payload        height=7075610 hash=0x92a83883ef6594c0cd9011bff107b98b1b91207109631e22c3733c65835f160c
[INFO] [11-14|13:48:48.686] [NewPayload] Handling new payload        height=7075611 hash=0x1c5e5e1912c45919f5efca69ca681f11452745ad650aae7a03b1d5daff468bb0
[INFO] [11-14|13:48:55.977] [EngineBlockDownloader] Downloading block bodies block_num=3795445 delivery/sec=6.2MB wasted/sec=4.4MB remaining=3279833 delivered=2018556 cache=0B alloc=4.8GB sys=7.9GB
[INFO] [11-14|13:49:00.300] [NewPayload] Handling new payload        height=7075612 hash=0xbb3f1b0575f72cfaaca8ce87c16300311a9d5b0dd75f54adc2562ba29c3707a6
[INFO] [11-14|13:49:12.551] [NewPayload] Handling new payload        height=7075613 hash=0x365060bb50aa3a77fab7205956c9c51defd633700c4f025041bbb78e6d41a4a9
[INFO] [11-14|13:49:24.404] [NewPayload] Handling new payload        height=7075614 hash=0xcebf7051c2d4b17e7b4ed39a547744e5e784cda96da53d64b66fa302af28b4bc
[INFO] [11-14|13:49:25.977] [EngineBlockDownloader] Downloading block bodies block_num=3797648 delivery/sec=9.5MB wasted/sec=280.8KB remaining=3277749 delivered=2020850 cache=6.2MB alloc=6.2GB sys=7.9GB
[INFO] [11-14|13:49:36.454] [NewPayload] Handling new payload        height=7075615 hash=0x0473e12e95fd0058622779bf602ed17c81ee81b80344c30c5c250ccbcdf57baa
[INFO] [11-14|13:49:42.745] [p2p] GoodPeers                          eth68=1 eth67=5
[INFO] [11-14|13:49:42.766] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=3.9GB sys=7.9GB
[INFO] [11-14|13:49:42.900] [mem] memory stats                       Rss=12.9GB Size=0B Pss=12.9GB SharedClean=4.0KB SharedDirty=0B PrivateClean=5.0GB PrivateDirty=8.0GB Referenced=12.9GB Anonymous=7.8GB Swap=0B alloc=3.9GB sys=7.9GB
[INFO] [11-14|13:49:48.391] [NewPayload] Handling new payload        height=7075616 hash=0xe04e5ecdca15fd8e77d1ec17b7f3d744b633e4d1433722cd10521be9ff9aefc3
[INFO] [11-14|13:49:55.977] [EngineBlockDownloader] Downloading block bodies block_num=3798875 delivery/sec=4.1MB wasted/sec=883.9KB remaining=3276414 delivered=2022019 cache=2.0MB alloc=4.2GB sys=7.9GB
[INFO] [11-14|13:50:00.510] [NewPayload] Handling new payload        height=7075617 hash=0x21a994db44f592c46b0c8ed34d5559a82cfc52fa0ebd1ebe5e02e427e35265ab
[INFO] [11-14|13:50:12.375] [NewPayload] Handling new payload        height=7075618 hash=0x8f8b02ead03436ccfa1cc156bf2ecbf888a0134195d2c2b2e53f7f837a33d86b
[INFO] [11-14|13:50:24.241] [NewPayload] Handling new payload        height=7075619 hash=0x256ed723a6b5fae8b9263ce8609928470ddcf1601943dd8b52a98117407caee6
[INFO] [11-14|13:50:25.977] [EngineBlockDownloader] Downloading block bodies block_num=3800467 delivery/sec=4.4MB wasted/sec=0B remaining=3274825 delivered=2023596 cache=894.6KB alloc=4.9GB sys=7.9GB
[INFO] [11-14|13:50:36.530] [NewPayload] Handling new payload        height=7075620 hash=0x9a6ef4b81db73ecb9d223c1bb7959dc4f3cd2b1cc7fba3ffd62276484d535613
[INFO] [11-14|13:50:48.473] [NewPayload] Handling new payload        height=7075621 hash=0x80e39af0dd6f8341dd0329768e37e1a6e6fc32d2f0b1dce0e3468abe42188d25
[INFO] [11-14|13:50:55.978] [EngineBlockDownloader] Downloading block bodies block_num=3802591 delivery/sec=5.8MB wasted/sec=0B remaining=3272700 delivered=2025718 cache=1.3MB alloc=5.7GB sys=7.9GB
[INFO] [11-14|13:51:00.503] [NewPayload] Handling new payload        height=7075622 hash=0x1b3489bcd32a98b3e8358f6f6a7174a64fa0db681d05bbceb267324368612344
[INFO] [11-14|13:51:13.389] [NewPayload] Handling new payload        height=7075623 hash=0x161513a05d78cfe7b17cafc79fdb4ae54a03e32d94ae80f02392196e29555be0
[INFO] [11-14|13:51:24.522] [NewPayload] Handling new payload        height=7075624 hash=0x30669dc580c0e87948ddb65419b0d718e96b30cbb22f91137c175e86712347ef
[INFO] [11-14|13:51:25.977] [EngineBlockDownloader] Downloading block bodies block_num=3804199 delivery/sec=4.9MB wasted/sec=439.7KB remaining=3271075 delivered=2027310 cache=0B alloc=3.7GB sys=7.9GB
[INFO] [11-14|13:51:36.584] [NewPayload] Handling new payload        height=7075625 hash=0x5e9b4a6cbeabf7cff37dad422b0d52fff0a662511739599b2d783ecacacd6135
[INFO] [11-14|13:51:48.379] [NewPayload] Handling new payload        height=7075626 hash=0xd5e801ab1d7b0bb4ae4d5affaea6618062528dfdedafddf4687a4dfdb121376d
[INFO] [11-14|13:51:55.977] [EngineBlockDownloader] Downloading block bodies block_num=3805036 delivery/sec=4.5MB wasted/sec=1.5MB remaining=3270231 delivered=2028325 cache=21.8MB alloc=4.6GB sys=7.9GB
[INFO] [11-14|13:52:00.514] [NewPayload] Handling new payload        height=7075627 hash=0x0260d9ad6a1fc0adffe0b65f995b8641fbdd69145a48f1f490a8788a6b9e4f1a
[INFO] [11-14|13:52:12.479] [NewPayload] Handling new payload        height=7075628 hash=0x95507b869c3c892432ebeacb10f0dbb516647694c2d73b820a2629e51e409493
[INFO] [11-14|13:52:24.590] [NewPayload] Handling new payload        height=7075629 hash=0x5e49eeefed7323fb1a04de4b07be504542e3301ddf94c9007274fb028ec03148
[INFO] [11-14|13:52:25.977] [EngineBlockDownloader] Downloading block bodies block_num=3807199 delivery/sec=6.7MB wasted/sec=752.5KB remaining=3268176 delivered=2030330 cache=1.1MB alloc=5.7GB sys=7.9GB
[INFO] [11-14|13:52:37.343] [NewPayload] Handling new payload        height=7075630 hash=0x562ad746ba4759706da9b3157f5d17ed3f105b10a100ec23b4f3c55c6d64fd4c
[INFO] [11-14|13:52:42.744] [p2p] GoodPeers                          eth68=1 eth67=5
[INFO] [11-14|13:52:42.766] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=6.2GB sys=7.9GB
[INFO] [11-14|13:52:42.904] [mem] memory stats                       Rss=13.1GB Size=0B Pss=13.1GB SharedClean=4.0KB SharedDirty=0B PrivateClean=5.3GB PrivateDirty=7.8GB Referenced=13.1GB Anonymous=7.6GB Swap=0B alloc=6.2GB sys=7.9GB
[INFO] [11-14|13:52:48.260] [NewPayload] Handling new payload        height=7075631 hash=0x9dedb940e5993676892c01d3be281ea853e9948d9e61a659a68624592af1e0c4
[INFO] [11-14|13:52:55.977] [EngineBlockDownloader] Downloading block bodies block_num=3809130 delivery/sec=5.1MB wasted/sec=117.7KB remaining=3266142 delivered=2032248 cache=3.0MB alloc=6.4GB sys=7.9GB
[INFO] [11-14|13:53:00.434] [NewPayload] Handling new payload        height=7075632 hash=0xbc6c0aabfcad65df8d3357914a6acc6a90a818165651153636b4929cccde10db
[INFO] [11-14|13:53:13.189] [NewPayload] Handling new payload        height=7075633 hash=0x7a5b0df90f8283362e06769e6e6abf057859f7718fa7db87064ef3f20a93cfd1
[INFO] [11-14|13:53:24.493] [NewPayload] Handling new payload        height=7075634 hash=0x5f1932f484d7ee8dc88ec35c4dd7008976310e33b4bcc73deeaa8bf42781635b
[INFO] [11-14|13:53:26.316] [EngineBlockDownloader] Writing block bodies block_num=3810698 remaining=3264570 alloc=4.2GB sys=7.9GB
[INFO] [11-14|13:53:49.104] [NewPayload] Handling new payload        height=7075635 hash=0x0946f3926c55e805a92b2c7120aac64e5386e524634f1d703d9ccad16ed6b784
[INFO] [11-14|13:53:55.977] [EngineBlockDownloader] Downloading block bodies block_num=3811716 delivery/sec=6.8MB wasted/sec=4.6MB remaining=3263557 delivered=2034827 cache=0B alloc=5.0GB sys=7.9GB
[INFO] [11-14|13:54:00.494] [NewPayload] Handling new payload        height=7075636 hash=0x938bc4e26066fb23559aa62e51e579d3db681d1d62ee20279a3a192eb1cafa33
[INFO] [11-14|13:54:12.400] [NewPayload] Handling new payload        height=7075637 hash=0x89210cc4e0e92c9c5d831e8a6513615ff376327e2bbe7b941fed1529336c5d0a
[INFO] [11-14|13:54:24.468] [NewPayload] Handling new payload        height=7075638 hash=0x2c2d01a1f9354b93a9d1be2a51dbb4fa5a194eedf69f4cd52ae4135ff3f3e61b
[INFO] [11-14|13:54:25.977] [EngineBlockDownloader] Downloading block bodies block_num=3813003 delivery/sec=2.9MB wasted/sec=2.9MB remaining=3262271 delivered=2036114 cache=0B alloc=5.9GB sys=7.9GB
[INFO] [11-14|13:54:36.521] [NewPayload] Handling new payload        height=7075639 hash=0x86bf63fd1194a9d6932cc6102cff6c4f0963a9e261ad91122fc865a02bea8ba4
[INFO] [11-14|13:54:48.356] [NewPayload] Handling new payload        height=7075640 hash=0xff11a5db4ceeb3b6e8b87698aa511adc1069e8a32914317caf2d5ed4229da9b7
[INFO] [11-14|13:54:56.212] [EngineBlockDownloader] Writing block bodies block_num=3814353 remaining=3260915 alloc=6.7GB sys=7.9GB
[INFO] [11-14|13:55:00.574] [NewPayload] Handling new payload        height=7075641 hash=0xe3c2de7653ae52002a5ae3735e2a30cb2ad98413b387f83a897f8cca7f494088
[INFO] [11-14|13:55:12.347] [NewPayload] Handling new payload        height=7075642 hash=0x74ff4214c41bf2b65a88ead9eb0e29818277c9da5a5ba06d1337a039c8504fe2
[INFO] [11-14|13:55:24.461] [NewPayload] Handling new payload        height=7075643 hash=0xf043eabf678da3f5416b7e7a877839830bef3fb10133d4bf7aa1f8cce618fa7d
[INFO] [11-14|13:55:25.978] [EngineBlockDownloader] Downloading block bodies block_num=3815792 delivery/sec=7.3MB wasted/sec=3.3MB remaining=3259478 delivered=2038903 cache=0B alloc=4.2GB sys=7.9GB

Thanks!
@AskAlexSharov
Copy link
Collaborator

  • cat logs | grep WARN
  • cat logs | grep EROR

@diegopazosrego
Copy link
Author

Hi @AskAlexSharov !

In one of the nodes we only see:

[WARN] [11-15|08:25:26.728] [txpool] flush: sender address not found by ID senderID=73437 [WARN] [11-15|08:25:41.824] [txpool] flush: sender address not found by ID senderID=73437 [WARN] [11-15|08:25:56.956] [txpool] flush: sender address not found by ID senderID=73437 [WARN] [11-15|08:26:12.055] [txpool] flush: sender address not found by ID senderID=73437 [WARN] [11-15|08:26:27.204] [txpool] flush: sender address not found by ID senderID=73437 [WARN] [11-15|08:26:42.301] [txpool] flush: sender address not found by ID senderID=73437

In the other one no WARN or EROR lines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@diegopazosrego @AskAlexSharov