Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement PeerDAS #14129

Draft
wants to merge 83 commits into
base: develop
Choose a base branch
from
Draft

Implement PeerDAS #14129

wants to merge 83 commits into from

Commits on Oct 23, 2024

  1. Configuration menu
    Copy the full SHA
    5e55834 View commit details
    Browse the repository at this point in the history
  2. add it (#13865)

    nisdas authored and nalepae committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    aef68db View commit details
    Browse the repository at this point in the history
  3. add in networking params (#13866)

    nisdas authored and nalepae committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    ac905f6 View commit details
    Browse the repository at this point in the history
  4. Add Support For Discovery Of Column Subnets (#13883)

    * Add Support For Discovery Of Column Subnets
    
    * Lint for SubnetsPerNode
    
    * Manu's Review
    
    * Change to a better name
    nisdas authored and nalepae committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    dee92e5 View commit details
    Browse the repository at this point in the history
  5. Add Data Column Gossip Handlers (#13894)

    * Add Data Column Subscriber
    
    * Add Data Column Vaidator
    
    * Wire all Handlers In
    
    * Fix Build
    
    * Fix Test
    
    * Fix IP in Test
    
    * Fix IP in Test
    nisdas authored and nalepae committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    ab917ba View commit details
    Browse the repository at this point in the history
  6. Add Request And Response RPC Methods For Data Columns (#13909)

    * Add RPC Handler
    
    * Add Column Requests
    
    * Update beacon-chain/db/filesystem/blob.go
    
    Co-authored-by: Manu NALEPA <[email protected]>
    
    * Update beacon-chain/p2p/rpc_topic_mappings.go
    
    Co-authored-by: Manu NALEPA <[email protected]>
    
    * Manu's Review
    
    * Manu's Review
    
    * Interface Fixes
    
    * mock manager
    
    ---------
    
    Co-authored-by: Manu NALEPA <[email protected]>
    nisdas and nalepae committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    ee2d0f4 View commit details
    Browse the repository at this point in the history
  7. Peer das core (#13877)

    * Bump `c-kzg-4844` lib to the `das` branch.
    
    * Implement `MerkleProofKZGCommitments`.
    
    * Implement `das-core.md`.
    
    * Use `peerdas.CustodyColumnSubnets` and `peerdas.CustodyColumns`.
    
    * `CustodyColumnSubnets`: Include `i` in the for loop.
    
    * Remove `computeSubscribedColumnSubnet`.
    
    * Remove `peerdas.CustodyColumns` out of the for loop.
    nalepae committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    bb70e1a View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    325a5b0 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. Update .bazelrc (#13931)

    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    b891d58 View commit details
    Browse the repository at this point in the history
  2. Implement peer DAS proposer RPC (#13922)

    * Remove capital letter from error messages.
    
    * `[4]byte` => `[fieldparams.VersionLength]byte`.
    
    * Prometheus: Remove extra `committee`.
    
    They are probably due to a bad copy/paste.
    
    Note: The name of the probe itself is remaining,
    to ensure backward compatibility.
    
    * Implement Proposer RPC for data columns.
    
    * Fix TestProposer_ProposeBlock_OK test.
    
    * Remove default peerDAS activation.
    
    * `validateDataColumn`: Workaround to return a `VerifiedRODataColumn`
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    a574b87 View commit details
    Browse the repository at this point in the history
  3. Add DA Check For Data Columns (#13938)

    * Add new DA check
    
    * Exit early in the event no commitments exist.
    
    * Gazelle
    
    * Fix Mock Broadcaster
    
    * Fix Test Setup
    
    * Update beacon-chain/blockchain/process_block.go
    
    Co-authored-by: Manu NALEPA <[email protected]>
    
    * Manu's Review
    
    * Fix Build
    
    ---------
    
    Co-authored-by: Manu NALEPA <[email protected]>
    nisdas and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    4b2d7bb View commit details
    Browse the repository at this point in the history
  4. Spectests (#13940)

    * Update `consensus_spec_version` to `v1.5.0-alpha.1`.
    
    * `CustodyColumns`: Fix and implement spec tests.
    
    * Make deepsource happy.
    
    * `^uint64(0)` => `math.MaxUint64`.
    
    * Fix `TestLoadConfigFile` test.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    004f913 View commit details
    Browse the repository at this point in the history
  5. SendDataColumnSidecarByRoot: Return RODataColumn instead of `ROBl…

    …ob`. (#13957)
    
    * `SendDataColumnSidecarByRoot`: Return `RODataColumn` instead of `ROBlob`.
    
    * Make deepsource happier.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    a50d7aa View commit details
    Browse the repository at this point in the history
  6. [PeerDAS] Upgrade c-kzg-4844 package (#13967)

    * Upgrade c-kzg-4844 package
    
    * Upgrade bazel deps
    jtraglia authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    9802e50 View commit details
    Browse the repository at this point in the history
  7. Enable E2E For PeerDAS (#13945)

    * Enable E2E And Add Fixes
    
    * Register Same Topic For Data Columns
    
    * Initialize Capacity Of Slice
    
    * Fix Initialization of Data Column Receiver
    
    * Remove Mix In From Merkle Proof
    
    * E2E: Subscribe to all subnets.
    
    * Remove Index Check
    
    * Remaining Bug Fixes to Get It Working
    
    * Change Evaluator to Allow Test to Finish
    
    * Fix Build
    
    * Add Data Column Verification
    
    * Fix LoopVar Bug
    
    * Do Not Allocate Memory
    
    * Update beacon-chain/blockchain/process_block.go
    
    Co-authored-by: Manu NALEPA <[email protected]>
    
    * Update beacon-chain/core/peerdas/helpers.go
    
    Co-authored-by: Manu NALEPA <[email protected]>
    
    * Update beacon-chain/core/peerdas/helpers.go
    
    Co-authored-by: Manu NALEPA <[email protected]>
    
    * Gofmt
    
    * Fix It Again
    
    * Fix Test Setup
    
    * Fix Build
    
    * Fix Trusted Setup panic
    
    * Fix Trusted Setup panic
    
    * Use New Test
    
    ---------
    
    Co-authored-by: Manu NALEPA <[email protected]>
    nisdas and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    b622e22 View commit details
    Browse the repository at this point in the history
  8. Implement Data Columns By Range Request And Response Methods (#13972)

    * Add Data Structure for New Request Type
    
    * Add Data Column By Range Handler
    
    * Add Data Column Request Methods
    
    * Add new validation for columns by range requests
    
    * Fix Build
    
    * Allow Prysm Node To Fetch Data Columns
    
    * Allow Prysm Node To Fetch Data Columns And Sync
    
    * Bug Fixes For Interop
    
    * GoFmt
    
    * Use different var
    
    * Manu's Review
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    8bf9481 View commit details
    Browse the repository at this point in the history
  9. Sample from peers some data columns. (#13980)

    * PeerDAS: Implement sampling.
    
    * `TestNewRateLimiter`: Fix with the new number of expected registered topics.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    cf42d6c View commit details
    Browse the repository at this point in the history
  10. Set Custody Count Correctly (#14004)

    * Set Custody Count Correctly
    
    * Fix Discovery Count
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    f5acce7 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    5072f2d View commit details
    Browse the repository at this point in the history
  12. Fix CustodyColumns to comply with alpha-2 spectests. (#14008)

    * Adding error wrapping
    
    * Fix `CustodyColumnSubnets` tests.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    3ddcbe6 View commit details
    Browse the repository at this point in the history
  13. Request Data Columns When Fetching Pending Blocks (#14007)

    * Support Data Columns For By Root Requests
    
    * Revert Config Changes
    
    * Fix Panic
    
    * Fix Process Block
    
    * Fix Flags
    
    * Lint
    
    * Support Checkpoint Sync
    
    * Manu's Review
    
    * Add Support For Columns in Remaining Methods
    
    * Unmarshal Uncorrectly
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    6fd443a View commit details
    Browse the repository at this point in the history
  14. Disable Evaluators For E2E (#14019)

    * Hack E2E
    
    * Fix it For Real
    
    * Gofmt
    
    * Remove
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    904b782 View commit details
    Browse the repository at this point in the history
  15. Fix Custody Columns (#14021)

    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    4f80111 View commit details
    Browse the repository at this point in the history
  16. PeerDAS: Implement reconstruction. (#14036)

    * Wrap errors, add logs.
    
    * `missingColumnRequest`: Fix blobs <-> data columns mix.
    
    * `ColumnIndices`: Return `map[uint64]bool` instead of `[fieldparams.NumberOfColumns]bool`.
    
    * `DataColumnSidecars`: `interfaces.SignedBeaconBlock` ==> `interfaces.ReadOnlySignedBeaconBlock`.
    
    We don't need any of the non read-only methods.
    
    * Fix comments.
    
    * `handleUnblidedBlock` ==> `handleUnblindedBlock`.
    
    * `SaveDataColumn`: Move log from debug to trace.
    
    If we attempt to save an already existing data column sidecar,
    a debug log was printed.
    
    This case could be quite common now with the data column reconstruction enabled.
    
    * `sampling_data_columns.go` --> `data_columns_sampling.go`.
    
    * Reconstruct data columns.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    5615f09 View commit details
    Browse the repository at this point in the history
  17. PeerDAS: Implement / use data column feed from database. (#14062)

    * Remove some `_` identifiers.
    
    * Blob storage: Implement a notifier system for data columns.
    
    * `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.
    
    * Address Nishant's comment.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    e7f4807 View commit details
    Browse the repository at this point in the history
  18. PeerDAS: Withhold data on purpose. (#14076)

    * Introduce hidden flag `data-columns-withhold-count`.
    
    * Address Nishant's comment.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    d72ce70 View commit details
    Browse the repository at this point in the history
  19. recoverBlobs: Cover the `0 < blobsCount < fieldparams.MaxBlobsPerBl…

    …ock` case. (#14066)
    
    * `recoverBlobs`: Cover the `0 < blobsCount < fieldparams.MaxBlobsPerBlock` case.
    
    * Fix Nishant's comment.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    b431acd View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    87152c0 View commit details
    Browse the repository at this point in the history
  21. PeerDAS: Gossip the reconstructed columns (#14079)

    * PeerDAS: Broadcast not seen via gossip but reconstructed data columns.
    
    * Address Nishant's comment.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    96f57c3 View commit details
    Browse the repository at this point in the history
  22. PeerDAS: Stop generating new P2P private key at start. (#14099)

    * `privKey`: Improve logs.
    
    * peerDAS: Move functions in file. Add documentation.
    
    * PeerDAS: Remove unused `ComputeExtendedMatrix` and `RecoverMatrix` functions.
    
    * PeerDAS: Stop generating new P2P private key at start.
    
    * Fix sammy' comment.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    5ae7885 View commit details
    Browse the repository at this point in the history
  23. [PeerDAS] rework ENR custody_subnet_count and add tests (#14077)

    * [PeerDAS] rework ENR custody_subnet_count related code
    
    * update according to proposed spec change
    
    * Run gazelle
    0x00101010 authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    bae92a7 View commit details
    Browse the repository at this point in the history
  24. [PeerDAS] fixes and tests for gossiping out data columns (#14102)

    * [PeerDAS] Minor fixes and tests for gossiping out data columns
    
    * Fix metrics
    0x00101010 authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    68e25dd View commit details
    Browse the repository at this point in the history
  25. [PeerDAS] implement DataColumnSidecarsByRootReq and fix related bugs (#…

    …14103)
    
    * [PeerDAS] add data column related protos and fix data column by root bug
    
    * Add more tests
    0x00101010 authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    cf90fae View commit details
    Browse the repository at this point in the history
  26. Fix columns sampling (#14118)

    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    9cf8cfc View commit details
    Browse the repository at this point in the history
  27. [PeerDAS] rework ENR custody_subnet_count and add tests (#14077)

    * [PeerDAS] rework ENR custody_subnet_count related code
    
    * update according to proposed spec change
    
    * Run gazelle
    0x00101010 authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    b5c798b View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    bbf8abe View commit details
    Browse the repository at this point in the history
  29. PeerDAS: Implement IncrementalDAS (#14109)

    * `ConvertPeerIDToNodeID`: Add tests.
    
    * Remove `extractNodeID` and uses `ConvertPeerIDToNodeID` instead.
    
    * Implement IncrementalDAS.
    
    * `DataColumnSamplingLoop` ==> `DataColumnSamplingRoutine`.
    
    * HypergeomCDF: Add test.
    
    * `GetValidCustodyPeers`: Optimize and add tests.
    
    * Remove blank identifiers.
    
    * Implement `CustodyCountFromRecord`.
    
    * Implement `TestP2P.CustodyCountFromRemotePeer`.
    
    * `NewTestP2P`: Add `swarmt.Option` parameters.
    
    * `incrementalDAS`: Rework and add tests.
    
    * Remove useless warning.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    66a37a6 View commit details
    Browse the repository at this point in the history
  30. chore: Encapsulate all kzg functionality for PeerDAS into the kzg pac…

    …kage (#14136)
    
    * chore: move all ckzg related functionality into kzg package
    
    * refactor code to match
    
    * run: bazel run //:gazelle -- fix
    
    * chore: add some docs and stop copying large objects when converting between types
    
    * fixes
    
    * manually add kzg.go dep to Build.Hazel
    
    * move kzg methods to kzg.go
    
    * chore: add RecoverCellsAndProofs method
    
    * bazel run //:gazelle -- fix
    
    * use BytesPerBlob constant
    
    * chore: fix some deepsource issues
    
    * one declaration for commans and blobs
    kevaundray authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    eb5c5e0 View commit details
    Browse the repository at this point in the history
  31. chore!: Refactor RecoverBlob to RecoverCellsAndProofs (#14160)

    * change recoverBlobs to recoverCellsAndProofs
    
    * modify code to take in the cells and proofs for a particular blob instead of the blob itself
    
    * add CellsAndProofs structure
    
    * modify recoverCellsAndProofs to return `cellsAndProofs` structure
    
    * modify `DataColumnSidecarsForReconstruct` to accept the `cellsAndKZGProofs` structure
    
    * bazel run //:gazelle -- fix
    
    * use kzg abstraction for kzg method
    
    * move CellsAndProofs to kzg.go
    kevaundray authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    6b18fb8 View commit details
    Browse the repository at this point in the history
  32. Activate PeerDAS with the EIP7594 Fork Epoch (#14184)

    * Save All the Current Changes
    
    * Add check for data sampling
    
    * Fix Test
    
    * Gazelle
    
    * Manu's Review
    
    * Fix Test
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    d20e441 View commit details
    Browse the repository at this point in the history
  33. Move log from error to debug. (#14194)

    Reason: If a peer does not exposes its `csc` field into it's ENR,
    then there is nothing we can do.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    9100fe7 View commit details
    Browse the repository at this point in the history
  34. chore!: Make Cell be a flat sequence of bytes (#14159)

    * chore: move all ckzg related functionality into kzg package
    
    * refactor code to match
    
    * run: bazel run //:gazelle -- fix
    
    * chore: add some docs and stop copying large objects when converting between types
    
    * fixes
    
    * manually add kzg.go dep to Build.Hazel
    
    * move kzg methods to kzg.go
    
    * chore: add RecoverCellsAndProofs method
    
    * bazel run //:gazelle -- fix
    
    * make Cells be flattened sequence of bytes
    
    * chore: add test for flattening roundtrip
    
    * chore: remove code that was doing the flattening outside of the kzg package
    
    * fix merge
    
    * fix
    
    * remove now un-needed conversion
    
    * use pointers for Cell parameters
    
    * linter
    
    * rename cell conversion methods (this only applies to old version of c-kzg)
    kevaundray authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    1cd0555 View commit details
    Browse the repository at this point in the history
  35. PeerDAS: Add KZG verification when sampling (#14187)

    * `validateDataColumn`: Add comments and remove debug computation.
    
    * `sampleDataColumnsFromPeer`: Add KZG verification
    
    * `VerifyKZGInclusionProofColumn`: Add unit test.
    
    * Make deepsource happy.
    
    * Address Nishant's comment.
    
    * Address Nishant's comment.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    8a218d7 View commit details
    Browse the repository at this point in the history
  36. Trigger PeerDAS At Deneb For E2E (#14193)

    * Trigger At Deneb
    
    * Fix Rate Limits
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    a11636a View commit details
    Browse the repository at this point in the history
  37. chore!: Use RecoverCellsAndKZGProofs instead of RecoverAllCells

    …-> `CellsToBlob` -> `ComputeCellsAndKZGProofs` (#14183)
    
    * use recoverCellsAndKZGProofs
    
    * make recoverAllCells and CellsToBlob private
    
    * chore: all methods now return CellsAndProof struct
    
    * chore: update code
    kevaundray authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    2ef0725 View commit details
    Browse the repository at this point in the history
  38. [PeerDAS] Parallelize data column sampling (#14105)

    * PeerDAS: parallelizing sample queries
    
    * PeerDAS: select sample from non custodied columns
    
    * Finish rebase
    
    * Add more test cases
    0x00101010 authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    1d83f33 View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    e59d059 View commit details
    Browse the repository at this point in the history
  40. Add Current Changes (#14231)

    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    f58c309 View commit details
    Browse the repository at this point in the history
  41. Update ckzg4844 to latest version of das branch (#14223)

    * Update ckzg4844 to latest version
    
    * Run go mod tidy
    
    * Remove unnecessary tests & run goimports
    
    * Remove fieldparams from blockchain/kzg
    
    * Add back blank line
    
    * Avoid large copies
    
    * Run gazelle
    
    * Use trusted setup from the specs & fix issue with struct
    
    * Run goimports
    
    * Fix mistake in makeCellsAndProofs
    
    ---------
    
    Co-authored-by: Manu NALEPA <[email protected]>
    jtraglia and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    18c1f8f View commit details
    Browse the repository at this point in the history
  42. PeerDAS: Run reconstruction in parallel. (#14236)

    * PeerDAS: Run reconstruction in parallel.
    
    * `isDataAvailableDataColumns` --> `isDataColumnsAvailable`
    
    * `isDataColumnsAvailable`: Return `nil` as soon as half of the columns are received.
    
    * Make deepsource happy.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    aa658e0 View commit details
    Browse the repository at this point in the history
  43. Make deepsource happy (#14237)

    * DeepSource: Pass heavy objects by pointers.
    
    * `removeBlockFromQueue`: Remove redundant error checking.
    
    * `fetchBlobsFromPeer`: Use same variable for `append`.
    
    * Remove unused arguments.
    
    * Combine types.
    
    * `Persist`: Add documentation.
    
    * Remove unused receiver
    
    * Remove duplicated import.
    
    * Stop using both pointer and value receiver at the same time.
    
    * `verifyAndPopulateColumns`: Remove unused parameter
    
    * Stop using mpty slice literal used to declare a variable.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    bf0342a View commit details
    Browse the repository at this point in the history
  44. PeerDAS: Fix initial sync (#14208)

    * `SendDataColumnsByRangeRequest`: Add some new fields in logs.
    
    * `BlobStorageSummary`: Implement `HasDataColumnIndex` and `AllDataColumnsAvailable`.
    
    * Implement `fetchDataColumnsFromPeers`.
    
    * `fetchBlobsFromPeer`: Return only one error.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    68721e8 View commit details
    Browse the repository at this point in the history
  45. Fix data columns sampling (#14263)

    * Fix the obvious...
    
    * Data columns sampling: Modify logging.
    
    * `waitForChainStart`: Set it threadsafe - Do only wait once.
    
    * Sampling: Wait for chain start before running the sampling.
    
    Reason: `newDataColumnSampler1D` needs `s.ctxMap`.
    `s.ctxMap` is only set when chain is started.
    
    Previously `waitForChainStart` was only called in `s.registerHandlers`, it self called in a go-routine.
    
    ==> We had a race condition here: Sometimes `newDataColumnSampler1D` were called once `s.ctxMap` were set, sometimes not.
    
    * Adresse Nishant's comments.
    
    * Sampling: Improve logging.
    
    * `waitForChainStart`: Remove `chainIsStarted` check.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    9ae8492 View commit details
    Browse the repository at this point in the history
  46. PeerDAS: Add MetadataV3 with custody_subnet_count (#14274)

    * `sendPingRequest`: Add some comments.
    
    * `sendPingRequest`: Replace `stream.Conn().RemotePeer()` by `peerID`.
    
    * `pingHandler`: Add comments.
    
    * `sendMetaDataRequest`: Add comments and implement an unique test.
    
    * Gather `SchemaVersion`s in the same `const` definition.
    
    * Define `SchemaVersionV3`.
    
    * `MetaDataV1`: Fix comment.
    
    * Proto: Define `MetaDataV2`.
    
    * `MetaDataV2`: Generate SSZ.
    
    * `newColumnSubnetIDs`: Use smaller lines.
    
    * `metaDataHandler` and `sendMetaDataRequest`: Manage `MetaDataV2`.
    
    * `RefreshPersistentSubnets`: Refactor tests (no functional change).
    
    * `RefreshPersistentSubnets`: Refactor and add comments (no functional change).
    
    * `RefreshPersistentSubnets`: Compare cache with both ENR & metadata.
    
    * `RefreshPersistentSubnets`: Manage peerDAS.
    
    * `registerRPCHandlersPeerDAS`: Register `RPCMetaDataTopicV3`.
    
    * `CustodyCountFromRemotePeer`: Retrieve the count from metadata.
    
    Then default to ENR, then default to the default value.
    
    * Update beacon-chain/sync/rpc_metadata.go
    
    Co-authored-by: Nishant Das <[email protected]>
    
    * Fix duplicate case.
    
    * Remove version testing.
    
    * `debug.proto`: Stop breaking ordering.
    
    ---------
    
    Co-authored-by: Nishant Das <[email protected]>
    nalepae and nisdas committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    8c5dac1 View commit details
    Browse the repository at this point in the history
  47. Add Data Column Verification (#14287)

    * Persist All Changes
    
    * Fix All Tests
    
    * Fix Build
    
    * Fix Build
    
    * Fix Build
    
    * Fix Test Again
    
    * Add missing verification
    
    * Add Test Cases for Data Column Validation
    
    * Fix comments for methods
    
    * Fix comments for methods
    
    * Fix Test
    
    * Manu's Review
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    75f06f4 View commit details
    Browse the repository at this point in the history
  48. Implement /eth/v1/beacon/blob_sidecars/{block_id} for peerDAS. (#14312

    )
    
    * `parseIndices`: `O(n**2)` ==> `O(n)`.
    
    * PeerDAS: Implement `/eth/v1/beacon/blob_sidecars/{block_id}`.
    
    * Update beacon-chain/core/peerdas/helpers.go
    
    Co-authored-by: Sammy Rosso <[email protected]>
    
    * Rename some functions.
    
    * `Blobs`: Fix empty slice.
    
    * `recoverCellsAndProofs` --> Move function in `beacon-chain/core/peerdas`.
    
    * peerDAS helpers: Add missing tests.
    
    * Implement `CustodyColumnCount`.
    
    * `RecoverCellsAndProofs`: Remove useless argument `columnsCount`.
    
    * Tests: Add cleanups.
    
    * `blobsFromStoredDataColumns`: Reconstruct if needed.
    
    * Make deepsource happy.
    
    * Beacon API: Use provided indices.
    
    * Make deepsource happier.
    
    ---------
    
    Co-authored-by: Sammy Rosso <[email protected]>
    nalepae and saolyn committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    afc69af View commit details
    Browse the repository at this point in the history
  49. Fix CI in PeerDAS (#14347)

    * Update go.yml
    
    * Disable mnd
    
    * Update .golangci.yml
    
    * Update go.yml
    
    * Update go.yml
    
    * Update .golangci.yml
    
    * Update go.yml
    
    * Fix Lint Issues
    
    * Remove comment
    
    * Update .golangci.yml
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    1304eb8 View commit details
    Browse the repository at this point in the history
  50. Update Config To Latest Value (#14352)

    * Update values
    
    * Update Spec To v1.5.0-alpha.5
    
    * Fix Discovery Tests
    
    * Hardcode Subnet Count For Tests
    
    * Fix All Initial Sync Tests
    
    * Gazelle
    
    * Less Chaotic Service Initialization
    
    * Gazelle
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    edae704 View commit details
    Browse the repository at this point in the history
  51. Use Data Column Validation Across Prysm (#14377)

    * Use Data Column Validation Everywhere
    
    * Fix Build
    
    * Fix Lint
    
    * Fix Clock Synchronizer
    
    * Fix Panic
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    b14e269 View commit details
    Browse the repository at this point in the history
  52. Change Custody Count to Uint8 (#14386)

    * Add Changes for Uint8 Csc
    
    * Fix Build
    
    * Fix Build for Sync
    
    * Fix Discovery Test
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    b242f40 View commit details
    Browse the repository at this point in the history
  53. Fix Bugs in PeerDAS Testing (#14396)

    * Fix Various Bugs in PeerDAS
    
    * Remove Log
    
    * Remove useless copy var.
    
    ---------
    
    Co-authored-by: Manu NALEPA <[email protected]>
    nisdas and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    a7ee0e7 View commit details
    Browse the repository at this point in the history
  54. PeerDAS: Improve logging and reduce the number of needed goroutines f…

    …or reconstruction (#14397)
    
    * `broadcastAndReceiveDataColumns`: Use real `sidecar.ColumnIndex` instead of position in the slice.
    
    And improve logging as well.
    
    * `isDataColumnsAvailable`: Improve logging.
    
    * `validateDataColumn`: Print `Accepted data column sidecar gossip` really at the end.
    
    * Subscriber: Improve logging.
    
    * `sendAndSaveDataColumnSidecars`: Use common used function for logging.
    
    * `dataColumnSidecarByRootRPCHandler`: Logging - Pring `all` instead of all the columns for a super node.
    
    * Verification: Improve logging.
    
    * `DataColumnsWithholdCount`: Set as `uint64` instead `int`.
    
    * `DataColumnFields`: Improve logging.
    
    * Logging: Remove now useless private `columnFields`function.
    
    * Avoid useless goroutines blocking for reconstruction.
    
    * Update beacon-chain/sync/subscriber.go
    
    Co-authored-by: Nishant Das <[email protected]>
    
    * Address Nishant's comment.
    
    * Improve logging.
    
    ---------
    
    Co-authored-by: Nishant Das <[email protected]>
    nalepae and nisdas committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    155ac62 View commit details
    Browse the repository at this point in the history
  55. Set Precompute at 8 (#14399)

    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    bbac508 View commit details
    Browse the repository at this point in the history
  56. Add Data Column Computation Metrics (#14400)

    * Add Data Column Metrics
    
    * Shift it All To Peerdas Package
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    f1d49a9 View commit details
    Browse the repository at this point in the history
  57. Fix Initial Sync with 128 data columns subnets (#14403)

    * `pingPeers`: Add log with new ENR when modified.
    
    * `p2p Start`: Use idiomatic go error syntax.
    
    * P2P `start`: Fix error message.
    
    * Use not bootnodes at all if the `--chain-config-file` flag is used and no `--bootstrap-node` flag is used.
    
    Before this commit, if the  `--chain-config-file` flag is used and no `--bootstrap-node` flag is used, then bootnodes are (incorrectly) defaulted on `mainnet` ones.
    
    * `validPeersExist`: Centralize logs.
    
    * `AddConnectionHandler`: Improve logging.
    
    "Peer connected" does not really reflect the fact that a new peer is actually connected. --> "New peer connection" is more clear.
    
    Also, instead of writing `0`, `1`or `2` for direction, now it's writted "Unknown", "Inbound", "Outbound".
    
    * Logging: Add 2 decimals for timestamt in text and JSON logs.
    
    * Improve "no valid peers" logging.
    
    * Improve "Some columns have no peers responsible for custody" logging.
    
    * `pubsubSubscriptionRequestLimit`: Increase to be consistent with data columns.
    
    * `sendPingRequest`: Improve logging.
    
    * `FindPeersWithSubnet`: Regularly recheck in our current set of peers if we have enough peers for this topic.
    
    Before this commit, new peers HAD to be found, even if current peers are eventually acceptable.
    For very small network, it used to lead to infinite search.
    
    * `subscribeDynamicWithSyncSubnets`: Use exactly the same subscription function initially and every slot.
    
    * Make deepsource happier.
    
    * Nishant's commend: Change peer disconnected log.
    
    * NIshant's comment: Change `Too many incoming subscription` log from error to debug.
    
    * `FindPeersWithSubnet`: Address Nishant's comment.
    
    * `batchSize`: Address Nishant's comment.
    
    * `pingPeers` ==> `pingPeersAndLogEnr`.
    
    * Update beacon-chain/sync/subscriber.go
    
    Co-authored-by: Nishant Das <[email protected]>
    
    ---------
    
    Co-authored-by: Nishant Das <[email protected]>
    nalepae and nisdas committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    d4b3a65 View commit details
    Browse the repository at this point in the history
  58. Revert "Change Custody Count to Uint8 (#14386)" (#14415)

    This reverts commit bd7ec3f.
    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    db424df View commit details
    Browse the repository at this point in the history
  59. Fix CPU usage in small devnets (#14446)

    * `CustodyCountFromRemotePeer`: Set happy path in the outer scope.
    
    * `FindPeersWithSubnet`: Improve logging.
    
    * `listenForNewNodes`: Avoid infinite loop in a small subnet.
    
    * Address Nishant's comment.
    
    * FIx Nishant's comment.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    4bf2c27 View commit details
    Browse the repository at this point in the history
  60. Peerdas: Full subnet sampling and sendBatchRootRequest fix. (#14452)

    * `sendBatchRootRequest`: Refactor and add comments.
    
    * `sendBatchRootRequest`: Do send requests to peers that custodies a superset of our columns.
    
    Before this commit, we sent "data columns by root requests" for data columns peers do not custody.
    
    * Data columns: Use subnet sampling only.
    
    (Instead of peer sampling.)
    
    aaa
    
    * `areDataColumnsAvailable`: Improve logs.
    
    * `GetBeaconBlock`: Improve logs.
    
    Rationale: A `begin` log should always be followed by a `success` log or a `failure` log.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    6b1b04e View commit details
    Browse the repository at this point in the history
  61. Peerdas: Several updates (#14459)

    * `validateDataColumn`: Refactor logging.
    
    * `dataColumnSidecarByRootRPCHandler`: Improve logging.
    
    * `isDataAvailable`: Improve logging.
    
    * Add hidden debug flag: `--data-columns-reject-slot-multiple`.
    
    * Add more logs about peer disconnection.
    
    * `validPeersExist` --> `enoughPeersAreConnected`
    
    * `beaconBlocksByRangeRPCHandler`: Add remote Peer ID in logs.
    
    * Stop calling twice `writeErrorResponseToStream` in case of rate limit.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    5ca87d5 View commit details
    Browse the repository at this point in the history
  62. Update c-kzg-4844 to v2.0.1 (#14421)

    jtraglia authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    4675f9e View commit details
    Browse the repository at this point in the history
  63. PeerDAS: Multiple improvements (#14467)

    * `scheduleReconstructedDataColumnsBroadcast`: Really minor refactor.
    
    * `receivedDataColumnsFromRootLock` -> `dataColumnsFromRootLock`
    
    * `reconstructDataColumns`: Stop looking into the DB to know if we have some columns.
    
    Before this commit:
    Each time we receive a column, we look into the filesystem for all columns we store.
    ==> For 128 columns, it looks for 1 + 2 + 3 + ... + 128 = 128(128+1)/2 = 8256 files look.
    
    Also, as soon as a column is saved into the file system, then if, right after, we
    look at the filesystem again, we assume the column will be available (strict consistency).
    It happens not to be always true.
    
    ==> Sometimes, we can reconstruct and reseed columns more than once, because of this lack of filesystem strict consistency.
    
    After this commit:
    We use a (strictly consistent) cache to determine if we received a column or not.
    ==> No more consistency issue, and less stress for the filesystem.
    
    * `dataColumnSidecarByRootRPCHandler`: Improve logging.
    
    Before this commit, logged values assumed that all requested columns correspond to
    the same block root, which is not always the case.
    
    After this commit, we know which columns are requested for which root.
    
    * Add a log when broadcasting a data column.
    
    This is useful to debug "lost data columns" in devnet.
    
    * Address Nishant's comment
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    d9a1f96 View commit details
    Browse the repository at this point in the history
  64. PeerDAS: Re-enable full node joining the main fork (#14475)

    * `columnErrBuilder`: Uses `Wrap` instead of `Join`.
    
    Reason: `Join` makes a carriage return. The log is quite unreadable.
    
    * `validateDataColumn`: Improve log.
    
    * `areDataColumnsAvailable`: Improve log.
    
    * `SendDataColumnSidecarByRoot` ==> `SendDataColumnSidecarsByRootRequest`.
    
    * `handleDA`: Refactor error message.
    
    * `sendRecentBeaconBlocksRequest` ==> `sendBeaconBlocksRequest`.
    
    Reason: There is no notion at all of "recent" in the function.
    
    If the caller decides to call this function only with "recent" blocks, that's fine.
    However, the function itself will know nothing about the "recentness" of these blocks.
    
    * `sendBatchRootRequest`: Improve comments.
    
    * `sendBeaconBlocksRequest`: Avoid `else` usage and use map of bool instead of `struct{}`.
    
    * `wrapAndReportValidation`: Remove `agent` from log.
    
    Reason: This prevent the log to hold on one line, and it is not really useful to debug.
    
    * `validateAggregateAndProof`: Add comments.
    
    * `GetValidCustodyPeers`: Fix typo.
    
    * `GetValidCustodyPeers` ==> `DataColumnsAdmissibleCustodyPeers`.
    
    * `CustodyHandler` ==> `DataColumnsHandler`.
    
    * `CustodyCountFromRemotePeer` ==> `DataColumnsCustodyCountFromRemotePeer`.
    
    * Implement `DataColumnsAdmissibleSubnetSamplingPeers`.
    
    * Use `SubnetSamplingSize` instead of `CustodySubnetCount` where needed.
    
    * Revert "`wrapAndReportValidation`: Remove `agent` from log."
    
    This reverts commit 55db351.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    4d3e389 View commit details
    Browse the repository at this point in the history
  65. PeerDAS: Misc improvements (#14482)

    * `retrieveMissingDataColumnsFromPeers`: Improve logging.
    
    * `dataColumnSidecarByRootRPCHandler`: Stop decreasing peer's score if asking for a column we do not custody.
    
    * `dataColumnSidecarByRootRPCHandler`: If a data column is unavailable, stop waiting for it.
    
    This behaviour was useful for peer sampling.
    Now, just return the data column if we store it.
    If we don't, skip.
    
    * Dirty code comment.
    
    * `retrieveMissingDataColumnsFromPeers`: Improve logs.
    
    * `SendDataColumnsByRangeRequest`: Improve logs.
    
    * `dataColumnSidecarsByRangeRPCHandler`: Improve logs.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    a62d616 View commit details
    Browse the repository at this point in the history
  66. Configuration menu
    Copy the full SHA
    1d9c41f View commit details
    Browse the repository at this point in the history
  67. Put Subscriber in Goroutine (#14486)

    nisdas authored and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    679737b View commit details
    Browse the repository at this point in the history
  68. PeerDAS: Fix initial sync (#14494)

    * `BestFinalized`: Refactor (no functional change).
    
    * `BestNonFinalized`: Refactor (no functional change).
    
    * `beaconBlocksByRangeRPCHandler`: Remove useless log.
    
    The same is already printed at the start of the function.
    
    * `calculateHeadAndTargetEpochs`: Avoid `else`.
    
    * `ConvertPeerIDToNodeID`: Improve error.
    
    * Stop printing noisy "peer should be banned" logs.
    
    * Initial sync: Request data columns from peers which:
    - custody a superset of columns we need, and
    - have a head slot >= our target slot.
    
    * `requestDataColumnsFromPeers`: Shuffle peers before requesting.
    
    Before this commit, we always requests peers in the same order,
    until one responds something.
    Without shuffling, we always requests data columns from the same
    peer.
    
    * `requestDataColumnsFromPeers`: If error from a peer, just log the error and skip the peer.
    
    * Improve logging.
    
    * Fix tests.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    190a27a View commit details
    Browse the repository at this point in the history
  69. PeerDAS: Fix initial sync with super nodes (#14495)

    * Improve logging.
    
    * `retrieveMissingDataColumnsFromPeers`: Limit to `512` items per request.
    
    * `retrieveMissingDataColumnsFromPeers`: Allow `nil` peers.
    
    Before this commit:
    If, when this funcion is called, we are not yet connected to enough peers, then `peers` will be possibly not be satisfaying,
    and, if new peers are connected, we will never see them.
    
    After this commit:
    If `peers` is `nil`, then we regularly check for all connected peers.
    If `peers` is not `nil`, then we use them.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    f61938c View commit details
    Browse the repository at this point in the history
  70. PeerDAS: Improve initial sync logs (#14496)

    * `retrieveMissingDataColumnsFromPeers`: Search only for needed peers.
    
    * Improve logging.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    6535ace View commit details
    Browse the repository at this point in the history
  71. Fix Commitments Check (#14493)

    * Fix Commitments Check
    
    * `highestFinalizedEpoch`: Refactor (no functional change).
    
    * `retrieveMissingDataColumnsFromPeers`: Fix logs.
    
    * `VerifyDataColumnSidecarKZGProofs`: Optimise with capacity.
    
    * Save data columns when initial syncing.
    
    * `dataColumnSidecarsByRangeRPCHandler`: Add logs when a request enters.
    
    * Improve logging.
    
    * Improve logging.
    
    * `peersWithDataColumns: Do not filter any more on peer head slot.
    
    * Fix Nishant's comment.
    
    ---------
    
    Co-authored-by: Manu NALEPA <[email protected]>
    nisdas and nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    9ecf085 View commit details
    Browse the repository at this point in the history
  72. Configuration menu
    Copy the full SHA
    9c67a71 View commit details
    Browse the repository at this point in the history
  73. PeerDAS: Fix major bug in dataColumnSidecarsByRangeRPCHandler and a…

    …llow syncing from full nodes. (#14532)
    
    * `validateDataColumnsByRange`: `current` ==> `currentSlot`.
    
    * `validateRequest`: Extract `remotePeer` variable.
    
    * `dataColumnSidecarsByRangeRPCHandler`: Small non functional refactor.
    
    * `streamDataColumnBatch`: Fix major bug.
    
    Before this commit, the node was unable to respond with a data column index higher than the count of stored data columns.
    For example, if there is 8 data columns stored for a given block, the node was
    able to respond for data columns indices 1, 3, and 5, but not for 10, 16 or 127.
    
    The issue was visible only for full nodes, since super nodes always store 128 data columns.
    
    * Initial sync: Fetch data columns from all peers.
    (Not only from supernodes.)
    
    * Nishant's comment: Fix `lastSlot` and `endSlot` duplication.
    
    * Address Nishant's comment.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    dfdb265 View commit details
    Browse the repository at this point in the history
  74. streamDataColumnBatch: Sort columns by index. (#14542)

    https://github.com/ethereum/consensus-specs/blob/dev/specs/_features/eip7594/p2p-interface.md#datacolumnsidecarsbyrange-v1
    
    The following data column sidecars, where they exist, MUST be sent in (slot, column_index) order.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    083f59b View commit details
    Browse the repository at this point in the history
  75. PeerDAS: Batch columns verifications (#14559)

    * `ColumnAlignsWithBlock`: Split lines.
    
    * Data columns verifications: Batch
    
    * Remove completely `DataColumnBatchVerifier`.
    
    Only `DataColumnsVerifier` (with `s`) on columns remains.
    It is the responsability of the function which receive the data column
    (either by gossip, by range request or by root request) to verify the
    data column wrt. corresponding checks.
    
    * Fix Nishant's comment.
    nalepae committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    2ab8c3b View commit details
    Browse the repository at this point in the history