Node Public Interfaces Team Weekly Update - 2021-06-25

[frol]

We are adopting a new template of these updates to align with the rest of the teams.

We have not had an update last week, so the previous update is from two weeks ago: #4361

Brief Summary

The major chunk of the work of Public Interfaces, Explorer, and SRE teams these days target data consistency, development experience, reliability of the data, and availability of the services. All the projects we have been working on for the past two weeks received quality of life improvements that ultimately affect day-to-day users of NEAR.

• Heads up about 1.20.0-rc.1 release scheduled on 2021-06-28 to be released on testnet, and 1.20.0 release scheduled on 2021-07-12 on mainnet [indexer] Get ready to 1.20.0 #4386 (@khorolets)

Results:

• nearcore

  • Exposed structured errors from JSON RPC (@khorolets)
    refactor(jsonrpc): Expose structured errors (VError) #4399
  • Introduced a strictly typed version of AccountId type in order to pave the way to type-safe changes in nearcore (@miraclx)
    master...miraclx:strict-account-id-validity
  • Ensured proper validation of AccountId's during borsh + serde deserialization (@miraclx)
    master...miraclx:strict-account-id-validity

• indexer

  • Heads-up about the nearcore 1.20 release! (see above)
  • NEAR Indexer Framework 0.9.2 release with an optimization tracking the delayed receipts (@khorolets)
  • Made proof of concept of Indexer with Apache Kaffka (@khorolets)

• indexer-for-explorer

  • Introduced --concurrency parameter to be able to control the number of parallel blocks being processed to reduce contention on blocks packed with transactions and receipts (@khorolets)
    refactor: Add --concurrency parameter. Bump version to 0.6.9 near-indexer-for-explorer#126

• near-cli-rs

  • Implemented Ledger Nano S and Nano X support 🎉 (@khorolets)
    Implement Ledger support near-cli-rs#23
  • Released near-ledger crate for Rust 🎉 (@khorolets)

• docs

  • The general doc about lockups is rewritten with new visualizations (@telezhnaya)
    lockup: add vesting info, add graphs with schedule to the doc docs#691

• explorer

  • Added transitioning logic for circulating supply from financial forecasts to real data (@telezhnaya)
    feat(backend): add transition logic for circulating supply near-explorer#642
  • Fixed validator's name and properly display validator's public key (@shelegdmitriy)
    fix(frontend): Fix confusing information on Nodes page near-explorer#657
  • Fixed failing end-to-end tests Fix end-to-end failing tests near-explorer#640 (@shelegdmitriy)
    fix(frontend): Fix failing e2e tests near-explorer#671

• operations

  • Pagerduty users are now managed by Terraform (@chefsale)
  • Created alerts dashbords via template and terraform for testnet (@mhalambek)
  • Cleaned up a bit reindexer terraform configs, still there is stuff to be changed (@chefsale)
  • Fixed broken betanet due to trunk release being broken, reverted the breaking changes and redeployed (@chefsale)
    Revert "Set block protocol version in a separate function in process_… #4363
  • Fixed broken betanet due to an incompatible change in nearup as the neard --verbose flag doesn't support empty values anymore due to clap refactor (@chefsale)
    fix(logging): fix the backwards incompatible issue after clap refactor nearup#177

• meta

  • New Oncall process was announced (@chefsale)
  • Bootcamp is almost over, now waiting to collect feedback and make a retrospective on how can we improve (@chefsale)
  • OKRs are defined for Public Interfaces team (Explorer and SRE teams are yet to be finalized)
  • Conducted two small internal workshops:
    • How to boot localnet and "frolnet" (@frol)
      https://drive.google.com/file/d/19mHnoFXLXC0RDiQASNqvqEH_NcTndbVt/view?usp=sharing
    • Boot "frolnet" and break it (the recording did not record the screen most of the time 🤦, the screen only gets shared on 24th minute) (@frol)
      https://drive.google.com/file/d/1Si5b3d4jT-cPY15aGLBQUqg4LPs_4kqU/view?usp=sharing (edited)

Challenges & Blockers:

• indexer-for-explorer
  • Data consistency due to the old bug leading to some transaction being applied on mainnet twice is still not resolved on Indexer for Explorer side
    Some rows are missing because of transaction hash collisions near-indexer-for-explorer#84 (comment)

Action Items:

• nearcore

  • Apply strictly-typed AccountId throughout the workspace. Revise structures previously using unchecked, owned Strings, empty strings, remove redundant validation down in the weeds (chain, runtime, etc..) (since this happens at AccountId level) and finally make a draft PR (@miraclx)

• indexer-for-explorer

  • Provide the users with historical data about circulating supply. We decided that the solution will be the part of Indexer for Explorer (@telezhnaya)
    Create separate microservice for circulating supply calculation & saving it to DB near-indexer-for-explorer#124
  • [In progress] Reindex from genesis on mainnet to fill up account_changes_new with proper data without duplicates (@khorolets)
    We had some problems with non-optimized delayed receipts tracker

• lockup:

  • Fix tests in the lockup contract and configure CI (@telezhnaya, @evgenykuzyakov, @chefsale)
    [Lockup]: Upgrade to near-sdk version 3.1.0 core-contracts#151
  • Fix code issues in lockup contract and lockup factory found in https://hackmd.io/@ailisp/rJbiDYr5_ (@telezhnaya)

• docs

  • Document the data flowing in NEAR Protocol (@khorolets)
    [DOC] Document the data flowing in NEAR Protocol docs#700

• explorer

  • Merge tabs 'Validators' and 'Proposals' into one tab and apply additional labels and also expose whole pools list (@shelegdmitriy)
    Validator nodes screen is confusing near-explorer#646
  • Display active access keys on Account Details page (@shelegdmitriy)
    Display active access keys on Account Details page near-explorer#509

• operations

  • Refactor a bit more of reindexer terraform (@chefsale)
  • Split dashboards template into per panel, integrate it with pagerduty, maybe even GCP (@mhalambek)

• meta

  • Create the new oncall schedule managed by terraform (@chefsale)
  • OKR plannings and reviews
  • Bootcamp review and next steps (@chefsale)
  • Whatever life or OKRs throws next 😄