Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

moving slow tests to nayduck - bls12381 #12456

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Conversation

wacban
Copy link
Contributor

@wacban wacban commented Nov 14, 2024

The bls fuzzer tests are really slow. This slows down and increases the cost of CI, and is generally annoying. Moving those to nayduck.

  • marked the bls fuzzer tests as expensive
  • added them to nayduck

The tests are implemented as some crazy macros so I'm not exactly sure what I'm doing is right.

Here are the tests when executed in nayduck - https://nayduck.nearone.org/#/run/705
I also checked that they can be run locally and it does with a small fix: ./scripts/nayduck.py -t nightly/bls12381.txt -l

Due to some features fun I needed to hackily add prepare feature to neard. I'm looking for suggestion how to do it better. This is because nayduck actually builds neard with the same features as for the test. The neard binary is not used, the test is then run using cargo e.g.

cargo test -pnear-vm-runner --features prepare,test_features,expensive_tests -- --exact --nocapture logic::tests::bls12381::tests::test_bls12381_g1_multiexp_incorrect_input_fuzzer

@wacban wacban requested a review from nagisa November 14, 2024 12:58
@wacban wacban requested a review from a team as a code owner November 14, 2024 12:58
@wacban
Copy link
Contributor Author

wacban commented Nov 14, 2024

cc @olga24912 Can you also have a look? I couldn't add you to reviewers for whatever reason.

Copy link

codecov bot commented Nov 14, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.43%. Comparing base (fecce01) to head (fc06f36).

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #12456      +/-   ##
==========================================
- Coverage   71.44%   71.43%   -0.02%     
==========================================
  Files         838      838              
  Lines      169339   169340       +1     
  Branches   169339   169340       +1     
==========================================
- Hits       120985   120965      -20     
- Misses      43006    43026      +20     
- Partials     5348     5349       +1     
Flag Coverage Δ
backward-compatibility 0.16% <ø> (ø)
db-migration 0.16% <ø> (ø)
genesis-check 1.29% <ø> (ø)
integration-tests 39.33% <0.00%> (-0.01%) ⬇️
linux 70.77% <100.00%> (-0.02%) ⬇️
linux-nightly 70.99% <100.00%> (-0.01%) ⬇️
macos 51.01% <100.00%> (-0.01%) ⬇️
pytests 1.60% <ø> (ø)
sanity-checks 1.40% <ø> (ø)
unittests 63.96% <100.00%> (-0.01%) ⬇️
upgradability 0.21% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@nagisa
Copy link
Collaborator

nagisa commented Nov 14, 2024

Hm, locally for me these tests are all <15s each and even in the most recent CI run I picked up the majority of them finish well within 30s which I don't think warrants these getting moved all the way to nayduck. But I otherwise agree these don't need to run locally.

How long is super long in your experience?

@wacban
Copy link
Contributor Author

wacban commented Nov 15, 2024

@nagisa Yeah I agree those aren't too bad each by itself. I was more concerned with the fact that there is like 30 of them. Now looking at the test run in CI, the majority of time is spent on building and other arbitrary things, not the test execution itself. Up to you, I still think those are slow enough and stable enough to put them in nayduck but I'm happy to abandon it also.

Extra argument for nayduck is that those are fuzzers but I'm not sure how relevant that is.

@nagisa
Copy link
Collaborator

nagisa commented Nov 15, 2024

Extra argument for nayduck is that those are fuzzers

This is somewhat misleading -- my understanding of bolero::check!() is that it will only run a few iterations of the code tested to ensure it works at all, kind of acting as an integration test for the fuzzer and the code tested therein. And the proper fuzzing would only run in our clusterfuzz infrastructure.

@wacban
Copy link
Contributor Author

wacban commented Nov 15, 2024

btw feel free to merge your PR first and I'll see if there is anything left to do here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants