Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve own test coverage and remove brittle expected_mutants_for_own_source_tree #296

Merged
merged 32 commits into from
Apr 7, 2024

Conversation

sourcefrog
Copy link
Owner

@sourcefrog sourcefrog commented Feb 26, 2024

This brittle test was hiding some mutants that were not otherwise well tested: coverage is better with the additions in this PR.

Also, this test needed to be updated on many commits, which was tedious and in hindsight a bad smell.

Fixes #295

@sourcefrog sourcefrog force-pushed the 295-own-tree-test branch 2 times, most recently from a0238fe to 106095b Compare February 27, 2024 06:30
@sourcefrog sourcefrog changed the title Remove expected_mutants_for_own_source_tree Improve own test coverage and remove brittle expected_mutants_for_own_source_tree Feb 28, 2024
@sourcefrog sourcefrog force-pushed the 295-own-tree-test branch 2 times, most recently from 839c5a0 to fe1dd62 Compare March 2, 2024 15:05
@sourcefrog
Copy link
Owner Author

There are now some timeouts in 93a850f:

src/outcome.rs:224:9: replace ScenarioOutcome::check_or_build_failed -> bool with false
src/outcome.rs:231:9: replace ScenarioOutcome::mutant_caught -> bool with false
src/outcome.rs:232:34: replace == with != in ScenarioOutcome::mutant_caught
src/process.rs:50:9: replace Process::run -> Result<ProcessStatus> with Err(::anyhow::anyhow!("mutated!"))
src/process.rs:129:9: replace Process::terminate -> Result<()> with Err(::anyhow::anyhow!("mutated!"))
src/process.rs:216:9: replace ProcessStatus::is_failure -> bool with false
src/scenario.rs:28:9: replace Scenario::is_mutant -> bool with false

(possibly more)

They seem to all originate in tree_with_symlink in build_dir.rs...

Perhaps this is because they're all running in the source tree, and having trouble locking files there?

@sourcefrog
Copy link
Owner Author

Trying to bisect for the change that started this:

git bisect start
# bad: [ace8b702e6e899df3218e8cf878604fe340d2153] Unviable mutation only needs a check build
git bisect bad ace8b702e6e899df3218e8cf878604fe340d2153
# good: [75a504cbf41017a8f22a9ec7d654428be4c2b610] Don't fail fast on mutants matrix (#297)
git bisect good 75a504cbf41017a8f22a9ec7d654428be4c2b610
# good: [b75f14491ec90a270521825e338008f760d1f367] Split out and improve tests for --check
git bisect good b75f14491ec90a270521825e338008f760d1f367

@sourcefrog
Copy link
Owner Author

Apparently one cause of the hangs was sourcefrog/nutmeg#16, a cute deadlock in Nutmeg, avoided for now by not sending trace messages while rendering progress.

@sourcefrog sourcefrog merged commit 7901e6d into main Apr 7, 2024
35 checks passed
@sourcefrog sourcefrog deleted the 295-own-tree-test branch April 7, 2024 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

"mutants in own tree" test might make tests too brittle for other self-tests
1 participant