Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky tests in PhoneNumberBuilderTest #200

Closed
wants to merge 1 commit into from

Conversation

aditya-kumbhar
Copy link

Motivation

PhoneNumberBuilderTest has the following tests that are flaky, detected using the NonDex tool:
org.apache.directory.scim.spec.resources.PhoneNumberBuilderTest#test_adding_params_for_GlobalPhoneNumberBuilder
org.apache.directory.scim.spec.resources.PhoneNumberBuilderTest#test_adding_params_for_LocalPhoneNumberBuilder

The tests fail due to non-deterministic ordering in the HashMap params of the PhoneNumber class.

Command to reproduce

mvn -pl scim-spec/scim-spec-schema edu.illinois:nondex-maven-plugin:2.1.1:nondex -Denforcer.skip=true -Dtest=org.apache.directory.scim.spec.resources.PhoneNumberBuilderTest

Assertions that fail

assertEquals(("tel:+1-888-888-5555;ext=1234;milhouse=simpson;example=gh234"), phoneNumber.getValue());
assertEquals(("tel:888-5555;isub=example.a.com;phone-context=+1-888;milhouse=simpson;example=gh234"), phoneNumber.getValue());

Log

[ERROR] Failures: 
[ERROR]   PhoneNumberBuilderTest.test_adding_params_for_GlobalPhoneNumberBuilder:712 expected: <tel:+1-888-888-5555;ext=1234;milhouse=simpson;example=gh234> but was: <tel:+1-888-888-5555;ext=1234;example=gh234;milhouse=simpson>
[ERROR]   PhoneNumberBuilderTest.test_adding_params_for_LocalPhoneNumberBuilder:728 expected: <tel:888-5555;[isub=example.a.com](http://isub%3Dexample.a.com/);phone-context=+1-888;milhouse=simpson;example=gh234> but was: <tel:888-5555;[isub=example.a.com](http://isub%3Dexample.a.com/);phone-context=+1-888;example=gh234;milhouse=simpson>

Changes

The non-deterministic part of phoneNumber.getValue() is the params, so assert that phoneNumber.getValue()contains the params in such a way that the order of the params is not assumed.

Another way to fix the tests is to change the datatype of param from HashMap to LinkedHashMap in the PhoneNumber class as LinkedHashMap maintains the order of its elements - but this would require changing the main code.

bdemers added a commit that referenced this pull request Dec 8, 2022
This should possibly be sorted map instead of an ordered one, but starting with predictable output is still an improvement.

Fixes: #200
@bdemers
Copy link
Member

bdemers commented Dec 8, 2022

Thanks @aditya-kumbhar!

I when the LinkedHashMap route, but added a note that it could be a SortedMap in the future.

nondex is a pretty cool Maven plugin! I tried running it on the rest of the project but encountered problems.
The biggest one is it wasn't obvious how to debug the test run. Any chance you can add an example to the README/wiki of the project that shows how to run a test and attach a debugger? (or even better, run the instrumented test from an IDE like IntelliJ)

Updating the plugin to work with Java 17+ is critical too.

@bdemers bdemers closed this Dec 8, 2022
@aditya-kumbhar
Copy link
Author

aditya-kumbhar commented Dec 11, 2022

Thanks @bdemers for the feedback!
Nondex will be updated to work with 17+ in a later version.

Regarding debugging/attaching a debugger while using Nondex, this method works for me (in IntelliJ):

  1. Add remote debug configuration in IntelliJ with port 5005.
  2. Add debug breakpoints.
  3. Run mvn nondex with -Dmaven.surefire.debug. For example, to run the command in this PR in debug mode:
    mvn -pl scim-spec/scim-spec-schema edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dmaven.surefire.debug -Denforcer.skip=true -Dtest=org.apache.directory.scim.spec.resources.PhoneNumberBuilderTest
  4. Run the remote debugger.
    The mvn command should attach to the IntelliJ debugger.

Another note: to reproduce a flaky test detected using Nondex without having to run all the Nondex iterations again, we can use the reported Nondex seed and pass it as a parameter.
For example, if the reported Nondex seed is 933178, following command can be used to replicate the Nondex configuration and reproduce the test failure:
mvn -pl scim-spec/scim-spec-schema edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dmaven.surefire.debug -Dtest=org.apache.directory.scim.spec.resources.PhoneNumberBuilderTest -DnondexSeed=933178

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants