[Reporting bug] Tests that passed on retry are not being reported accurately

[ravimandala]

I noticed an issue in Bluepill version 5.10.0 that supports Xcode 13. In a successful execution, 11 tests are reported to have failed. But the execution was successful because those tests passed on retry. Usually when tests pass on retry it should be reported accordingly in 0-test-report.html, which doesn’t seem to be the case in this execution. This might be an edge case specific to the latest version. I will try to get more details or a test case to reproduce this.

[jschear]

I saw similar behavior, and the root cause was a small bug in the collation of test results: #515

Are you seeing invalid XML files created in any of your executions? If an execution produces a Junit XML file that isn't a valid XML document (which we were seeing when a simulator failed to boot; the file contained just the string "NO LOG: JUnitReporter"), other (perfectly valid) test reports would be treated as invalid, and not merged into the final combined report.

[ravimandala]

True. I see the executions where this did happen and I see how it could skip the subsequent reports that might be perfectly valid. Thanks for the fix.

[ravimandala]

@jschear Your PR(#515) might fix this issue. I will resolve this issue after verifying.

[cccCody]

This doesn't seem to be fixed. I'm using 5.11, and I see results for each individual attempt in the xml, so if a test fails, retries, and then passes, I still have a failure in the output. There's an old open PR to add a flag for this here: #467

...and one comment even mentions making it the default behavior, but that doesn't seem to have happened.