Have meaningful exceptions savvy of the Node 19 test runner (node:test)

[jammi]

Currently, when using chai with node:test runners of the latest node v19 releases, the exceptions of expect and such are pretty bad, with no good idea of what actually failed. So considering the example here:

import test from 'node:test';
import assert from 'node:assert';
import { expect } from 'chai';

await test('chai example', async t => {
  await t.test('subtest that is ok', () => {
    expect(true)
      .to.be.a('boolean')
      .that.equals(true);
  });
  await t.test('subtest that fails', () => {
    expect({foo: 'hello'})
      .to.be.an('object')
      .that.has.property('foo')
      .that.is.a('number', 'should be a number')
      .that.equals(123);
  });
});

await test('node:assert example', async t => {
  await t.test('subtest that is ok', () => {
    assert.equal(typeof true, 'boolean');
    assert.equal(true, true);
  });
  await t.test('subtest that fails', () => {
    const obj = {foo: 'hello'};
    assert.equal(typeof obj, 'object');
    assert.ok(obj?.foo, 'has property "foo"');
    assert.equal(typeof obj.foo, 'number', 'should be a number');
    assert.equal(obj.foo, 123);
  });
});

..and then see how much better node's own assert library throws, although not perfectly.

❯ node --test --test-reporter=spec test/*-spec.mjs
▶ ./test/eslint-spec.mjs
  ▶ Running eslint validation
    ✔ expecting 'index.mjs' to lint cleanly (1.748583ms)
    ✔ expecting 'lib/sourcefile.mjs' to lint cleanly (0.044084ms)
    ✔ expecting 'lib/another-source.mjs' to lint cleanly (0.036041ms)
    ✔ expecting 'lib/more-stuff.mjs' to lint cleanly (0.036542ms)
    ✔ expecting 'lib/yet-another.mjs' to lint cleanly (0.034417ms)
    ✔ expecting 'test/eslint-spec.mjs' to lint cleanly (0.033958ms)
    ✔ expecting 'test/sample-spec.mjs' to lint cleanly (0.030375ms)
  ▶ Running eslint validation (210.977042ms)

▶ ./test/eslint-spec.mjs (369.277041ms)

▶ ./test/sample-spec.mjs
  ▶ chai example
    ✔ subtest that is ok (2.068292ms)
    ✖ subtest that fails (0.637042ms)
          at TestContext.<anonymous> (file://./test/sample-spec.mjs:15:16)
          at Test.runInAsyncScope (node:async_hooks:203:9)
          at Test.run (node:internal/test_runner/test:549:25)
          at Test.start (node:internal/test_runner/test:465:17)
          at TestContext.test (node:internal/test_runner/test:136:20)
          at TestContext.<anonymous> (file://./test/sample-spec.mjs:11:11)
          at async Test.run (node:internal/test_runner/test:550:9)
          at async file://./test/sample-spec.mjs:5:1 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: 'hello',
        expected: undefined,
        operator: 'strictEqual'
      }

  ▶ chai example (3.725833ms)

  ▶ node:assert example
    ✔ subtest that is ok (0.068667ms)
    ✖ subtest that fails (0.317875ms)
          at TestContext.<anonymous> (file://./test/sample-spec.mjs:29:12)
          at Test.runInAsyncScope (node:async_hooks:203:9)
          at Test.run (node:internal/test_runner/test:549:25)
          at Test.start (node:internal/test_runner/test:465:17)
          at TestContext.test (node:internal/test_runner/test:136:20)
          at TestContext.<anonymous> (file://./test/sample-spec.mjs:25:11)
          at async Test.run (node:internal/test_runner/test:550:9)
          at async file://./test/sample-spec.mjs:20:1 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: 'string',
        expected: 'number',
        operator: '=='
      }

  ▶ node:assert example (0.508833ms)

▶ ./test/sample-spec.mjs (74.725333ms)

[jammi]

I used Chai previously with Mocha, which has its own problems, especially when writing mostly async code. However, when a test throws in Mocha, it's obvious why something failed. I tried evaluating other BDD "expect" libs such as Jasmine and Jest, but I love the Chai syntax, and of course I have the routine from years of using it, so I'd love some solution to this, even if it was some temporary wrapper thing.

[keithamus]

That is very bad 😞. It's hard to tell exactly what is going wrong but I imagine our actual/expected properties aren't correctly updating based on the chain. This is likely going to need a fairly significant refactor for Chai.

What I'm a little confused over is why node:test does not print out the message of the error. That has all pertinent info.

[cjihrig]

I think this may have gotten resolved. Running the repro today I see the following output for chai:

▶ chai example
  ✔ subtest that is ok (0.7715ms)
  ✖ subtest that fails (0.618583ms)
    Error [AssertionError]: should be a number: expected 'hello' to be a number
        at TestContext.<anonymous> (file:///private/tmp/bug.mjs:15:16)
        at Test.runInAsyncScope (node:async_hooks:206:9)
        at Test.run (node:internal/test_runner/test:865:25)
        at Test.start (node:internal/test_runner/test:762:17)
        at TestContext.test (node:internal/test_runner/test:310:20)
        at TestContext.<anonymous> (file:///private/tmp/bug.mjs:11:11)
        at async Test.run (node:internal/test_runner/test:866:9)
        at async Test.processPendingSubtests (node:internal/test_runner/test:574:7) {
      actual: 'hello',
      expected: undefined,
      showDiff: false,
      operator: 'strictEqual'
    }

▶ chai example (2.105167ms)