APM Server 8.16 test plan

[kruskall]

Manual Test Plan

List of changes: 8.15...8.16

Smoke Testing ESS setup

Thanks to #8303 further smoke tests are run automatically on ESS now.
Consider extending the smoke tests to include more test cases which we'd like to cover

go-docappender library

• Add dynamic templates to bulk indexer item go-docappender#189 [N/A]

apm-data library

• Fix OTEL Exception Message Propagation apm-data#341 [@inge4pres]
• Remove memory pooling apm-data#358 [N/A]
• Stop generating code for pooling apm-data#359 [N/A]

opentelemetry-lib library

• fix: udate hostmetrics scrapers names opentelemetry-lib#90 [@inge4pres ]

Test cases from the GitHub board

• fix: only retry when receiving a 429 status code #13523 [@endorama ]
• Jaeger Deprecation #13809 [@marclop ]
• [8.x] Fix concurrent map write panic in monitoring middleware (backport #14335) #14350 [@1pkg ]
• [8.x] Apply shutdown timeout to http server to limit reload delay (backport #14339) #14362 [@rubvs ]
• [8.16] Limit badger gc concurrency to 1 to avoid panic (backport #14340) #14404 [@kruskall ]
• monitoring: enable tracing self instrumentation in APM Server #14231 [@lahsivjar ]

Regressions

[simitt]

@kruskall IMO we do not need to test the changes in the opentelemetry-lib; confirmed by @lahsivjar

in the test plan, only https://github.com/elastic/opentelemetry-lib/pull/90 is needed to be tested if it hasn't already been tested in the testathon.

[marclop]

Marked elastic/go-docappender#189 as done since it doesn't need to be tested. DynamicTemplates is only exposed in BulkIndexer. APM Server doesn't use this newly added field, nor does the Appender by default.

Marked elastic/apm-data#358 and elastic/apm-data#359 as done since I don't think they need any manual testing.

[endorama]

I added a unit test for the behavior changes around retry in #13523. PR here: #14433

Does this suffice for considering it done for the test plan?

[1pkg]

Similar to above, I had to add an additional concurrent auto test for #14335 in PR here #14443 since it's hard to reproduce manually.

[rubvs]

Was able to reproduce #14339 via the comments in #14337. However, to see the double in gc ticks, I had to replace time.Sleep(time.Minute) with an uneven count like time.Sleep(5*time.Second)

[lahsivjar]

Successfully tested #14231, details here.

[inge4pres]

Tested elastic/opentelemetry-lib#90 with elastic/opentelemetry-lib#90 (comment)

[endorama]

PR #14433 merged, marking 429 changes tested.

[kruskall]

Tested #14404 thanks to the comment in #14337

Thanks everyone!