RmiRegistrySslTest fails on JDK 8 during SSL lookup with handshake_failure

[jiekang]

Seen during 2024 October CPU (#5692)

RmiRegistrySslTest is failing on all platforms with logs similar to:

RmiRegistrySslTest : SSL RMIRegistry - SSL Lookup
08:45:43  -------------------------------------------------------------
08:45:43  RmiRegistry lookup...
08:45:43  KO: Got unexpected exception!
08:45:43  java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
08:45:43  	javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
08:45:43  	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:307)
08:45:43  	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
08:45:43  	at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)
08:45:43  	at sun.rmi.registry.RegistryImpl_Stub.list(RegistryImpl_Stub.java:90)
08:45:43  	at RmiRegistrySslTest.main(RmiRegistrySslTest.java:75)
08:45:43  Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
08:45:43  	at sun.security.ssl.Alert.createSSLException(Alert.java:131)
08:45:43  	at sun.security.ssl.Alert.createSSLException(Alert.java:117)
08:45:43  	at sun.security.ssl.TransportContext.fatal(TransportContext.java:318)
08:45:43  	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
08:45:43  	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185)
08:45:43  	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152)
08:45:43  	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1401)
08:45:43  	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1309)
08:45:43  	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
08:45:43  	at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:822)
08:45:43  	at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:73)
08:45:43  	at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1184)
08:45:43  	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
08:45:43  	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
08:45:43  	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
08:45:43  	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229)
08:45:43  	... 4 more

[jiekang]

Also related: RmiSslBootstrapTest is failing on a lot of JDK 8 platforms with:

[2024-10-17T12:45:43.422Z] Connection failed for {measure, QED}: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
[2024-10-17T12:45:43.423Z] 	javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
[2024-10-17T12:45:43.423Z] Connection failed for {modify, R&D}: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
[2024-10-17T12:45:43.423Z] 	javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
[2024-10-17T12:45:43.423Z] Connection failed as expected for {bad.user, R&D}: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
[2024-10-17T12:45:43.423Z] 	javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
[2024-10-17T12:45:43.423Z] Connection failed as expected for {measure, bad.password}: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
[2024-10-17T12:45:43.423Z] 	javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
[2024-10-17T12:45:43.423Z] Test Failed: Failed to connect to agent {url=service:jmx:rmi://e9a48cb84f6b/jndi/rmi://localhost:36732/jmxrmi}: java.lang.RuntimeException: Test /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_aarch64_linux_rerun/aqa-tests/TKG/output_17291679023488/jdk_management_0/work/classes/2/sun/management/jmxremote/bootstrap/management_ssltest06_ok.properties failed with 2 error(s)
[2024-10-17T12:45:43.423Z] STDERR:
[2024-10-17T12:45:43.423Z] /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_aarch64_linux_rerun/aqa-tests/openjdk/openjdk-jdk/jdk/test/sun/management/jmxremote/bootstrap/RmiSslBootstrapTest.sh: 37: /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_aarch64_linux_rerun/aqa-tests/openjdk/openjdk-jdk/jdk/test/sun/management/jmxremote/bootstrap/GeneratePropertyPassword.sh: [[: not found

[jiekang]

Related test code has seen one recent change in August 2024, the backport of JDK-8145919
openjdk/jdk8u@09bab6e

From history:
https://github.com/openjdk/jdk8u/commits/master/jdk/test/sun/management/jmxremote/bootstrap

[jiekang]

The JDK-8145919 backport mentioned required followup PR for JDK-8205653, however this was not merged in time for this release:

openjdk/jdk8u-dev#548

[jiekang]

I strongly suspect JDK-8205653 is required for these tests to pass properly.

Based on above analysis, I would not consider these test failures as release blockers.

[jiekang]

I suppose the most correct thing to do is run a build with openjdk/jdk8u-dev#548 incorporated to see if the test passes.

[sophia-guo]

Upstream fix openjdk/jdk8u-dev#548, currently excluded in aqa #5744

[sophia-guo]

When upstream fix is in, will verify and re-include the tests.

[smlambert]

We have excluded it under this upstream issue https://bugs.openjdk.org/browse/JDK-8205653 (not sure if it should have been https://bugs.openjdk.org/browse/JDK-8145919, but the point is, we intend to run our disabledTests script with the specific options that allow us to check the state of the issue the test is excluded under (and if "fixed") can advise re-inclusion...

Both JDK-8205653 and JDK-8145919 show as fixed, so is it expected that the EA builds would contain the fix (since we build -dev branch)?

[jiekang]

https://bugs.openjdk.org/browse/JDK-8205653 is not yet merged into upstream OpenJDK 8u; the backports list mentions 8u261 but that is for Oracle releases, the openjdk one would look like 'openjdk8u432'.

We need to wait for openjdk/jdk8u-dev#548 to be merged and part of a tag.