SSL verification option is not applied

[immanuelfodor]

I'm running Matrix in Kubernetes with in-cluster TLS encryption enabled, and the bridge is deployed there next to Matrix. When the bridge is configured with the external address of the homeserver (e.g., https://matrix.example.com), the connection is fine as the certificate is valid. However, when I try to use the in-cluster Matrix service (https://matrix:8448), and so turn off the verify_ssl option of the bridge, the bridge is still trying to check the cert's validity as if the verification option wasn't there at all. I also tried adding the service name as another SAN to the cert to circumvent the verification issue but it doesn't help. I'd like to avoid using the external Matrix URL as it means an unnecessary roundtrip for all the packets and depending on external services, so it would be great if the verify_ssl option could be fixed to apply when false.

Bridge startup error log:

 INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.                                                                                                                                                                            
 INFO  [alembic.runtime.migration] Will assume transactional DDL.                                                                                                                                                                          
 [2020-12-16 06:47:56,864] [[email protected][] Initializing mautrix-facebook 0.1.2                                                                                                                                                            
 [2020-12-16 06:47:56,986] [[email protected][] Initialization complete in 2.1 seconds                                                                                                                                                         
 [2020-12-16 06:47:56,986] [[email protected][] Running startup actions...                                                                                                                                                                    
 [2020-12-16 06:47:56,987] [[email protected][] Starting appservice...                                                                                                                                                                        
 [2020-12-16 06:47:56,987] [[email protected][] Starting appservice web server on 0.0.0.0:29319                                                                                                                                                 
 [2020-12-16 06:47:56,987] [[email protected][] Ensuring connectivity to homeserver                                                                                                                                                              
 [2020-12-16 06:47:57,188] [[email protected][] Unexpected error in main event loop                                                                                                                                                        
 Traceback (most recent call last):                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection                                                                                                                                      
     return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa                                                                                                                                                    
   File "/usr/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection                                                                                                                                                       
     transport, protocol = await self._create_connection_transport(                                                                                                                                                                        
   File "/usr/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport                                                                                                                                            
     await waiter                                                                                                                                                                                                                          
   File "/usr/lib/python3.8/asyncio/sslproto.py", line 529, in data_received                                                                                                                                                               
     ssldata, appdata = self._sslpipe.feed_ssldata(data)                                                                                                                                                                                   
   File "/usr/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata                                                                                                                                                                
     self._sslobj.do_handshake()                                                                                                                                                                                                           
   File "/usr/lib/python3.8/ssl.py", line 944, in do_handshake                                                                                                                                                                             
     self._sslobj.do_handshake()                                                                                                                                                                                                           
 ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)                                                                                            
 The above exception was the direct cause of the following exception:                                                                                                                                                                      
 Traceback (most recent call last):                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/mautrix/api.py", line 233, in request                                                                                                                                                            
     return await self._send(method, full_url, content, query_params, headers or {})                                                                                                                                                       
   File "/usr/lib/python3.8/site-packages/mautrix/api.py", line 149, in _send                                                                                                                                                              
     async with request as response:                                                                                                                                                                                                       
   File "/usr/lib/python3.8/site-packages/aiohttp/client.py", line 1012, in __aenter__                                                                                                                                                     
     self._resp = await self._coro                                                                                                                                                                                                         
   File "/usr/lib/python3.8/site-packages/aiohttp/client.py", line 480, in _request                                                                                                                                                        
     conn = await self._connector.connect(                                                                                                                                                                                                 
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 523, in connect                                                                                                                                                      
     proto = await self._create_connection(req, traces, timeout)                                                                                                                                                                           
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 858, in _create_connection                                                                                                                                           
     _, proto = await self._create_direct_connection(                                                                                                                                                                                      
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection                                                                                                                                   
     raise last_exc                                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _create_direct_connection                                                                                                                                    
     transp, proto = await self._wrap_create_connection(                                                                                                                                                                                   
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 938, in _wrap_create_connection                                                                                                                                      
     raise ClientConnectorCertificateError(                                                                                                                                                                                                
 aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host matrix:8448 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate 
  (_ssl.c:1123)')]                                                                                                                                                                                                                         
 The above exception was the direct cause of the following exception:                                                                                                                                                                      
 Traceback (most recent call last):                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/mautrix/util/program.py", line 190, in _run                                                                                                                                                      
     self.loop.run_until_complete(self.start())                                                                                                                                                                                            
   File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete                                                                                                                                                       
     return future.result()                                                                                                                                                                                                                
   File "/usr/lib/python3.8/site-packages/mautrix_facebook/__main__.py", line 81, in start                                                                                                                                                 
     await super().start()                                                                                                                                                                                                                 
   File "/usr/lib/python3.8/site-packages/mautrix/bridge/bridge.py", line 149, in start                                                                                                                                                    
     await self.matrix.init_encryption()                                                                                                                                                                                                   
   File "/usr/lib/python3.8/site-packages/mautrix/bridge/matrix.py", line 126, in init_encryption                                                                                                                                          
     if not await self.e2ee.check_server_support():                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/mautrix/bridge/e2ee.py", line 174, in check_server_support                                                                                                                                       
     flows = await self.client.get_login_flows()                                                                                                                                                                                           
   File "/usr/lib/python3.8/site-packages/mautrix/client/api/authentication.py", line 36, in get_login_flows                                                                                                                               
     resp = await self.api.request(Method.GET, Path.login)                                                                                                                                                                                 
   File "/usr/lib/python3.8/site-packages/mautrix/api.py", line 235, in request                                                                                                                                                            
     raise MatrixConnectionError(str(e)) from e                                                                                                                                                                                            
 mautrix.errors.base.MatrixConnectionError: Cannot connect to host matrix:8448 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)') 
 ]                                                                                                                                                                                                                                         

Bridge config:

homeserver:
    address: https://matrix:8448
    verify_ssl: false
    ...

Cert check run from within the Matrix pod:

$ openssl s_client -connect matrix:8448 </dev/null 2>/dev/null | openssl x509 -noout -text | grep DNS:
DNS:matrix.example.com, DNS:matrix

[tulir]

The problem here is that the bridge end-to-bridge encryption helper creates its own HTTP client that doesn't use the appservice config: https://github.com/tulir/mautrix-python/blob/v0.8.8/mautrix/bridge/e2ee.py#L80-L81