TCP sockets for health checks make the broker crash in containers

[ravaga]

I have deployed a containerized Orion-LD instance using the latest version (both in Docker and Kubernetes) with the -socketService and (optionally) the -ssPort 1027 CLI arguments in order to be able to perform health cheks in the broker using TCP sockets instead of calling to the /version endpoint.

When the broker starts without any work (no entities nor requests), the TCP socket works as expected and even the Kubernetes health checks pass with the configuration of the official Helm chart:

readinessProbe:
  tcpSocket:
    port: 1027
  initialDelaySeconds: 30
  periodSeconds: 10
  successThreshold: 1
  failureThreshold: 3
  timeoutSeconds: 30
livenessProbe:
  tcpSocket:
    port: 1027
  initialDelaySeconds: 30
  periodSeconds: 10
  successThreshold: 1
  failureThreshold: 3
  timeoutSeconds: 30

However, when the broker starts with some work (multiple entities being updated every second), it crashes unexpectedly without any error message in its logs.

Furthermore, the same happens when an external program performs this health checks (it opens a TCP connection to check if the broker is healthy and then closes it). When the broker is idle, everything works as expected (even if a message is sent the broker displays a A message was sent over the socket service - the broker is not ready for that - closing connection message in its logs), but if it has some amount of work, it crashes after a second or third health check (when a second/third TCP connection is opened).

Sometimes, when inspecting the logs/events of the pod in K8s, a command terminated with exit code 137 message is shown, which seems to be an OOM error. Nevertheless, the pod doesn't have any memory limit and I have tested it in different clusters with enough free memory in the nodes.