When running interpreter.chat(...) with python multiprocessing, can occasionally get zmq.error.ZMQError: Address already in use error

[timxzz]

Describe the bug

As the title stated, I occasionally get the zmq.error.ZMQError: Address already in use error when running interpreter.chat(...) with multiple process using python multiprocessing.Pool. I was running as many process as cpu cores.

The trace stack looks like this:

Traceback (most recent call last):
  File ".../python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File ".../python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File ".../python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
    app.launch_new_instance()
  File ".../python3.10/site-packages/traitlets/config/application.py", line 1074, in launch_instance
    app.initialize(argv)
  File ".../python3.10/site-packages/traitlets/config/application.py", line 118, in inner
    return method(app, *args, **kwargs)
  File ".../python3.10/site-packages/ipykernel/kernelapp.py", line 692, in initialize
    self.init_sockets()
  File ".../python3.10/site-packages/ipykernel/kernelapp.py", line 336, in init_sockets
    self.stdin_port = self._bind_socket(self.stdin_socket, self.stdin_port)
  File ".../python3.10/site-packages/ipykernel/kernelapp.py", line 253, in _bind_socket
    return self._try_bind_socket(s, port)
  File ".../python3.10/site-packages/ipykernel/kernelapp.py", line 229, in _try_bind_socket
    s.bind("tcp://%s:%i" % (self.ip, port))
  File ".../python3.10/site-packages/zmq/sugar/socket.py", line 311, in bind
    super().bind(addr)
  File "_zmq.py", line 917, in zmq.backend.cython._zmq.Socket.bind
  File "_zmq.py", line 179, in zmq.backend.cython._zmq._check_rc
zmq.error.ZMQError: Address already in use (addr='tcp://127.0.0.1:42857')

Reproduce

Something like:

from interpreter import interpreter
from multiprocessing import Pool

def worker(prompt):
        return interpreter.chat(prompt, stream=False, display=False)

args = ["...",...,"..."]
with Pool(processes=n_cpu) as pool:
        results = pool.map(worker, args)

Expected behavior

In a separate repo, similar issue has been reported see this. In which, some claims that this can be fixed by allowing the use of self defined jupyter KernelManager that select ipc transport instead of tcp.

Screenshots

No response

Open Interpreter version

0.3.6

Python version

3.10.14

Operating System name and version

Ubuntu 22.04.3 LTS

Additional context

No response