`python-libjuju` connection to Kubernetes fails with `ProxyNotConnectedError`

[dariofaccin]

Description

Long running integration tests are failing with this error:

WARNING  juju.client.connection:connection.py:579 RPC: Connection closed, reconnecting
WARNING  juju.client.connection:connection.py:579 RPC: Connection closed, reconnecting
ERROR    asyncio:base_events.py:1821 Task exception was never retrieved
future: <Task finished name='Task-3287' coro=<Connection.reconnect() done, defined at /home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py:697> exception=ProxyNotConnectedError()>
Traceback (most recent call last):
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py", line 709, in reconnect
    res = await connector(
          ^^^^^^^^^^^^^^^^
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py", line 800, in _connect_with_login
    await self._connect(endpoints)
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py", line 743, in _connect
    result = await task
             ^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/tasks.py", line 631, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py", line 729, in _try_endpoint
    return await self._open(endpoint, cacert)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py", line 328, in _open
    sock = self.proxy.socket()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/proxy/kubernetes/proxy.py", line 80, in socket
    raise ProxyNotConnectedError()
juju.client.proxy.proxy.ProxyNotConnectedError
ERROR    juju.client.connection:connection.py:587 RPC: Automatic reconnect failed
ERROR    asyncio:base_events.py:1821 Task exception was never retrieved
future: <Task finished name='Task-3288' coro=<Connection.reconnect() done, defined at /home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py:697> exception=ProxyNotConnectedError()>
Traceback (most recent call last):
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py", line 709, in reconnect
    res = await connector(
          ^^^^^^^^^^^^^^^^
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py", line 800, in _connect_with_login
    await self._connect(endpoints)
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py", line 743, in _connect
    result = await task
             ^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/tasks.py", line 631, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py", line 729, in _try_endpoint
    return await self._open(endpoint, cacert)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/connection.py", line 328, in _open
    sock = self.proxy.socket()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/training-operator/training-operator/.tox/integration/lib/python3.12/site-packages/juju/client/proxy/kubernetes/proxy.py", line 80, in socket
    raise ProxyNotConnectedError()
juju.client.proxy.proxy.ProxyNotConnectedError
ERROR    juju.client.connection:connection.py:587 RPC: Automatic reconnect failed
Error: The operation was canceled.

This originates from calling unit.run(). The issue is intermittent and happens more frequently in slow environments.

Urgency

Blocker for our release

Python-libjuju version

3.6.1.2

Juju version

3.6.11

Reproduce / Test

N/A since it requires the whole test suite to run to reproduce.