Runner version 2.320.0 no longer has SSH installed

[mad-it]

Describe the bug
As of version 2.320.0 it seems that SSH is no longer installed causing issues downloading submodules or using git's ssh access.

To Reproduce
Steps to reproduce the behavior:
Try downloading a git repository using ssh with the latest version of the runner.

Expected behavior
Downloading git repositories with ssh should be working

Runner Version and Platform

2.320.0

Linux

What's not working?

Downloading repositories using SSH configuration is no longer working as of 2.320.0.
This was working fine with version 2.319.1 (previous version).

Job Log Output

prefix=[/home/runner/_work/<redacted>] 
time=2024-10-05T04:08:39Z level=error msg=1 error occurred:
	* error downloading 'ssh://[email protected]/<redacted>.git?ref=1.5.70': /usr/bin/git exited with 128: Cloning into '/home/runner/_work/<redacted>/.terragrunt-cache/BuO11ILvWkWbNBYawXOCXG91[62](https://github.com/<redacted>/actions/runs/11190248795/job/31111770261#step:14:63)M/YDALZ8lPdyU20Ud6W9B3_a75Gtc'...
error: cannot run ssh: No such file or directory
fatal: unable to fork

time=2024-10-05T04:08:39Z level=error msg=Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[edjshelton]

So let me get this straight...

GitHub requires you to update to the latest stable version of the runner every month, otherwise your runners will just stop working.

Then when you update to the latest version of the runner image, SSH mysteriously disappears and breaks most CI workflows.

Fantastic. 👏

[mehraman1508]

I had a python build working like a breeze on the 319 version and after couple of days the same build is running for 50 mins plus and fails on me. Sigh!

[TingluoHuang]

I can't find we have ssh installed on the old runner runner image either.

@TingluoHuang ➜ /workspaces/github $ docker run -it --rm ghcr.io/actions/actions-runner:2.319.1 bash
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

runner@14faec2ba5a6:~$ ssh
bash: ssh: command not found
runner@14faec2ba5a6:~$ 

What runner image did you guys used?

[mad-it]

I can't find we have ssh installed on the old runner runner image either.

@TingluoHuang ➜ /workspaces/github $ docker run -it --rm ghcr.io/actions/actions-runner:2.319.1 bash
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

runner@14faec2ba5a6:~$ ssh
bash: ssh: command not found
runner@14faec2ba5a6:~$ 

What runner image did you guys used?

All I can say is that running 2.319.1 cloning repositories using SSH as authentication method was working perfectly, the only thing that changed over the weekend was version 2.320.0 was released and things magically started breaking.
Try cloning this repository using SSH method with both versions and see if it works (I did not test it myself to be completely honest).

So let me get this straight...

GitHub requires you to update to the latest stable version of the runner every month, otherwise your runners will just stop working.

Then when you update to the latest version of the runner image, SSH mysteriously disappears and breaks most CI workflows.

Fantastic. 👏

The thing that gets me the most is that you cannot even do anything if you are using hosted runners... 🙈

[ChristopherHX]

If I would guess, ssh could be a recommend dependency of git.
Now this line got added

runner/images/Dockerfile

Line 50 in 9b3b554

&& apt install -y --no-install-recommends git

So ssh is like before not in the image, but every image that extends this and did apt install git manually did install ssh implicitly before, now with git already installed ssh is never requested for installation via git.

hosted runners... 🙈

Those have a really different much bigger image

[mrohlof-protofy]

We also noticed that we had to install sqlite3 and terraform on the latest ubuntu images which we did not have to do previously. What is happening here?