Image resource errors with "Can't close tar writer" ... "failed to read downloaded context ... status 403"

[AaronFriel]

What happened?

After enabling the "Containerd image store" (see docker/roadmap#371), I see errors during a Pulumi deployment like the following:

  pulumi:pulumi:Stack (pulumi-ai-...):
    ...
    time="2024-01-29T11:01:45-08:00" level=error msg="Can't add file [redacted] to tar: io: read/write on closed pipe"
    time="2024-01-29T11:01:45-08:00" level=error msg="Can't close tar writer: io: read/write on closed pipe"
    time="2024-01-29T11:01:45-08:00" level=error msg="Can't add file [redacted] to tar: io: read/write on closed pipe"
    time="2024-01-29T11:01:45-08:00" level=error msg="Can't close tar writer: io: read/write on closed pipe"

  docker:index:Image (lambda-image):
    error: error reading build output: failed to read downloaded context: failed to load cache key: invalid response status 403

Example

The examples should be sufficient to trigger this bug.

Output of pulumi about

Dependencies:
NAME             VERSION
@pulumi/aws      5.42.0
@pulumi/random   4.13.2
ts-node          10.9.1
@types/node      16.18.38
@pulumi/pulumi   3.74.0
@pulumi/docker   4.5.0
@pulumi/command  0.8.2

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[bernadinm]

Confirmed. I ran into this issue as well.

[didlawowo]

got the same

[shmanny]

I believe I'm seeing the same. If I build the image locally without Pulumi, everything runs fine but if I build the image with Pulumi Docker then I get the same errors as above. Is there a way to downgrade something as a work around?

[AaronFriel]

In Docker Desktop, can you try disabling the "containerd image store"?

[shmanny]

Apologies, it looks like containerd image store was already disabled in my environment so my issue must be unrelated.

[KrzysztofZawisla]

I have the same issue. It happens on MacOS and Windows. Switching the containerd image store option doesn't help either

[blampe]

@KrzysztofZawisla @shmanny @AaronFriel @didlawowo @bernadinm and anyone else affected, can you please provide the output of pulumi about as well as docker version? That will help narrow things down -- thanks!

[yukimotochern]

$ pulumi about [21:59:19]
CLI
Version 3.115.2
Go Version go1.22.2
Go Compiler gc

Host
OS darwin
Version 14.4.1
Arch arm64

$ docker version [22:00:52]
Client:
Cloud integration: v1.0.35+desktop.13
Version: 26.1.1
API version: 1.45
Go version: go1.21.9
Git commit: 4cf5afa
Built: Tue Apr 30 11:44:56 2024
OS/Arch: darwin/arm64
Context: desktop-linux

Server: Docker Desktop 4.30.0 (149282)
Engine:
Version: 26.1.1
API version: 1.45 (minimum version 1.24)
Go version: go1.21.9
Git commit: ac2de55
Built: Tue Apr 30 11:48:04 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.31
GitCommit: e377cd56a71523140ca6ae87e30244719194a521
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0

[blampe]

This repros for me using the docker-container-registry/ts example and the containerd backend. There's some log parsing logic which I suspect isn't handling containerd messages correctly.

You may want to consider the new docker-build provider as a workaround.

[blampe]

Unfortunately the legacy build APIs we use in this provider aren't compatible with containerd due to moby/moby#47717. Until that's resolved the recommendation is to disable the containerd snapshotter ( "features": { "containerd-snapshotter": false } in your engine config) or use the newer docker-build provider as previously mentioned.

For anyone who's disabled containerd and is still experiencing problems, please include the full error message you're seeing as well as the output of docker system info. As far as I can tell this "failed to load cache key" error is unique to containerd, so I suspect something else is going on.

[bogdannazarenko]

Disabling "Use containerd for pulling and storing images" fixed for me

[pavelbinar]

@blampe Switched to docker-build - all good, thanks 👍🏻