AWS Prefixed Tag Key Error in Pulumi (aws-native/servicecatalogappregistry)

[druskus20]

What happened?

I believe I have encountered a bug while working with Pulumi's AWS-native provider, specifically in the @pulumi/aws-native/servicecatalogappregistry module.

The bug

The bug manifests in the following scenario:

1. On the first run of pulumi up, everything works as expected. The application is created with the correct tags:

Updating (dev):
     Type                                                    Name                            Status            Info
 +   pulumi:pulumi:Stack                                     pulumi-aws-dev                  created (27s)     3 messages
 +   ├─ elk                                                  TaggedApplication::pulumi-test  created (1s)
 +      └─ aws-native:servicecatalogappregistry:Application  pulumi-test                     created (1s)

If I subsequently modify a tag and apply the changes with pulumi up, I encounter the following error:
**updating failed** [diff: ~tags]; 1 error
aws: prefixed tag key names are not allowed for external use.

Do you want to perform this update? yes
Updating (dev):
     Type                                                    Name                            Status                  Info
     pulumi:pulumi:Stack                                     pulumi-aws-dev                  **failed**              1 error; 5 messages
     └─ elk                                                  TaggedApplication::pulumi-test
 ~      └─ aws-native:servicecatalogappregistry:Application  pulumi-test                     **updating failed**     [diff: ~tags]; 1 error


Diagnostics:
  aws-native:servicecatalogappregistry:Application (pulumi-test):
    error: operation error CloudControl: UpdateResource, https response error StatusCode: 400, RequestID: f4eee19e-44c7-4231-a99c-994d9ff11081, api error ValidationException: aws: prefixed tag key names are not allowed for external use.

  pulumi:pulumi:Stack (pulumi-aws-dev):
    error: update failed

// console.log(merged_args) 
    {
      name: 'pulumi-test',
      description: 'pulumi-test',
      tags: { name_extra: 'pulumi-test', 'elk:created-by': 'pulumi' }
    }

Example

The following code is used to create the application:

let app = new application.TaggedApplication("pulumi-test", {"name": "pulumi-test", "description": "pulumi-test", "tags" : {"name_extra": "pulumi-test"}});

import * as pulumi from "@pulumi/pulumi";
import * as app from "@pulumi/aws-native/servicecatalogappregistry";
import { mergeTagsIntoArgs } from "../utils";
import { ComponentResource } from "../component";

export class TaggedApplication extends ComponentResource {
    public readonly application: pulumi.Output<app.Application>;

    constructor(name: string, args: app.ApplicationArgs, opts?: pulumi.ComponentResourceOptions) {
        super(TaggedApplication, name, {}, opts);
        const tags = {
          "elk:created-by": "pulumi" 
        }
        
       // -------- BUG(?) -----------

        const merged_args = mergeTagsIntoArgs(args, tags);
        console.log(merged_args) // <-- Tags

        const application = new app.Application("pulumi-test", merged_args, { parent: this });
        this.registerOutputs({
            application: application
        });

       // -------------------
    }
}

Output of pulumi about

CLI
Version      3.131.0
Go Version   go1.23.0
Go Compiler  gc

Plugins
KIND      NAME        VERSION
resource  aws         6.50.1
resource  aws-native  0.121.0
resource  awsx        2.14.0
resource  docker      4.5.5
resource  docker      3.6.1
language  nodejs      unknown

Host
OS       ubuntu
Version  22.04
Arch     x86_64

This project is written in nodejs: executable='/home/drusk/.config/nvm/versions/node/v20.12.0/bin/node' version='v20.12.0'

Backend
Name           tincan
URL            file://~
User           drusk
Organizations
Token type     personal

Pulumi locates its logs in /tmp by default

Additional context

Possible explanation

The AWS Application automatically gets a aws:servicecatalog:applicationName: pulumi-test tag on creation

It seems that when applying a tag update, Pulumi includes an AWS-managed tag key, such as aws:servicecatalog:applicationName: pulumi-test, in the request. This results in the error message: aws: prefixed tag key names are not allowed for external use.

I am using a simple wrapper around the Application class to add custom tags to an AWS Application.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[flostadler]

Key @druskus20, I'm sorry you're running into this! I was able to reproduce it and started investigating how to fix this. We'll need to filter out the AWS system tags when doing the updates.

[druskus20]

Wow I didn't expect this quick reply.
Yeah indeed my idea of a fix would be to filter out the aws tags

[flostadler]

@druskus20 While digging deeper into this, I noticed that the AWS reserved tags are not included in the request on the Pulumi side. It's actually AWS CloudControl that's wrongly including those in the request to the downstream service.

I opened an issue on their end to track this: aws-cloudformation/cloudformation-coverage-roadmap#2128

In the meantime, you could perform updates to the tags of this resource by replacing it on tag changes. You can do this by using the replaceOnChanges resource option like so:

const application = new aws.servicecatalogappregistry.Application("repro", {
    tags: {
        name_extra: 'pulumi-test',
        additional: 'new-val'
    }
}, {replaceOnChanges: ['tags']});

[druskus20]

Oh wow.
Thank you so much, that is exactly what I am currently doing 👍