Support multiple provider instances per namespace

[kleinmt]

Using Azure App Configuration Kubernetes Provider v1.1.0, only one AzureAppConfigurationProvider resource can be created per namespace, due to conflicting child resource names, such as:

• Service Account: az-appconfig-k8s-provider
• Config Map: az-appconfig-controller-manager-config

Use Case

When deploying multiple applications in one namespace, each of them may assume their own user assigned identity via Azure Workload Identity. Retrieving configuration values from the App Configuration store should use the application's identity.
Since the providers service account name is not configurable, only one federated credential can be used to establish an OIDC-backed relationship, and as a result only one user assigned identity can be used per namespace.
Leader election, which uses a shared config map for each provider instance, would probably suffer from a similar naming conflict.

Would it be possible to provision provider child resources with unique names, so that multiple instances of the provider can co-exist in one namespace?

Tobias Kleinmann <[email protected]>, Mercedes-Benz Tech Innovation GmbH (Provider Information)

[linglingye001]

Hi @kleinmt , you can create multiple provider instances within each namespace. The property values in the spec section can be set differently based on the application. If each application requires access to the App Configuration store using its own user-assigned identity, you can refer to the documentation use workload identity for guidance on creating federated credentials for each identity. Subsequently, in each instance, you can replace the managedIdentityClientId with the user-assigned identity specific to the application.

apiVersion: azconfig.io/v1
kind: AzureAppConfigurationProvider
metadata:
  name: appconfigurationprovider-sample
spec:
  endpoint: <your-app-configuration-store-endpoint>
  target:
    configMapName: <target-configmap-name>
  auth:
    workloadIdentity:
      managedIdentityClientId: <your-managed-identity-client-id>

Should you have any further questions or need additional assistance, please feel free to reach out.

[kleinmt]

This would still require establishing a federated credential on each user-assigned identity with the single 'az-appconfig-k8s-provider' service account, which is technically possible but I'd rather have a 1:1 relationship between service accounts and user-assigned identities to ensure a fair separation of concerns.

This could be enabled by allowing the user to set the a serviceAccountName property, for example:

[...]
spec:
  endpoint: <your-app-configuration-store-endpoint>
  target:
    configMapName: <target-configmap-name>
  auth:
    workloadIdentity:
      managedIdentityClientId: <your-managed-identity-client-id>
      serviceAccountName: <name-of-service-account>

Would this be something that you could consider for future releases of the provider?

[RichardChen820]

@kleinmt The provider is using its own service account to support workload identity because it can only access its own service account's mounted token file. Support arbitrary service accounts means the provider should be able to
1）access all the service account's mounted token files
or
2）need to talk with the API server to dynamically issue tokens.

Both these two ways do not make much sense to me. The provider is generating configMap(s) from the date in App Configuration, no matter how many pods are running in your namespace, you just need one identity to connect one App Configuration endpoint.

[kleinmt]

@RichardChen820 Agreed, one provider consuming tokens from multiple service accounts doesn't make sense.
What I had in mind when proposing the serviceAccountName property, was an architecture where each workload would have it's own provider instance, but I don't think that's an implementation supported by the current provider version.

Generally speaking, I'd just like to see an architecture option where two or more workloads within the same namespace get their own provider and dedicated service account, so that we have a separation of concerns.

[RichardChen820]

@kleinmt the provider itself is a deployment, you can have more than 1 replica count for redundancy, load balancing, etc. But seems it's not a quite solid reason that having multiple instances just for having more service accounts to have a separation of concerns.

But your requirement is reasonable, we will look into it to see if there's any other resolution. The provider is implemented with "CRD+ controller", it's a quite common pattern, we can check if other controllers are facing the same issue of workload identity.