Tracking Issue AWS integration

[davidspek]

Creating a quick list of things that still need to be done so we can track progress.

What still needs to be done:

• Setup Cognito with OAuth2-Proxy
• Documentation for Cognito integration and IAM setup
• Use RDS for pipelines and katib database
• Use S3 for pipelines instead of MinIO
• AWS plugin for profile controller?

[davidspek]

@karlschriek @matty1979 I'm expecting you guys will run into a lot more things while working on this deployment so it would be good to document that here.

Ping @samdobson In case you are interesting in helping out.

Also ping @PatrickXYS in response to kubeflow/manifests#1832 (comment) if anybody from AWS would like to help out with this effort.

[PatrickXYS]

Use RDS for pipelines database

This is part of KFP support, if we want to enable this, this should be added in kubeflow/pipelines instead.

Use S3 for pipelines instead of MinIO

Same for KFP

For other parts of Cognito stuff, we will explore those in AWS distribution, so I don't think we will be able to contribute to argoflow.

[davidspek]

@PatrickXYS The AWS integrations for things like Cognito in ArgoFlow would all be kustomize manifests, so they would likely be the same as for the AWS distribution.

[PatrickXYS]

@davidspek I agree, so you can simply check out AWS side best practice after we're done with those features.

[davidspek]

Somewhat related PR/issue about enabling AWS ALB authentication in KFP SDK Client: kubeflow/pipelines#4182

[soleares]

@davidspek Are all of the AWS integrations relatively complete besides Cognito? We currently have the standard KF 1.2 distribution deployed on AWS and I’m wondering if this distribution is at a good place for us to try it.

AWS plugin for profile controller

Is this required if we’re not using Cognito?

[davidspek]

@soleares Most of the AWS integrations are complete. Cognito we just haven’t gotten around to yet, but from what I’ve read it shouldn’t be difficult to setup with Oauth2-Proxy.

The plug-in for the profile controller is just to delegate IAM roles to users and their workloads, it shouldn’t be necessary for anything else.

[soleares]

@davidspek What Kubeflow components use this plug-in? Is it required for basic pipeline functionality?

[davidspek]

@soleares The profile controller uses the plugin for service accounts. It is not necessary for functionality of the components.