Implement HA settings for Eirini

[jandubois]

sizing.bits.instances and sizing.eirini.instances are defined in values.yaml, but not used for anything. The subcharts don't have any mechanism to set a replica count.

Acceptance Criteria

• bits should have settings in values.yaml to control instance count
• eirini should have settings in values.yaml to control instance count
• eirinix should have settings in values.yaml to control instance count
• remove any unused sizing configs from the kubecf values.yaml
• test to see if each of the components can be HA
• document configuration for HA eirini

Relates

#1277

[mudler]

With regards to EiriniX - eirini-loggregator-bridge needs to be adapted to be HA. Currently it's not supported. It might be as easy as setting nodeselector for each replica, but needs to be handled in the bridge first.

Edit: opened cloudfoundry-incubator/eirini-loggregator-bridge#23 with a raw idea on how to implement it - any feedback is welcome

[jandubois]

After discussion with @troytop in standup we decided to remove "suse-cap" label.

I suspect that most components will not be ready for HA yet (will Eirinix create separate webhooks for each instance of an extension? Or will the webhook itself always be single-instance?).

Either way, there is not sufficient time left for the current "suse-cap" timeline to actually test this properly, even if it looks like it might be working.