From f6226bbfdd4b94a53f208b36ec0b9d39965b4c1b Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Fri, 19 Apr 2024 11:56:15 +0200 Subject: [PATCH 01/21] Add draft for unified networking standard Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-networking-standard.md | 88 ++++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 Standards/scs-xxxx-v1-networking-standard.md diff --git a/Standards/scs-xxxx-v1-networking-standard.md b/Standards/scs-xxxx-v1-networking-standard.md new file mode 100644 index 000000000..bdd063f35 --- /dev/null +++ b/Standards/scs-xxxx-v1-networking-standard.md @@ -0,0 +1,88 @@ +--- +title: Networking Standard +type: Standard +status: Proposal +track: IaaS +--- + +## Introduction + +[...] + +## Motivation + +[...] + +## Design Considerations + +Networking setups can vary between OpenStack clouds in a number of aspects. The focus of this standard are those aspects that have an impact on the cloud user, which can be broadly grouped into two areas: +* API behavior, which includes policy rules and the availability of API extensions, +* and CSP-managed resources, which includes provider networks and their subnets, as well as floating IPs and subnet pools. + +### Options considered + +#### API Extensions + +The OpenStack Networking API is more modular than many other OpenStack APIs, adding new functionality as optional API extensions, rather than a linear sequence of micro-versions. Which of those extensions are available from Neutron depends on the enabled plugins and back-end drivers. + +Mandating a minimum set of enabled extensions will ensure that users have a consistent feature-set available across SCS clouds, but should not unnecessarily restrict the choice of back-end drivers for CSPs. + +[...] + +#### Policies + +[...] + +#### Provider Networks + +Provider networks are CSP-managed networks that are made available to projects, such that project-internal resources can be connected to them. In Openstack, provider networks can be made available as _external_, as _shared_, or both: +* External networks provide the standard way of connecting project-internal resources to networks outside of the cloud environment, such as the internet. They can be connected to project-internal networks with a virtual router, but do not allow VMs to connect directly. +* Shared networks are typically used to connect resources of different projects within the same cloud environment. Unlike external networks, VMs can be directly attached to them. +* Shared external networks are a special case that allows external access, but also direct attachment of VMs. + +Networks can be shared or external to either a specified subset of projects or to all projects (the latter is the default for external networks). For the purpose of standardization we primarily care about provider networks that are available to all projects. +Purely shared networks between all projects have very few use-cases in a public cloud. They are more of special case solution to connect a specific set of related projects, so they will not be further considered here. +External networks, on the other hand, are required to make VMs in a project externally accessible, which is an essential functionality, and as such should be mandatory in an SCS compliant cloud. + +Connecting a VM to a non-shared external network does at minimum require the creation of a virtual router and a project-internal network and subnet. +Assigning public IPs to VMs in an internal network requires an internal subnet with public IP allocation ranges, that will usually be allocated from a CSP-managed subnet pool. +To allow users the dynamic allocation and deallocation of publicly routable subnets, CSPs need to provide [dynamic routing via BGP](https://docs.openstack.org/neutron/latest/admin/config-bgp-dynamic-routing.html). + +Alternatively, at least for IPv4, virtual routers can also act as source NAT, allowing VMs in the internal network to access the internet through the routers external gateway IP. +To make VMs in a NATed subnet externally accessible, OpenStack also supports destination NAT in the form of Floating IPs. +A floating IP represents an address from the external network which can be allocated from a CSP-managed pool and mapped onto an IP within an internal network. + +NAT and floating IPs do not require dynamic routing and can greatly reduce the number of required IPv4 addresses, which are an increasingly sparse resource and may be subject to tight project-quotas by the CSP. +Being ultimately a workaround for the limited IPv4 address space, neither are available for IPv6. +It is possible, however, to create a dual-stack setup in which VMs attached to a routed internal network can have both public IPv6 addresses and private, NATed IPv4 addresses. + +[...] + +A shared external network may provide some usability benefits over a purely external one, as users can make their VMs directly accessible from the internet without creating additional resources. +Beyond that, skipping virtual routers may be important for projects implementing some form of network function virtualisation (NFV) within a VM, but this is more of a niche application, that may also require disabling port security. + +[...] + +## Decision +### API Extensions + +[...] + +### Policies + +[...] + +### Provider Networks + +CSPs **MUST** provide a standard external network that gives projects access to the internet. + +The external network **MUST** have an IPv6 subnet. +CSPs **MUST** provide a subnet pool for the allocation of at least one public /64 IPv6 prefix per project. + +The external network **SHOULD** have an IPv4 subnet. +If CSPs provide an IPv4 subnet, then CSPs **MUST** at least one public Floating IP per project. +They **SHOULD** also provide a subnet pool for the allocation of IPv4 prefixes to project networks. + +CSPs **MUST** provide dynamic routing for all project-allocated IP-prefixes. + +All subnets of external networks **MUST** be configured with `--no-dhcp`. From c6cd13a056f15ee3f97c16777bcb658446609908 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Thu, 25 Apr 2024 17:17:21 +0200 Subject: [PATCH 02/21] Start refocusing networking standard draft towards provider networks Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-networking-standard.md | 86 ++++++++++---------- 1 file changed, 42 insertions(+), 44 deletions(-) diff --git a/Standards/scs-xxxx-v1-networking-standard.md b/Standards/scs-xxxx-v1-networking-standard.md index bdd063f35..c440ac1ee 100644 --- a/Standards/scs-xxxx-v1-networking-standard.md +++ b/Standards/scs-xxxx-v1-networking-standard.md @@ -1,5 +1,5 @@ --- -title: Networking Standard +title: Provider Networks type: Standard status: Proposal track: IaaS @@ -7,72 +7,70 @@ track: IaaS ## Introduction -[...] +Connecting a virtual machine to the internet requires a virtual networks that is also connected to the non-virtualised network infrastructure outside of the cloud environment. +Because such networks break the isolation of the cloud environment, they must be created and managed by the CSP and made available to projects only in a controlled manner. -## Motivation +In OpenStack, such CSP-managed networks are called _provider networks_, though not all provider networks have to provide access to the Internet. +And those who do may impose different restrictions and use different methods of allocating IP addresses. + +External access to and from virtualised resources is a fundamental property of IaaS, and providing this access is a hard requirement for any cloud provider. +This document outlines a standardized setup of provider networks to ensure a consistent, secure, and efficient methodology for managing public IP addresses and providing external access in SCS clouds. -[...] +## Motivation ## Design Considerations -Networking setups can vary between OpenStack clouds in a number of aspects. The focus of this standard are those aspects that have an impact on the cloud user, which can be broadly grouped into two areas: -* API behavior, which includes policy rules and the availability of API extensions, -* and CSP-managed resources, which includes provider networks and their subnets, as well as floating IPs and subnet pools. +In OpenStack, ownership of resources is generally tracked through projects, and, as per default policy, only members of a project have access to its resources +This is also true for CSP-managed resources, such as provider networks, which have to be created in a designated internal projec, and are initially only accessible in this project. -### Options considered +The Network API's Role Based Access Control (RBAC) extension can then be used to share it with other projects. +RBAC rules for networks support the two actions `access_as_external` and `access_as_shared`, and can be created automatically on `openstack network create` with the options `--external` and `--share`. +* `access_as_external` allows networks to be used as external gateway for virtual routers in the target projects. Such networks are in the following referred to as _external networks_. +* `access_as_shared` allows networks to be attached directly to VMs in the target projects. Such networks are in the following referred to as _shared networks_. -#### API Extensions +Rules can be created with either a specific target project ID or with a wild card (`*`) to target all projects. +They can also overlap, allowing a network to be both external and shared to the same target projects. -The OpenStack Networking API is more modular than many other OpenStack APIs, adding new functionality as optional API extensions, rather than a linear sequence of micro-versions. Which of those extensions are available from Neutron depends on the enabled plugins and back-end drivers. +CSPs can assign a subnet to a provider network to supply connected VMs or routers with public IP addresses, making them externally accessible. +This works well for a shared provider network, but connecting VMs behind a virtual router to the internet is a bit more complicated. -Mandating a minimum set of enabled extensions will ensure that users have a consistent feature-set available across SCS clouds, but should not unnecessarily restrict the choice of back-end drivers for CSPs. +* subnet pools +* NAS -[...] +### Options considered -#### Policies +#### Single Default Provider Network -[...] +* multiple provider networks may confusing, user has to figure out the "right" one +* dual stack possible with multiple subnets in a single networks -#### Provider Networks +#### Shared Provider Network -Provider networks are CSP-managed networks that are made available to projects, such that project-internal resources can be connected to them. In Openstack, provider networks can be made available as _external_, as _shared_, or both: -* External networks provide the standard way of connecting project-internal resources to networks outside of the cloud environment, such as the internet. They can be connected to project-internal networks with a virtual router, but do not allow VMs to connect directly. -* Shared networks are typically used to connect resources of different projects within the same cloud environment. Unlike external networks, VMs can be directly attached to them. -* Shared external networks are a special case that allows external access, but also direct attachment of VMs. +* simplicity of use, no extra resources +* port security is essential +* no quota for address use -Networks can be shared or external to either a specified subset of projects or to all projects (the latter is the default for external networks). For the purpose of standardization we primarily care about provider networks that are available to all projects. -Purely shared networks between all projects have very few use-cases in a public cloud. They are more of special case solution to connect a specific set of related projects, so they will not be further considered here. -External networks, on the other hand, are required to make VMs in a project externally accessible, which is an essential functionality, and as such should be mandatory in an SCS compliant cloud. +#### External Provider Network and subnet allocation -Connecting a VM to a non-shared external network does at minimum require the creation of a virtual router and a project-internal network and subnet. -Assigning public IPs to VMs in an internal network requires an internal subnet with public IP allocation ranges, that will usually be allocated from a CSP-managed subnet pool. -To allow users the dynamic allocation and deallocation of publicly routable subnets, CSPs need to provide [dynamic routing via BGP](https://docs.openstack.org/neutron/latest/admin/config-bgp-dynamic-routing.html). +* requires virtual router +* requires dynamic routing +* IPv4+IPv6 +* prefer over shared network because: FWaaS, Quota -Alternatively, at least for IPv4, virtual routers can also act as source NAT, allowing VMs in the internal network to access the internet through the routers external gateway IP. -To make VMs in a NATed subnet externally accessible, OpenStack also supports destination NAT in the form of Floating IPs. -A floating IP represents an address from the external network which can be allocated from a CSP-managed pool and mapped onto an IP within an internal network. +#### NAT and Floating IPs -NAT and floating IPs do not require dynamic routing and can greatly reduce the number of required IPv4 addresses, which are an increasingly sparse resource and may be subject to tight project-quotas by the CSP. -Being ultimately a workaround for the limited IPv4 address space, neither are available for IPv6. -It is possible, however, to create a dual-stack setup in which VMs attached to a routed internal network can have both public IPv6 addresses and private, NATed IPv4 addresses. +* requires virtual router +* IPv4 only +* dual stack use with IPv6 from subnetpool +* IPv4: prefer NAT/FIP over subnet pool to discourage wasting addresses -[...] +#### Dual Stack -A shared external network may provide some usability benefits over a purely external one, as users can make their VMs directly accessible from the internet without creating additional resources. -Beyond that, skipping virtual routers may be important for projects implementing some form of network function virtualisation (NFV) within a VM, but this is more of a niche application, that may also require disabling port security. +#### Security Considerations -[...] +* disallow creation of rbac rules for users to prevent creation of faux provider networks ## Decision -### API Extensions - -[...] - -### Policies - -[...] - -### Provider Networks CSPs **MUST** provide a standard external network that gives projects access to the internet. From cbf9c0f81674bbadb3a97334287c912db62f66bd Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Mon, 29 Apr 2024 17:17:15 +0200 Subject: [PATCH 03/21] Expand design considerations section Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-networking-standard.md | 40 ++++++++++++++++---- 1 file changed, 33 insertions(+), 7 deletions(-) diff --git a/Standards/scs-xxxx-v1-networking-standard.md b/Standards/scs-xxxx-v1-networking-standard.md index c440ac1ee..e0946a6cb 100644 --- a/Standards/scs-xxxx-v1-networking-standard.md +++ b/Standards/scs-xxxx-v1-networking-standard.md @@ -20,6 +20,8 @@ This document outlines a standardized setup of provider networks to ensure a con ## Design Considerations +### Provider Network Access Control + In OpenStack, ownership of resources is generally tracked through projects, and, as per default policy, only members of a project have access to its resources This is also true for CSP-managed resources, such as provider networks, which have to be created in a designated internal projec, and are initially only accessible in this project. @@ -28,14 +30,33 @@ RBAC rules for networks support the two actions `access_as_external` and `access * `access_as_external` allows networks to be used as external gateway for virtual routers in the target projects. Such networks are in the following referred to as _external networks_. * `access_as_shared` allows networks to be attached directly to VMs in the target projects. Such networks are in the following referred to as _shared networks_. -Rules can be created with either a specific target project ID or with a wild card (`*`) to target all projects. +The rules can be created with either a specific target project ID, or with a wild card (`*`) to target all projects. They can also overlap, allowing a network to be both external and shared to the same target projects. -CSPs can assign a subnet to a provider network to supply connected VMs or routers with public IP addresses, making them externally accessible. -This works well for a shared provider network, but connecting VMs behind a virtual router to the internet is a bit more complicated. +### Address Allocation and Routing + +CSPs can assign a subnet to a provider network to supply connected VMs or virtual routers with public IP addresses, making them externally accessible. +This works well for shared networks where VMs can be attached directly, although there is no quota option to limit the number of allocated addresses per project. + +Making VMs in a project-internal network externally accessible through a virtual router is a bit more complicated, though. +One option is for the user to create a subnet with a public IP range for the internal network, and then ask the CSP to configure a static route to the subnet via the gateway IP of a virtual router. +This is cumbersome to set up manually, but can be automated with the `bgp` extension of the Network API, which is implemented by the `neutron-dynamic-routing` project [^bgp]. +For users this takes the form of a CSP-managed, shared subnet pool that they can create externally routable subnets from, limited by a per-project quota. + +For IPv6, there is also the option of prefix delegation, where a DHCPv6 server automatically assigns an IPv6 prefix to a subnet whenever it is connected to the external provider network [^pd]. +This also means that ports in the subnet can lose their addresses and get new ones if the subnet is removed from the external network and later reattached. +With prefix delegation there is no quota on the number of prefixes per project. -* subnet pools -* NAS +For IPv4, OpenStack virtual routers support source NAT, allowing all VMs in the internal subnet to access the external network with the gateway IP of the virtual router. +They also support destination NAT in the form of floating IPs, addresses from the external network that can be mapped onto specific VMs in the internal subnet to make them externally accessible. +Floating IPs are allocated from a CSP-managed pool, and can be controlled by a per-project quota. +There is also a set of API extensions that allow more fine grained port-forwarding, mapping different TCP or UDP ports of a floating IP to different internal IP addresses [^pf]. + +### Port Security and Spoofing + +OpenStack networks have the flag `port_security_enabled`, that is set to true by default and can only be changed by it's owner. +In Neutron, besides enabling security groups for ports in this network, it also enables a built-in DHCP, MAC and IP address spoofing protection. +Whether this flag is set is primarily of concern for shared provider networks, as users only have limited control over the gateway ports of virtual routers. ### Options considered @@ -50,7 +71,7 @@ This works well for a shared provider network, but connecting VMs behind a virtu * port security is essential * no quota for address use -#### External Provider Network and subnet allocation +#### External Provider Network * requires virtual router * requires dynamic routing @@ -69,6 +90,7 @@ This works well for a shared provider network, but connecting VMs behind a virtu #### Security Considerations * disallow creation of rbac rules for users to prevent creation of faux provider networks +* require port security in shared provider networks ## Decision @@ -83,4 +105,8 @@ They **SHOULD** also provide a subnet pool for the allocation of IPv4 prefixes t CSPs **MUST** provide dynamic routing for all project-allocated IP-prefixes. -All subnets of external networks **MUST** be configured with `--no-dhcp`. +## References + +[^bgp]: https://docs.openstack.org/neutron/latest/admin/config-bgp-dynamic-routing.html +[^pd]: https://docs.openstack.org/neutron/latest/admin/config-ipv6.html#prefix-delegation +[^pf]: https://docs.openstack.org/api-ref/network/v2/index.html#floating-ips-port-forwarding From 9d735f63ad45c433a13ce930ef13e9a794f47eb1 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Tue, 30 Apr 2024 18:11:39 +0200 Subject: [PATCH 04/21] Add motivation and expand on port security Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-networking-standard.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/Standards/scs-xxxx-v1-networking-standard.md b/Standards/scs-xxxx-v1-networking-standard.md index e0946a6cb..1fbf1f6bb 100644 --- a/Standards/scs-xxxx-v1-networking-standard.md +++ b/Standards/scs-xxxx-v1-networking-standard.md @@ -18,6 +18,10 @@ This document outlines a standardized setup of provider networks to ensure a con ## Motivation +The ability to interface virtualised resources with networks outside of the cloud environment, such as the internet, is an integral part of IaaS. +Providing external access in an OpenStack cloud requires a number of configuration choices from the CSP, some of which have direct implications on how users interact with the cloud. +This standard identifies some of these options and defines a baseline setup that provides flexibility and consistency to users but is also realistic to implement for CSPs. + ## Design Considerations ### Provider Network Access Control @@ -41,7 +45,7 @@ This works well for shared networks where VMs can be attached directly, although Making VMs in a project-internal network externally accessible through a virtual router is a bit more complicated, though. One option is for the user to create a subnet with a public IP range for the internal network, and then ask the CSP to configure a static route to the subnet via the gateway IP of a virtual router. This is cumbersome to set up manually, but can be automated with the `bgp` extension of the Network API, which is implemented by the `neutron-dynamic-routing` project [^bgp]. -For users this takes the form of a CSP-managed, shared subnet pool that they can create externally routable subnets from, limited by a per-project quota. +For users, this takes the form of a CSP-managed shared subnet pool that they can create externally routable subnets from, limited by a per-project quota. For IPv6, there is also the option of prefix delegation, where a DHCPv6 server automatically assigns an IPv6 prefix to a subnet whenever it is connected to the external provider network [^pd]. This also means that ports in the subnet can lose their addresses and get new ones if the subnet is removed from the external network and later reattached. @@ -55,8 +59,13 @@ There is also a set of API extensions that allow more fine grained port-forwardi ### Port Security and Spoofing OpenStack networks have the flag `port_security_enabled`, that is set to true by default and can only be changed by it's owner. -In Neutron, besides enabling security groups for ports in this network, it also enables a built-in DHCP, MAC and IP address spoofing protection. +In Neutron, besides enabling security groups for ports in this network, it also enables a set of built-in spoofing protections. + Whether this flag is set is primarily of concern for shared provider networks, as users only have limited control over the gateway ports of virtual routers. +A lack of spoofing protection in a shared network, however, does enable a number of attacks that a malicious user or compromised VM could perform against other VMs in the network, such as DHCP-spoofing or ARP-Poisoning. + +There are legitimate use-cases for networks without port security, such as the implementations of network function virtualisation (NFV) within a VM. +However, this seems to be more of a niche use-case and may warrant the creation of a project-specific provider network, rather than making all other projects vulnerable to spoofing attacks. ### Options considered From b52c6c3984dff5f935789777da9361ba35775b52 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Thu, 2 May 2024 17:44:29 +0200 Subject: [PATCH 05/21] Update Design Considerations and Decision Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-networking-standard.md | 45 ++++++++++++++------ 1 file changed, 33 insertions(+), 12 deletions(-) diff --git a/Standards/scs-xxxx-v1-networking-standard.md b/Standards/scs-xxxx-v1-networking-standard.md index 1fbf1f6bb..b48824ccd 100644 --- a/Standards/scs-xxxx-v1-networking-standard.md +++ b/Standards/scs-xxxx-v1-networking-standard.md @@ -24,6 +24,8 @@ This standard identifies some of these options and defines a baseline setup that ## Design Considerations +(discuss required API extensions?) + ### Provider Network Access Control In OpenStack, ownership of resources is generally tracked through projects, and, as per default policy, only members of a project have access to its resources @@ -59,23 +61,38 @@ There is also a set of API extensions that allow more fine grained port-forwardi ### Port Security and Spoofing OpenStack networks have the flag `port_security_enabled`, that is set to true by default and can only be changed by it's owner. -In Neutron, besides enabling security groups for ports in this network, it also enables a set of built-in spoofing protections. +In Neutron, besides enabling security groups for ports in this network, it also enables a number of built-in spoofing protections. Whether this flag is set is primarily of concern for shared provider networks, as users only have limited control over the gateway ports of virtual routers. A lack of spoofing protection in a shared network, however, does enable a number of attacks that a malicious user or compromised VM could perform against other VMs in the network, such as DHCP-spoofing or ARP-Poisoning. -There are legitimate use-cases for networks without port security, such as the implementations of network function virtualisation (NFV) within a VM. +There are legitimate use-cases for networks without port security, such as the implementation of network function virtualisation (NFV) within a VM. However, this seems to be more of a niche use-case and may warrant the creation of a project-specific provider network, rather than making all other projects vulnerable to spoofing attacks. ### Options considered +#### IPv6 + +The OpenStack Network API allows the creation of subnets with either IPv4 or IPv6 address ranges, as indicated by the `ip_version` field. +However, to allow external access to either, the CSP needs to provide projects with externally routable addresses for that IP version. + +While it is possible (and common) for CSPs to provide both IPv4 and IPv6, the increasing scarcity (and cost) of IPv4 address space may at some point become a barrier to entry for new CSPs. +Mandatory support for IPv6 but not IPv4 addresses this problem while providing users with a consistent feature set across SCS clouds. + #### Single Default Provider Network -* multiple provider networks may confusing, user has to figure out the "right" one -* dual stack possible with multiple subnets in a single networks +In principle, CSPs can create multiple provider networks for a number of reasons, for example to make cloud-internal shared services available to projects. +VMs can be connected to multiple networks, and connecting to additional provider networks would not interfere with their ability be externally accessible. + +CSPs may also create multiple provider networks with different options for external access, such as separate networks for IPv4 and IPv6, or one external network for use with virtual routers and a separate shared network for direct connection. +This mostly just adds complexity to the setup, though, as a provider network can be both external and shared at the same time, and can even provide both IPv4 and IPv6 subnets in a dual stack setup [^ds]. + +Another problem with multiple provider networks is that user may only be able to distinguish their respective function by their name. +A single default provider network leaves no ambiguity by the user in this regard and is thus preferable from a standardisation perspective. #### Shared Provider Network +A shared provider network has the benefit of being easy to * simplicity of use, no extra resources * port security is essential * no quota for address use @@ -94,25 +111,29 @@ However, this seems to be more of a niche use-case and may warrant the creation * dual stack use with IPv6 from subnetpool * IPv4: prefer NAT/FIP over subnet pool to discourage wasting addresses -#### Dual Stack - -#### Security Considerations +#### RBAC for Users * disallow creation of rbac rules for users to prevent creation of faux provider networks -* require port security in shared provider networks ## Decision -CSPs **MUST** provide a standard external network that gives projects access to the internet. +CSPs **MUST** provide a standard provider network to every project to access to the internet. +There **SHOULD** be no other provider networks available to projects by default. + +The standard provider network **MUST** be an external network, allowing it to be used as external gateway by virtual routers. +The standard provider network **MAY** be a shared network, allowing direct attachment of VMs. +If the standard provider network is a shared network, it **MUST** enable port security. The external network **MUST** have an IPv6 subnet. CSPs **MUST** provide a subnet pool for the allocation of at least one public /64 IPv6 prefix per project. The external network **SHOULD** have an IPv4 subnet. -If CSPs provide an IPv4 subnet, then CSPs **MUST** at least one public Floating IP per project. -They **SHOULD** also provide a subnet pool for the allocation of IPv4 prefixes to project networks. +If CSPs provide an IPv4 subnet, then CSPs **MUST** provide at least one public Floating IP per project. +They **MAY** also provide a subnet pool for the allocation of public IPv4 prefixes to project networks. + +CSPs **MUST** provide dynamic routing for all project-allocated public IP-prefixes. -CSPs **MUST** provide dynamic routing for all project-allocated IP-prefixes. +Users **SHOULD** by default be prohibited by policy from creating RBAC rules for networks in their projects, to prevent the creation of faux provider networks. ## References From 11d7c13a40a9fade27b2c7b904641b313341741c Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Mon, 6 May 2024 18:42:57 +0200 Subject: [PATCH 06/21] Update and extend Design Considerations section Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-networking-standard.md | 66 +++++++++++++------- 1 file changed, 43 insertions(+), 23 deletions(-) diff --git a/Standards/scs-xxxx-v1-networking-standard.md b/Standards/scs-xxxx-v1-networking-standard.md index b48824ccd..2aedb53f7 100644 --- a/Standards/scs-xxxx-v1-networking-standard.md +++ b/Standards/scs-xxxx-v1-networking-standard.md @@ -29,7 +29,7 @@ This standard identifies some of these options and defines a baseline setup that ### Provider Network Access Control In OpenStack, ownership of resources is generally tracked through projects, and, as per default policy, only members of a project have access to its resources -This is also true for CSP-managed resources, such as provider networks, which have to be created in a designated internal projec, and are initially only accessible in this project. +This is also true for CSP-managed resources, such as provider networks, which have to be created in a CSP-internal project, and are initially only accessible in this project. The Network API's Role Based Access Control (RBAC) extension can then be used to share it with other projects. RBAC rules for networks support the two actions `access_as_external` and `access_as_shared`, and can be created automatically on `openstack network create` with the options `--external` and `--share`. @@ -49,9 +49,9 @@ One option is for the user to create a subnet with a public IP range for the int This is cumbersome to set up manually, but can be automated with the `bgp` extension of the Network API, which is implemented by the `neutron-dynamic-routing` project [^bgp]. For users, this takes the form of a CSP-managed shared subnet pool that they can create externally routable subnets from, limited by a per-project quota. -For IPv6, there is also the option of prefix delegation, where a DHCPv6 server automatically assigns an IPv6 prefix to a subnet whenever it is connected to the external provider network [^pd]. -This also means that ports in the subnet can lose their addresses and get new ones if the subnet is removed from the external network and later reattached. -With prefix delegation there is no quota on the number of prefixes per project. +For IPv6, there is also the option of prefix delegation, where a DHCPv6 server automatically assigns an IPv6 prefix to a subnet when it connects to the external provider network [^pd]. +However, this also means that ports in the subnet can lose their addresses and get assigned new ones if the subnet is removed from the external network and later reattached. +The documentation at [^pd] still marks prefix delegation as an experimental feature in Neutron and notes low test coverage. For IPv4, OpenStack virtual routers support source NAT, allowing all VMs in the internal subnet to access the external network with the gateway IP of the virtual router. They also support destination NAT in the form of floating IPs, addresses from the external network that can be mapped onto specific VMs in the internal subnet to make them externally accessible. @@ -66,8 +66,8 @@ In Neutron, besides enabling security groups for ports in this network, it also Whether this flag is set is primarily of concern for shared provider networks, as users only have limited control over the gateway ports of virtual routers. A lack of spoofing protection in a shared network, however, does enable a number of attacks that a malicious user or compromised VM could perform against other VMs in the network, such as DHCP-spoofing or ARP-Poisoning. -There are legitimate use-cases for networks without port security, such as the implementation of network function virtualisation (NFV) within a VM. -However, this seems to be more of a niche use-case and may warrant the creation of a project-specific provider network, rather than making all other projects vulnerable to spoofing attacks. +There are legitimate use-cases for networks without port security, such as the implementation of network function virtualisation (NFV) within a VM (i.e. using a VM as a virtual router). +This seems to be more of a niche use-case, however, and may warrant the creation of a project-specific provider network, rather than making all other projects vulnerable to spoofing attacks. ### Options considered @@ -77,7 +77,7 @@ The OpenStack Network API allows the creation of subnets with either IPv4 or IPv However, to allow external access to either, the CSP needs to provide projects with externally routable addresses for that IP version. While it is possible (and common) for CSPs to provide both IPv4 and IPv6, the increasing scarcity (and cost) of IPv4 address space may at some point become a barrier to entry for new CSPs. -Mandatory support for IPv6 but not IPv4 addresses this problem while providing users with a consistent feature set across SCS clouds. +Mandatory support for IPv6 but not IPv4 addresses this problem, while also providing users with a consistent feature set across SCS clouds. #### Single Default Provider Network @@ -87,33 +87,53 @@ VMs can be connected to multiple networks, and connecting to additional provider CSPs may also create multiple provider networks with different options for external access, such as separate networks for IPv4 and IPv6, or one external network for use with virtual routers and a separate shared network for direct connection. This mostly just adds complexity to the setup, though, as a provider network can be both external and shared at the same time, and can even provide both IPv4 and IPv6 subnets in a dual stack setup [^ds]. -Another problem with multiple provider networks is that user may only be able to distinguish their respective function by their name. +Another problem with multiple provider networks is, that users may only be able to distinguish their respective function by their name. + A single default provider network leaves no ambiguity by the user in this regard and is thus preferable from a standardisation perspective. #### Shared Provider Network -A shared provider network has the benefit of being easy to -* simplicity of use, no extra resources -* port security is essential -* no quota for address use +A shared provider network has the benefit of being easy to use. +VMs can be attached directly and will be accessible immediately, without the requirement to create virtual routers, project-internal networks, subnets, or floating IPs. -#### External Provider Network +Users do have a higher control over VM ports than they have over virtual router ports, so it is important to enable Neutrons port security feature on shared provider networks to prevent spoofing. -* requires virtual router -* requires dynamic routing -* IPv4+IPv6 -* prefer over shared network because: FWaaS, Quota +The lack of address quota may be a problem if IPv4 is used and the number of available addresses is so limited that fairness between projects needs to be enforced. +In that case, CSPs may restrict the number of VMs, routers, or ports to limit the addresses used by individual projects. + +#### External Provider Network and Subnet Allocation + +Creating a project-internal network to connect to an external provider network with a virtual router does take more effort than just using a shared network, but also offers some additional flexibility and control. +In an internal network, users have greater control over IP allocation and may also choose to disable port security. +With the FWaaS API extensions, they can also assign firewall rules to the virtual router, to control which traffic can pass between project-internal and provider networks. + +* why subnet pool allocation? + * Floating IP limited to IPv4 + * Prefix Delegation limited to IPv6 +* what benefits does a CSP have from this setup? + * Quota for subnet pools +* what are drawbacks + * complexity of setting up dynamic routing #### NAT and Floating IPs * requires virtual router * IPv4 only * dual stack use with IPv6 from subnetpool -* IPv4: prefer NAT/FIP over subnet pool to discourage wasting addresses +* IPv4: prefer NAT/FIP over subnet pool to discourage wasting addresses? + +#### Disable RBAC for Users + +Per default policy, Neutron allows any user the creation RBAC rules to share resources of their projects with other projects. +Only the use of the `*` wildcard target is limited to admin users. + +However, how a network was shared, and who shared it, is not immediately obvious from the perspecive of a target project, the `openstack network list` command will not even show project IDs by default. +Even if a user determines the project ID using the `--long` option or `network show`, they are by default forbidden from listing any other projects metadata, including it's name. -#### RBAC for Users +Under these conditions, a malicious user could create a network with a misleading name, share it with target projects to trick them into using it like a provider network, and then intercept their traffic. -* disallow creation of rbac rules for users to prevent creation of faux provider networks +For this attack to work, the attacker has to find out the target's project ID, and the target has to be sufficiently oblivious to the CSP's provider network setup. +CSPs can try to educate users on the correct provider networks to use, and can avoid leaking project IDs, but the best protection is to disable the creation of RBAC rules for non-admin users. ## Decision @@ -133,10 +153,10 @@ They **MAY** also provide a subnet pool for the allocation of public IPv4 prefix CSPs **MUST** provide dynamic routing for all project-allocated public IP-prefixes. -Users **SHOULD** by default be prohibited by policy from creating RBAC rules for networks in their projects, to prevent the creation of faux provider networks. +By default, users **SHOULD** be prohibited by policy from creating RBAC rules for networks in their projects, to prevent the creation of faux provider networks. ## References -[^bgp]: https://docs.openstack.org/neutron/latest/admin/config-bgp-dynamic-routing.html -[^pd]: https://docs.openstack.org/neutron/latest/admin/config-ipv6.html#prefix-delegation +[^bgp]: https://docs.openstack.org/neutron/2024.1/admin/config-bgp-dynamic-routing.html +[^pd]: https://docs.openstack.org/neutron/2024.1/admin/config-ipv6.html#prefix-delegation [^pf]: https://docs.openstack.org/api-ref/network/v2/index.html#floating-ips-port-forwarding From 67a63b81ce9d48a24a5b48069de7be5f77decf6e Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Wed, 8 May 2024 08:57:02 +0200 Subject: [PATCH 07/21] Extend considered options section Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-networking-standard.md | 29 ++++++++++---------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/Standards/scs-xxxx-v1-networking-standard.md b/Standards/scs-xxxx-v1-networking-standard.md index 2aedb53f7..ce0138091 100644 --- a/Standards/scs-xxxx-v1-networking-standard.md +++ b/Standards/scs-xxxx-v1-networking-standard.md @@ -105,30 +105,31 @@ In that case, CSPs may restrict the number of VMs, routers, or ports to limit th Creating a project-internal network to connect to an external provider network with a virtual router does take more effort than just using a shared network, but also offers some additional flexibility and control. In an internal network, users have greater control over IP allocation and may also choose to disable port security. -With the FWaaS API extensions, they can also assign firewall rules to the virtual router, to control which traffic can pass between project-internal and provider networks. +With the FWaaS API extensions, they can also assign firewall rules to the virtual router, to control which traffic can pass between internal and provider networks. -* why subnet pool allocation? - * Floating IP limited to IPv4 - * Prefix Delegation limited to IPv6 -* what benefits does a CSP have from this setup? - * Quota for subnet pools -* what are drawbacks - * complexity of setting up dynamic routing +As described above, there are multiple methods for allocating public IP addresses to project-internal networks. +For IPv6, the currently best option seems to be subnet allocation from a CSP-managed subnet pool, because support for Prefix Delegation is still experimental. +For IPv4, NAT and floating IPs are generally preferred over subnet allocation because of scarcity of IPv4 address space. #### NAT and Floating IPs -* requires virtual router -* IPv4 only -* dual stack use with IPv6 from subnetpool -* IPv4: prefer NAT/FIP over subnet pool to discourage wasting addresses? +Subnets can only have a size that is a power of two, and will thus usually be oversized for the project. +They also need to reserve one address for the gateway and, if DHCP is used, one or more addresses for DHCP service ports. +Each subnet also has a broadcast and a network address, which for small subnets make up a noticeable part of the address space. + +Source NAT, combined with selective use of floating IPs can significantly reduce the number of required addresses over a public IPv4 subnet. +The floating IP quota also offers a finer granularity for distributing IPs among projects, though it is important to note that the routers external gateway IP which is used for the source NAT is not subject to any quotas. + +IPv4 NAT can also be used in a dual stack setup alongside a routed IPv6 subnet (source?). #### Disable RBAC for Users Per default policy, Neutron allows any user the creation RBAC rules to share resources of their projects with other projects. Only the use of the `*` wildcard target is limited to admin users. -However, how a network was shared, and who shared it, is not immediately obvious from the perspecive of a target project, the `openstack network list` command will not even show project IDs by default. -Even if a user determines the project ID using the `--long` option or `network show`, they are by default forbidden from listing any other projects metadata, including it's name. +However, how a network was shared, and who shared it, is not immediately obvious from the perspective of the target project. +The `openstack network list` command will by default not even show the project IDs of the networks. +And even though users can determine the project ID of networks by using `network list --long` or `network show`, they are by default forbidden from accessing any details of other projects, including the project name. Under these conditions, a malicious user could create a network with a misleading name, share it with target projects to trick them into using it like a provider network, and then intercept their traffic. From 30d4d2a3469ada3105dba64ba5dd34dd3e0d9a12 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Wed, 5 Jun 2024 09:48:43 +0200 Subject: [PATCH 08/21] Add glossary and discussion of API extensions Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-networking-standard.md | 85 ++++++++++++++------ 1 file changed, 59 insertions(+), 26 deletions(-) diff --git a/Standards/scs-xxxx-v1-networking-standard.md b/Standards/scs-xxxx-v1-networking-standard.md index ce0138091..66bf0a4e7 100644 --- a/Standards/scs-xxxx-v1-networking-standard.md +++ b/Standards/scs-xxxx-v1-networking-standard.md @@ -1,5 +1,5 @@ --- -title: Provider Networks +title: SCS Provider Network Standard type: Standard status: Proposal track: IaaS @@ -7,14 +7,34 @@ track: IaaS ## Introduction -Connecting a virtual machine to the internet requires a virtual networks that is also connected to the non-virtualised network infrastructure outside of the cloud environment. +Connecting a virtual machine to the internet requires a virtual network that is also connected to the non-virtualised network infrastructure outside of the cloud environment. Because such networks break the isolation of the cloud environment, they must be created and managed by the CSP and made available to projects only in a controlled manner. -In OpenStack, such CSP-managed networks are called _provider networks_, though not all provider networks have to provide access to the Internet. -And those who do may impose different restrictions and use different methods of allocating IP addresses. +In OpenStack, such CSP-managed networks are called _provider networks_, though providing access to the internet is not their only use case. +Depending on the configuration, they may impose different usage restrictions and offer different methods of allocating IP addresses. -External access to and from virtualised resources is a fundamental property of IaaS, and providing this access is a hard requirement for any cloud provider. -This document outlines a standardized setup of provider networks to ensure a consistent, secure, and efficient methodology for managing public IP addresses and providing external access in SCS clouds. +External access to and from virtualised resources is a fundamental feature of IaaS, and providing this access is a hard requirement for any cloud provider. +However, OpenStack provider networks offer some design space regarding usage restrictions and IP address allocation. +This document defines a standardized setup for provider networks that offers external access in a way that is portable across SCS clouds. + +### Glossary + +The following terms are used throughout this document: + +| Term | Meaning | +|------|---------| +| CSP | Cloud Service Provider, provider managing the OpenStack infrastructure. | +| Server | Server object in the OpenStack Compute API, usually represents a virtual machine, though there are also compute drivers for containers and bare metal servers. | +| Virtual Network | A virtualized network managed by OpenStack, provides connectivity between servers and other network resources, such as virtual routers. | +| Virtual Router | OpenStack resource that can be used to route and bridge between virtual networks and provide other features such as NAT | +| Subnet | Subdivision of available IP address space using address prefixes. In OpenStack also an abstraction for controlling IP address allocation in a virtual network. | +| DHCP | Dynamic Host Configuration Protocol: Used to automatically configure hosts in a network with IP addresses, default routes, and other information such as DNS servers. | +| Prefix | IP address prefix of a given bit-length N, written _/N_. Divides addresses into a network and a host part, a shorter prefix allows more hosts but takes up more address space. | +| NAT | Network Address Translation: mapping (usually public) IPv4 addresses onto a different (usually private) address space. May allows multiple hosts to share a public address by multiplexing TCP/UDP ports. | +| RBAC | Role-based Access Control: A mechanism in the Network API to give projects limited access to resources owned by other projects. Typically used by CSPs to create provider networks. | +| Shared Network | Virtual network that is shared between projects in a way that allows direct attachment of servers. | +| External Network | Virtual network that is shared between projects in a way that only allows virtual routers to use it as external gateway. Typically used by CSPs to provide access to networks outside of the cloud environment. | +| Provider Network | Any shared or external network that is managed by the CSP. | ## Motivation @@ -24,7 +44,7 @@ This standard identifies some of these options and defines a baseline setup that ## Design Considerations -(discuss required API extensions?) +This section will provide some general background on OpenStack provider networks before considering specific options to standardize. ### Provider Network Access Control @@ -34,17 +54,17 @@ This is also true for CSP-managed resources, such as provider networks, which ha The Network API's Role Based Access Control (RBAC) extension can then be used to share it with other projects. RBAC rules for networks support the two actions `access_as_external` and `access_as_shared`, and can be created automatically on `openstack network create` with the options `--external` and `--share`. * `access_as_external` allows networks to be used as external gateway for virtual routers in the target projects. Such networks are in the following referred to as _external networks_. -* `access_as_shared` allows networks to be attached directly to VMs in the target projects. Such networks are in the following referred to as _shared networks_. +* `access_as_shared` allows networks to be attached directly to servers in the target projects. Such networks are in the following referred to as _shared networks_. The rules can be created with either a specific target project ID, or with a wild card (`*`) to target all projects. They can also overlap, allowing a network to be both external and shared to the same target projects. ### Address Allocation and Routing -CSPs can assign a subnet to a provider network to supply connected VMs or virtual routers with public IP addresses, making them externally accessible. -This works well for shared networks where VMs can be attached directly, although there is no quota option to limit the number of allocated addresses per project. +CSPs can assign a subnet to a provider network to supply connected servers or virtual routers with public IP addresses, making them externally accessible. +This works well for shared networks, where servers can be attached directly, although there is no quota option to limit the number of allocated addresses per project. -Making VMs in a project-internal network externally accessible through a virtual router is a bit more complicated, though. +Making servers in a project-internal network externally accessible through a virtual router is a bit more complicated, though. One option is for the user to create a subnet with a public IP range for the internal network, and then ask the CSP to configure a static route to the subnet via the gateway IP of a virtual router. This is cumbersome to set up manually, but can be automated with the `bgp` extension of the Network API, which is implemented by the `neutron-dynamic-routing` project [^bgp]. For users, this takes the form of a CSP-managed shared subnet pool that they can create externally routable subnets from, limited by a per-project quota. @@ -53,8 +73,8 @@ For IPv6, there is also the option of prefix delegation, where a DHCPv6 server a However, this also means that ports in the subnet can lose their addresses and get assigned new ones if the subnet is removed from the external network and later reattached. The documentation at [^pd] still marks prefix delegation as an experimental feature in Neutron and notes low test coverage. -For IPv4, OpenStack virtual routers support source NAT, allowing all VMs in the internal subnet to access the external network with the gateway IP of the virtual router. -They also support destination NAT in the form of floating IPs, addresses from the external network that can be mapped onto specific VMs in the internal subnet to make them externally accessible. +For IPv4, OpenStack virtual routers support source NAT, allowing all servers in the internal subnet to access the external network with the gateway IP of the virtual router. +They also support destination NAT in the form of floating IPs, addresses from the external network that can be mapped onto specific servers in the internal subnet to make them externally accessible. Floating IPs are allocated from a CSP-managed pool, and can be controlled by a per-project quota. There is also a set of API extensions that allow more fine grained port-forwarding, mapping different TCP or UDP ports of a floating IP to different internal IP addresses [^pf]. @@ -64,9 +84,9 @@ OpenStack networks have the flag `port_security_enabled`, that is set to true by In Neutron, besides enabling security groups for ports in this network, it also enables a number of built-in spoofing protections. Whether this flag is set is primarily of concern for shared provider networks, as users only have limited control over the gateway ports of virtual routers. -A lack of spoofing protection in a shared network, however, does enable a number of attacks that a malicious user or compromised VM could perform against other VMs in the network, such as DHCP-spoofing or ARP-Poisoning. +A lack of spoofing protection in a shared network, however, does enable a number of attacks that a malicious user or compromised server could perform against other servers in the network, such as DHCP-spoofing or ARP-Poisoning. -There are legitimate use-cases for networks without port security, such as the implementation of network function virtualisation (NFV) within a VM (i.e. using a VM as a virtual router). +There are legitimate use-cases for networks without port security, such as the implementation of network function virtualisation (NFV) within a server (i.e. having a server perform the function of a virtual router). This seems to be more of a niche use-case, however, and may warrant the creation of a project-specific provider network, rather than making all other projects vulnerable to spoofing attacks. ### Options considered @@ -82,7 +102,7 @@ Mandatory support for IPv6 but not IPv4 addresses this problem, while also provi #### Single Default Provider Network In principle, CSPs can create multiple provider networks for a number of reasons, for example to make cloud-internal shared services available to projects. -VMs can be connected to multiple networks, and connecting to additional provider networks would not interfere with their ability be externally accessible. +Servers can be connected to multiple networks, and connecting to additional provider networks would not interfere with their ability be externally accessible. CSPs may also create multiple provider networks with different options for external access, such as separate networks for IPv4 and IPv6, or one external network for use with virtual routers and a separate shared network for direct connection. This mostly just adds complexity to the setup, though, as a provider network can be both external and shared at the same time, and can even provide both IPv4 and IPv6 subnets in a dual stack setup [^ds]. @@ -94,12 +114,12 @@ A single default provider network leaves no ambiguity by the user in this regard #### Shared Provider Network A shared provider network has the benefit of being easy to use. -VMs can be attached directly and will be accessible immediately, without the requirement to create virtual routers, project-internal networks, subnets, or floating IPs. +Servers can be attached directly and will be accessible immediately, without the requirement to create virtual routers, project-internal networks, subnets, or floating IPs. -Users do have a higher control over VM ports than they have over virtual router ports, so it is important to enable Neutrons port security feature on shared provider networks to prevent spoofing. +Users do have a higher control over server ports than they have over virtual router ports, so it is important to enable Neutrons port security feature on shared provider networks to prevent spoofing. The lack of address quota may be a problem if IPv4 is used and the number of available addresses is so limited that fairness between projects needs to be enforced. -In that case, CSPs may restrict the number of VMs, routers, or ports to limit the addresses used by individual projects. +In that case, CSPs may restrict the number of servers, routers, or ports to limit the addresses used by individual projects. #### External Provider Network and Subnet Allocation @@ -136,19 +156,31 @@ Under these conditions, a malicious user could create a network with a misleadin For this attack to work, the attacker has to find out the target's project ID, and the target has to be sufficiently oblivious to the CSP's provider network setup. CSPs can try to educate users on the correct provider networks to use, and can avoid leaking project IDs, but the best protection is to disable the creation of RBAC rules for non-admin users. -## Decision +#### Required API extensions + +The OpenStack Network API has a more modular design than other OpenStack APIs. +New features are added as optional API extensions rather than a linear sequence of micro-version, and different Neutron core-plugins, service plugins, and mechanism drivers may provide different extensions. + +In practice, the great majority of OpenStack deployments will use Neutron with the ML2 core plugin and either the `router` or the `ovn-router` service plugins, which should cover all requirements of this standard. +As this standard tries to target the OpenStack API rather than it's specific implementation, we could consider standardizing a minimal set of API extensions. + +On the other hand, the mandatory set of API extensions follows directly from the mandated features, e.g. the `external-net`, `provider`, and `router` extensions, which must all be available if an external provider network is required. +So, just specifying mandatory features rather than a certain set of extensions may actually be preferable, as it removes redundancy and thus the potential for inconsistency. + +## Standard -CSPs **MUST** provide a standard provider network to every project to access to the internet. -There **SHOULD** be no other provider networks available to projects by default. +CSPs **MUST** provide a provider network to every project to access to the internet. +This provider network will in the following be referred to as the _standard provider network_. +To avoid ambiguity, the standard provider network **SHOULD** be the only provider networks available to projects by default. The standard provider network **MUST** be an external network, allowing it to be used as external gateway by virtual routers. -The standard provider network **MAY** be a shared network, allowing direct attachment of VMs. -If the standard provider network is a shared network, it **MUST** enable port security. +The standard provider network **MAY** be a shared network, allowing direct attachment of servers. +If the standard provider network is a shared network, it **MUST** enable port security to prevent projects from interfering with each other. -The external network **MUST** have an IPv6 subnet. +The standard provider network **MUST** have an IPv6 subnet. CSPs **MUST** provide a subnet pool for the allocation of at least one public /64 IPv6 prefix per project. -The external network **SHOULD** have an IPv4 subnet. +The standard provider network **SHOULD** have an IPv4 subnet. If CSPs provide an IPv4 subnet, then CSPs **MUST** provide at least one public Floating IP per project. They **MAY** also provide a subnet pool for the allocation of public IPv4 prefixes to project networks. @@ -161,3 +193,4 @@ By default, users **SHOULD** be prohibited by policy from creating RBAC rules fo [^bgp]: https://docs.openstack.org/neutron/2024.1/admin/config-bgp-dynamic-routing.html [^pd]: https://docs.openstack.org/neutron/2024.1/admin/config-ipv6.html#prefix-delegation [^pf]: https://docs.openstack.org/api-ref/network/v2/index.html#floating-ips-port-forwarding +[^ds]: https://docs.openstack.org/neutron/2024.1/admin/config-ipv6.html From a715f8cf2df86fcb6650e3dd2ac05a6cf2a9601d Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Wed, 5 Jun 2024 09:50:58 +0200 Subject: [PATCH 09/21] Rename provider network document Signed-off-by: Konrad Gube --- ...rking-standard.md => scs-xxxx-v1-provider-network-standard.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Standards/{scs-xxxx-v1-networking-standard.md => scs-xxxx-v1-provider-network-standard.md} (100%) diff --git a/Standards/scs-xxxx-v1-networking-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md similarity index 100% rename from Standards/scs-xxxx-v1-networking-standard.md rename to Standards/scs-xxxx-v1-provider-network-standard.md From b10b8b0ff486768d79f4ea6a0edffaef14053f63 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Wed, 5 Jun 2024 09:57:28 +0200 Subject: [PATCH 10/21] Add stub for conformance tests section Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-provider-network-standard.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index 66bf0a4e7..64173737b 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -188,6 +188,10 @@ CSPs **MUST** provide dynamic routing for all project-allocated public IP-prefix By default, users **SHOULD** be prohibited by policy from creating RBAC rules for networks in their projects, to prevent the creation of faux provider networks. +## Conformance Tests + +(TBD, current requirements should all be testable by API, though dynamic routing might be a bit tricky) + ## References [^bgp]: https://docs.openstack.org/neutron/2024.1/admin/config-bgp-dynamic-routing.html From 257b457555c587038d6f2bd326c8ac9848d1e9bd Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Mon, 10 Jun 2024 12:05:28 +0200 Subject: [PATCH 11/21] Update some descriptions Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-provider-network-standard.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index 64173737b..70fdabc34 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -34,7 +34,7 @@ The following terms are used throughout this document: | RBAC | Role-based Access Control: A mechanism in the Network API to give projects limited access to resources owned by other projects. Typically used by CSPs to create provider networks. | | Shared Network | Virtual network that is shared between projects in a way that allows direct attachment of servers. | | External Network | Virtual network that is shared between projects in a way that only allows virtual routers to use it as external gateway. Typically used by CSPs to provide access to networks outside of the cloud environment. | -| Provider Network | Any shared or external network that is managed by the CSP. | +| Provider Network | A CSP-managed virtual network made available to projects as either shared or external, typically connected to non-virtualized infrastructure. | ## Motivation @@ -54,6 +54,7 @@ This is also true for CSP-managed resources, such as provider networks, which ha The Network API's Role Based Access Control (RBAC) extension can then be used to share it with other projects. RBAC rules for networks support the two actions `access_as_external` and `access_as_shared`, and can be created automatically on `openstack network create` with the options `--external` and `--share`. * `access_as_external` allows networks to be used as external gateway for virtual routers in the target projects. Such networks are in the following referred to as _external networks_. + External networks have some special properties, such as allowing the creation of floating IPs, which will be discussed in the next section. * `access_as_shared` allows networks to be attached directly to servers in the target projects. Such networks are in the following referred to as _shared networks_. The rules can be created with either a specific target project ID, or with a wild card (`*`) to target all projects. @@ -80,8 +81,9 @@ There is also a set of API extensions that allow more fine grained port-forwardi ### Port Security and Spoofing -OpenStack networks have the flag `port_security_enabled`, that is set to true by default and can only be changed by it's owner. -In Neutron, besides enabling security groups for ports in this network, it also enables a number of built-in spoofing protections. +OpenStack ports have the flag `port_security_enabled`, that is set to true by default and can only be changed by the owner of the corresponding network. +The default value of that flag is controlled by a `port_security_enabled` flag on the network. +Besides enabling security groups for a port, it also enables a number of built-in spoofing protections. Whether this flag is set is primarily of concern for shared provider networks, as users only have limited control over the gateway ports of virtual routers. A lack of spoofing protection in a shared network, however, does enable a number of attacks that a malicious user or compromised server could perform against other servers in the network, such as DHCP-spoofing or ARP-Poisoning. From d5806596b9d8ac2b784bced3c871d6689eae6d27 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Mon, 10 Jun 2024 13:00:54 +0200 Subject: [PATCH 12/21] Fix linting issues Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-provider-network-standard.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index 70fdabc34..3a4938640 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -44,7 +44,7 @@ This standard identifies some of these options and defines a baseline setup that ## Design Considerations -This section will provide some general background on OpenStack provider networks before considering specific options to standardize. +This section will provide some general background on OpenStack provider networks before considering specific options to standardize. ### Provider Network Access Control @@ -53,6 +53,7 @@ This is also true for CSP-managed resources, such as provider networks, which ha The Network API's Role Based Access Control (RBAC) extension can then be used to share it with other projects. RBAC rules for networks support the two actions `access_as_external` and `access_as_shared`, and can be created automatically on `openstack network create` with the options `--external` and `--share`. + * `access_as_external` allows networks to be used as external gateway for virtual routers in the target projects. Such networks are in the following referred to as _external networks_. External networks have some special properties, such as allowing the creation of floating IPs, which will be discussed in the next section. * `access_as_shared` allows networks to be attached directly to servers in the target projects. Such networks are in the following referred to as _shared networks_. @@ -196,7 +197,7 @@ By default, users **SHOULD** be prohibited by policy from creating RBAC rules fo ## References -[^bgp]: https://docs.openstack.org/neutron/2024.1/admin/config-bgp-dynamic-routing.html -[^pd]: https://docs.openstack.org/neutron/2024.1/admin/config-ipv6.html#prefix-delegation -[^pf]: https://docs.openstack.org/api-ref/network/v2/index.html#floating-ips-port-forwarding -[^ds]: https://docs.openstack.org/neutron/2024.1/admin/config-ipv6.html +[^bgp]: +[^pd]: +[^pf]: +[^ds]: From 0c418f3fdd96b04d662f45522b62882b2fed39b5 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Tue, 16 Jul 2024 10:29:01 +0200 Subject: [PATCH 13/21] Update introduction and add internet access to considered options Signed-off-by: Konrad Gube --- .../scs-xxxx-v1-provider-network-standard.md | 47 ++++++++++++------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index 3a4938640..4bd021e14 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -7,14 +7,13 @@ track: IaaS ## Introduction -Connecting a virtual machine to the internet requires a virtual network that is also connected to the non-virtualised network infrastructure outside of the cloud environment. -Because such networks break the isolation of the cloud environment, they must be created and managed by the CSP and made available to projects only in a controlled manner. +Many use-cases of IaaS require virtual servers to be able to connect to network resources outside of the cloud environment, often to the internet. +Openstack supports this by allowing CSPs to map non-virtualized networks onto specific virtual networks, such that virtual routers and servers can connect to them. -In OpenStack, such CSP-managed networks are called _provider networks_, though providing access to the internet is not their only use case. -Depending on the configuration, they may impose different usage restrictions and offer different methods of allocating IP addresses. +Such networks will usually be created in a CSP-managed project and then shared to user projects using the network API's role-based access control feature. +Because they have to be set up by the cloud provider, networks of this type are generally referred to as _provider networks_, though that term can also be used to refer to other types of CSP-managed virtual networks. -External access to and from virtualised resources is a fundamental feature of IaaS, and providing this access is a hard requirement for any cloud provider. -However, OpenStack provider networks offer some design space regarding usage restrictions and IP address allocation. +When setting up provider networks for external access, CSPs have a number of different options regarding usage restrictions and the allocation of IP addresses. This document defines a standardized setup for provider networks that offers external access in a way that is portable across SCS clouds. ### Glossary @@ -29,7 +28,7 @@ The following terms are used throughout this document: | Virtual Router | OpenStack resource that can be used to route and bridge between virtual networks and provide other features such as NAT | | Subnet | Subdivision of available IP address space using address prefixes. In OpenStack also an abstraction for controlling IP address allocation in a virtual network. | | DHCP | Dynamic Host Configuration Protocol: Used to automatically configure hosts in a network with IP addresses, default routes, and other information such as DNS servers. | -| Prefix | IP address prefix of a given bit-length N, written _/N_. Divides addresses into a network and a host part, a shorter prefix allows more hosts but takes up more address space. | +| Prefix | IP address prefix of a given bit-length N, written as _
/_. Divides addresses into a network and a host part, a shorter prefix allows more hosts but takes up more address space. | | NAT | Network Address Translation: mapping (usually public) IPv4 addresses onto a different (usually private) address space. May allows multiple hosts to share a public address by multiplexing TCP/UDP ports. | | RBAC | Role-based Access Control: A mechanism in the Network API to give projects limited access to resources owned by other projects. Typically used by CSPs to create provider networks. | | Shared Network | Virtual network that is shared between projects in a way that allows direct attachment of servers. | @@ -38,7 +37,7 @@ The following terms are used throughout this document: ## Motivation -The ability to interface virtualised resources with networks outside of the cloud environment, such as the internet, is an integral part of IaaS. +The ability to interface virtualised resources with networks outside of the cloud environment, such as the internet, is an important part of IaaS. Providing external access in an OpenStack cloud requires a number of configuration choices from the CSP, some of which have direct implications on how users interact with the cloud. This standard identifies some of these options and defines a baseline setup that provides flexibility and consistency to users but is also realistic to implement for CSPs. @@ -63,16 +62,16 @@ They can also overlap, allowing a network to be both external and shared to the ### Address Allocation and Routing -CSPs can assign a subnet to a provider network to supply connected servers or virtual routers with public IP addresses, making them externally accessible. -This works well for shared networks, where servers can be attached directly, although there is no quota option to limit the number of allocated addresses per project. +CSPs can assign a subnet to a provider network to supply connected servers or virtual routers with externally routable (e.g. public) IP addresses. +This works well for shared networks, where servers can be attached directly, but there is no quota option to limit the number of allocated addresses per project. Making servers in a project-internal network externally accessible through a virtual router is a bit more complicated, though. -One option is for the user to create a subnet with a public IP range for the internal network, and then ask the CSP to configure a static route to the subnet via the gateway IP of a virtual router. -This is cumbersome to set up manually, but can be automated with the `bgp` extension of the Network API, which is implemented by the `neutron-dynamic-routing` project [^bgp]. -For users, this takes the form of a CSP-managed shared subnet pool that they can create externally routable subnets from, limited by a per-project quota. +One option is for the user to create a subnet with an external IP range for the internal network, and then ask the CSP to configure a static route to the subnet via the gateway IP of a virtual router. +This is cumbersome to set up manually, but can be automated using the `bgp` extension of the Network API, which is implemented by the `neutron-dynamic-routing` project [^bgp]. +For users, this takes the form of a CSP-managed shared subnet pool, which they can use to create externally routable subnets, limited by a per-project quota. For IPv6, there is also the option of prefix delegation, where a DHCPv6 server automatically assigns an IPv6 prefix to a subnet when it connects to the external provider network [^pd]. -However, this also means that ports in the subnet can lose their addresses and get assigned new ones if the subnet is removed from the external network and later reattached. +This means that ports in the subnet can lose their addresses and get assigned new ones if the subnet is removed from the external network and later reattached. The documentation at [^pd] still marks prefix delegation as an experimental feature in Neutron and notes low test coverage. For IPv4, OpenStack virtual routers support source NAT, allowing all servers in the internal subnet to access the external network with the gateway IP of the virtual router. @@ -94,6 +93,18 @@ This seems to be more of a niche use-case, however, and may warrant the creation ### Options considered +#### Internet Access + +For public clouds, external access generally means access to (and from) the internet, with allocation of public IP addresses. +Providing a standardized approach for internet access is the main motivation for this standard. + +However, the SCS Standards are intended to be applicable not just to public clouds, but also to private or even air-gaped cloud environments. +One way to reconcile these requirements would be to mandate internet access, but limit the scope of this standard to only cover public clouds. + +However, private clouds may also profit from a standardized approach to external access, even when external in this context may be a private network or VPN of an organization. +Even air-gaped environments may have use for standardized provider networks, e.g. to provide local network-based services such as NTP. +So, rather than scope all requirements to a specific type of cloud environment, this standard will scope individual requirements where necessary. + #### IPv6 The OpenStack Network API allows the creation of subnets with either IPv4 or IPv6 address ranges, as indicated by the `ip_version` field. @@ -108,11 +119,11 @@ In principle, CSPs can create multiple provider networks for a number of reasons Servers can be connected to multiple networks, and connecting to additional provider networks would not interfere with their ability be externally accessible. CSPs may also create multiple provider networks with different options for external access, such as separate networks for IPv4 and IPv6, or one external network for use with virtual routers and a separate shared network for direct connection. -This mostly just adds complexity to the setup, though, as a provider network can be both external and shared at the same time, and can even provide both IPv4 and IPv6 subnets in a dual stack setup [^ds]. +This mostly just adds complexity to the setup, however, as a provider network can be both external and shared at the same time, and can even provide both IPv4 and IPv6 subnets in a dual stack setup [^ds]. -Another problem with multiple provider networks is, that users may only be able to distinguish their respective function by their name. +Another problem with multiple provider networks is that users may only be able to distinguish their respective function by their name. -A single default provider network leaves no ambiguity by the user in this regard and is thus preferable from a standardisation perspective. +A single default provider network leaves no ambiguity by the user in this regard and is thus preferable from a standardization perspective. #### Shared Provider Network @@ -130,7 +141,7 @@ Creating a project-internal network to connect to an external provider network w In an internal network, users have greater control over IP allocation and may also choose to disable port security. With the FWaaS API extensions, they can also assign firewall rules to the virtual router, to control which traffic can pass between internal and provider networks. -As described above, there are multiple methods for allocating public IP addresses to project-internal networks. +As described above, there are multiple methods for allocating external IP addresses to project-internal networks. For IPv6, the currently best option seems to be subnet allocation from a CSP-managed subnet pool, because support for Prefix Delegation is still experimental. For IPv4, NAT and floating IPs are generally preferred over subnet allocation because of scarcity of IPv4 address space. From 67c20fdbf6dfc4c99d55fa182cc13c274add1b80 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Wed, 24 Jul 2024 12:52:52 +0200 Subject: [PATCH 14/21] Update standard to only apply to public IP allocation Signed-off-by: Konrad Gube --- .../scs-xxxx-v1-provider-network-standard.md | 42 +++++++++---------- 1 file changed, 20 insertions(+), 22 deletions(-) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index 4bd021e14..cd757a935 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -10,11 +10,11 @@ track: IaaS Many use-cases of IaaS require virtual servers to be able to connect to network resources outside of the cloud environment, often to the internet. Openstack supports this by allowing CSPs to map non-virtualized networks onto specific virtual networks, such that virtual routers and servers can connect to them. -Such networks will usually be created in a CSP-managed project and then shared to user projects using the network API's role-based access control feature. -Because they have to be set up by the cloud provider, networks of this type are generally referred to as _provider networks_, though that term can also be used to refer to other types of CSP-managed virtual networks. +Such networks will usually be created in a provider-managed project and then shared to user projects using the RBAC-feature of the network API. +Because they have to be set up by the cloud provider, networks of this type are generally referred to as _provider networks_, though that term can also be used in a broader senseto refer to other types of CSP-managed virtual networks. When setting up provider networks for external access, CSPs have a number of different options regarding usage restrictions and the allocation of IP addresses. -This document defines a standardized setup for provider networks that offers external access in a way that is portable across SCS clouds. +This document defines a standardized approach for using provider networks to allocate public IP addresses and provide external access in a way that is portable across SCS clouds. ### Glossary @@ -28,7 +28,7 @@ The following terms are used throughout this document: | Virtual Router | OpenStack resource that can be used to route and bridge between virtual networks and provide other features such as NAT | | Subnet | Subdivision of available IP address space using address prefixes. In OpenStack also an abstraction for controlling IP address allocation in a virtual network. | | DHCP | Dynamic Host Configuration Protocol: Used to automatically configure hosts in a network with IP addresses, default routes, and other information such as DNS servers. | -| Prefix | IP address prefix of a given bit-length N, written as _
/_. Divides addresses into a network and a host part, a shorter prefix allows more hosts but takes up more address space. | +| Prefix | IP address prefix of a given bit-length N, written as _ADDRESS/N_. Divides addresses into a network and a host part, a shorter prefix allows more hosts but takes up more address space. | | NAT | Network Address Translation: mapping (usually public) IPv4 addresses onto a different (usually private) address space. May allows multiple hosts to share a public address by multiplexing TCP/UDP ports. | | RBAC | Role-based Access Control: A mechanism in the Network API to give projects limited access to resources owned by other projects. Typically used by CSPs to create provider networks. | | Shared Network | Virtual network that is shared between projects in a way that allows direct attachment of servers. | @@ -93,17 +93,17 @@ This seems to be more of a niche use-case, however, and may warrant the creation ### Options considered -#### Internet Access +#### Public IP Address Allocation For public clouds, external access generally means access to (and from) the internet, with allocation of public IP addresses. -Providing a standardized approach for internet access is the main motivation for this standard. +Providing a standardized approach for the allocation of public IP addresses is the main motivation for this standard. However, the SCS Standards are intended to be applicable not just to public clouds, but also to private or even air-gaped cloud environments. -One way to reconcile these requirements would be to mandate internet access, but limit the scope of this standard to only cover public clouds. +One way to reconcile these requirements would be to find a common basis between public and private clouds, and then build the standard around that, scoping individual rules where necessary. -However, private clouds may also profit from a standardized approach to external access, even when external in this context may be a private network or VPN of an organization. -Even air-gaped environments may have use for standardized provider networks, e.g. to provide local network-based services such as NTP. -So, rather than scope all requirements to a specific type of cloud environment, this standard will scope individual requirements where necessary. +However, private and public clouds may have quite different requirements. +Public IPv4 addresses, for example, are sparse and expensive, so having tight quotas and support for NAT makes a lot of sense in a public cloud, but may be completely unnecessary in a private environment without public IPs. +So, rather than trying to find common ground between the networking requirements of all SCS clouds, the requirements of this standard will be scoped to only apply to cloud environment that support the allocation of public IP addresses. #### IPv6 @@ -154,7 +154,7 @@ Each subnet also has a broadcast and a network address, which for small subnets Source NAT, combined with selective use of floating IPs can significantly reduce the number of required addresses over a public IPv4 subnet. The floating IP quota also offers a finer granularity for distributing IPs among projects, though it is important to note that the routers external gateway IP which is used for the source NAT is not subject to any quotas. -IPv4 NAT can also be used in a dual stack setup alongside a routed IPv6 subnet (source?). +IPv4 NAT can also be used in a dual stack setup alongside a routed IPv6 subnet. #### Disable RBAC for Users @@ -183,28 +183,26 @@ So, just specifying mandatory features rather than a certain set of extensions m ## Standard -CSPs **MUST** provide a provider network to every project to access to the internet. +If CSPs offer public IP addresses to projects, they **MUST** provide a provider network to every project to allocate public addresses. This provider network will in the following be referred to as the _standard provider network_. -To avoid ambiguity, the standard provider network **SHOULD** be the only provider networks available to projects by default. +To avoid ambiguity, the standard provider network **SHOULD** be the only provider network available to projects by default. The standard provider network **MUST** be an external network, allowing it to be used as external gateway by virtual routers. -The standard provider network **MAY** be a shared network, allowing direct attachment of servers. +The standard provider network **MAY** be a shared network, allowing direct attachment of virtual servers. If the standard provider network is a shared network, it **MUST** enable port security to prevent projects from interfering with each other. -The standard provider network **MUST** have an IPv6 subnet. -CSPs **MUST** provide a subnet pool for the allocation of at least one public /64 IPv6 prefix per project. +If CSPs offer public IP addresses at all, they **MUST** provide a subnet pool for the allocation of at least one public /64 IPv6 prefix per project. +If CSPs offer public IP addresses, they **SHOULD** also offer public IPv4 addresses. +If they do offer public IPv4 addresses, they **MUST** provide at least one public Floating IP per project, but **MAY** also provide a subnet pool for the allocation of public IPv4 prefixes to project networks. -The standard provider network **SHOULD** have an IPv4 subnet. -If CSPs provide an IPv4 subnet, then CSPs **MUST** provide at least one public Floating IP per project. -They **MAY** also provide a subnet pool for the allocation of public IPv4 prefixes to project networks. - -CSPs **MUST** provide dynamic routing for all project-allocated public IP-prefixes. +CSPs **MUST** externally route any public IP addresses allocated from subnets of the standard provider network. +CSPs **MUST** provide dynamic routing for all project-allocated public IP-prefixes via the standard provider network. By default, users **SHOULD** be prohibited by policy from creating RBAC rules for networks in their projects, to prevent the creation of faux provider networks. ## Conformance Tests -(TBD, current requirements should all be testable by API, though dynamic routing might be a bit tricky) +(TBD, current requirements should mostly be testable by API. Testing external routing is more tricky and will require external testing infrastructure of some sort) ## References From 2e06a8643c1c80cfde6a62f2db3a9dd610ed9c03 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Tue, 27 Aug 2024 15:15:22 +0200 Subject: [PATCH 15/21] Add reference to ovn-bgp-agent Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-provider-network-standard.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index cd757a935..82feb265b 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -67,7 +67,7 @@ This works well for shared networks, where servers can be attached directly, but Making servers in a project-internal network externally accessible through a virtual router is a bit more complicated, though. One option is for the user to create a subnet with an external IP range for the internal network, and then ask the CSP to configure a static route to the subnet via the gateway IP of a virtual router. -This is cumbersome to set up manually, but can be automated using the `bgp` extension of the Network API, which is implemented by the `neutron-dynamic-routing` project [^bgp]. +This is cumbersome to set up manually, but can be automated using the `bgp` extension of the Network API, which is currently implemented both by the `neutron-dynamic-routing` project [^bgp] and by the `ovn` mechanism driver when used with the `ovn-bgp-agent` [^ovn-bgp]. For users, this takes the form of a CSP-managed shared subnet pool, which they can use to create externally routable subnets, limited by a per-project quota. For IPv6, there is also the option of prefix delegation, where a DHCPv6 server automatically assigns an IPv6 prefix to a subnet when it connects to the external provider network [^pd]. @@ -207,6 +207,7 @@ By default, users **SHOULD** be prohibited by policy from creating RBAC rules fo ## References [^bgp]: +[^ovn-bgp]: [^pd]: [^pf]: [^ds]: From cf02189821bb7e4ae57b5c3367fc9168ffea931f Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Fri, 13 Sep 2024 17:50:34 +0200 Subject: [PATCH 16/21] Change "Network API" to "Networking API" Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-provider-network-standard.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index 82feb265b..7b9fc2dbe 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -10,7 +10,7 @@ track: IaaS Many use-cases of IaaS require virtual servers to be able to connect to network resources outside of the cloud environment, often to the internet. Openstack supports this by allowing CSPs to map non-virtualized networks onto specific virtual networks, such that virtual routers and servers can connect to them. -Such networks will usually be created in a provider-managed project and then shared to user projects using the RBAC-feature of the network API. +Such networks will usually be created in a provider-managed project and then shared to user projects using the RBAC-feature of the Networking API. Because they have to be set up by the cloud provider, networks of this type are generally referred to as _provider networks_, though that term can also be used in a broader senseto refer to other types of CSP-managed virtual networks. When setting up provider networks for external access, CSPs have a number of different options regarding usage restrictions and the allocation of IP addresses. @@ -30,7 +30,7 @@ The following terms are used throughout this document: | DHCP | Dynamic Host Configuration Protocol: Used to automatically configure hosts in a network with IP addresses, default routes, and other information such as DNS servers. | | Prefix | IP address prefix of a given bit-length N, written as _ADDRESS/N_. Divides addresses into a network and a host part, a shorter prefix allows more hosts but takes up more address space. | | NAT | Network Address Translation: mapping (usually public) IPv4 addresses onto a different (usually private) address space. May allows multiple hosts to share a public address by multiplexing TCP/UDP ports. | -| RBAC | Role-based Access Control: A mechanism in the Network API to give projects limited access to resources owned by other projects. Typically used by CSPs to create provider networks. | +| RBAC | Role-based Access Control: A mechanism in the Networking API to give projects limited access to resources owned by other projects. Typically used by CSPs to create provider networks. | | Shared Network | Virtual network that is shared between projects in a way that allows direct attachment of servers. | | External Network | Virtual network that is shared between projects in a way that only allows virtual routers to use it as external gateway. Typically used by CSPs to provide access to networks outside of the cloud environment. | | Provider Network | A CSP-managed virtual network made available to projects as either shared or external, typically connected to non-virtualized infrastructure. | @@ -50,7 +50,7 @@ This section will provide some general background on OpenStack provider networks In OpenStack, ownership of resources is generally tracked through projects, and, as per default policy, only members of a project have access to its resources This is also true for CSP-managed resources, such as provider networks, which have to be created in a CSP-internal project, and are initially only accessible in this project. -The Network API's Role Based Access Control (RBAC) extension can then be used to share it with other projects. +The Networking API's Role Based Access Control (RBAC) extension can then be used to share it with other projects. RBAC rules for networks support the two actions `access_as_external` and `access_as_shared`, and can be created automatically on `openstack network create` with the options `--external` and `--share`. * `access_as_external` allows networks to be used as external gateway for virtual routers in the target projects. Such networks are in the following referred to as _external networks_. @@ -67,7 +67,7 @@ This works well for shared networks, where servers can be attached directly, but Making servers in a project-internal network externally accessible through a virtual router is a bit more complicated, though. One option is for the user to create a subnet with an external IP range for the internal network, and then ask the CSP to configure a static route to the subnet via the gateway IP of a virtual router. -This is cumbersome to set up manually, but can be automated using the `bgp` extension of the Network API, which is currently implemented both by the `neutron-dynamic-routing` project [^bgp] and by the `ovn` mechanism driver when used with the `ovn-bgp-agent` [^ovn-bgp]. +This is cumbersome to set up manually, but can be automated using the `bgp` extension of the Networking API, which is currently implemented both by the `neutron-dynamic-routing` project [^bgp] and by the `ovn` mechanism driver when used with the `ovn-bgp-agent` [^ovn-bgp]. For users, this takes the form of a CSP-managed shared subnet pool, which they can use to create externally routable subnets, limited by a per-project quota. For IPv6, there is also the option of prefix delegation, where a DHCPv6 server automatically assigns an IPv6 prefix to a subnet when it connects to the external provider network [^pd]. @@ -107,7 +107,7 @@ So, rather than trying to find common ground between the networking requirements #### IPv6 -The OpenStack Network API allows the creation of subnets with either IPv4 or IPv6 address ranges, as indicated by the `ip_version` field. +The OpenStack Networking API allows the creation of subnets with either IPv4 or IPv6 address ranges, as indicated by the `ip_version` field. However, to allow external access to either, the CSP needs to provide projects with externally routable addresses for that IP version. While it is possible (and common) for CSPs to provide both IPv4 and IPv6, the increasing scarcity (and cost) of IPv4 address space may at some point become a barrier to entry for new CSPs. @@ -172,7 +172,7 @@ CSPs can try to educate users on the correct provider networks to use, and can a #### Required API extensions -The OpenStack Network API has a more modular design than other OpenStack APIs. +The OpenStack Networking API has a more modular design than other OpenStack APIs. New features are added as optional API extensions rather than a linear sequence of micro-version, and different Neutron core-plugins, service plugins, and mechanism drivers may provide different extensions. In practice, the great majority of OpenStack deployments will use Neutron with the ML2 core plugin and either the `router` or the `ovn-router` service plugins, which should cover all requirements of this standard. From cb2fadafa4e4d004fc2931f700749c4dea801786 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Fri, 13 Sep 2024 18:58:09 +0200 Subject: [PATCH 17/21] Fix some typos and wordings Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-provider-network-standard.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index 7b9fc2dbe..c089a380d 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -8,10 +8,10 @@ track: IaaS ## Introduction Many use-cases of IaaS require virtual servers to be able to connect to network resources outside of the cloud environment, often to the internet. -Openstack supports this by allowing CSPs to map non-virtualized networks onto specific virtual networks, such that virtual routers and servers can connect to them. +OpenStack supports this by allowing CSPs to map non-virtualized networks onto specific virtual networks, such that virtual routers and servers can connect to them. Such networks will usually be created in a provider-managed project and then shared to user projects using the RBAC-feature of the Networking API. -Because they have to be set up by the cloud provider, networks of this type are generally referred to as _provider networks_, though that term can also be used in a broader senseto refer to other types of CSP-managed virtual networks. +Because they have to be set up by the cloud provider, networks of this type are generally referred to as _provider networks_, though that term can also be used in a broader sense to refer to other types of CSP-managed virtual networks. When setting up provider networks for external access, CSPs have a number of different options regarding usage restrictions and the allocation of IP addresses. This document defines a standardized approach for using provider networks to allocate public IP addresses and provide external access in a way that is portable across SCS clouds. @@ -158,7 +158,7 @@ IPv4 NAT can also be used in a dual stack setup alongside a routed IPv6 subnet. #### Disable RBAC for Users -Per default policy, Neutron allows any user the creation RBAC rules to share resources of their projects with other projects. +Per default policy, Neutron allows any user the creation of RBAC rules to share resources of their projects with other projects. Only the use of the `*` wildcard target is limited to admin users. However, how a network was shared, and who shared it, is not immediately obvious from the perspective of the target project. @@ -167,7 +167,7 @@ And even though users can determine the project ID of networks by using `network Under these conditions, a malicious user could create a network with a misleading name, share it with target projects to trick them into using it like a provider network, and then intercept their traffic. -For this attack to work, the attacker has to find out the target's project ID, and the target has to be sufficiently oblivious to the CSP's provider network setup. +For this attack to work, the attacker has to find out the target's project ID, and the target has to be sufficiently oblivious to the CSPs provider network setup. CSPs can try to educate users on the correct provider networks to use, and can avoid leaking project IDs, but the best protection is to disable the creation of RBAC rules for non-admin users. #### Required API extensions @@ -175,8 +175,8 @@ CSPs can try to educate users on the correct provider networks to use, and can a The OpenStack Networking API has a more modular design than other OpenStack APIs. New features are added as optional API extensions rather than a linear sequence of micro-version, and different Neutron core-plugins, service plugins, and mechanism drivers may provide different extensions. -In practice, the great majority of OpenStack deployments will use Neutron with the ML2 core plugin and either the `router` or the `ovn-router` service plugins, which should cover all requirements of this standard. -As this standard tries to target the OpenStack API rather than it's specific implementation, we could consider standardizing a minimal set of API extensions. +In practice, the great majority of OpenStack deployments will use Neutron with the ML2 core plugin and either the `router` or the `ovn-router` service plugins, which should support all extensions required by this standard. +Because this standard tries to target the OpenStack API, rather than it's specific implementation, we might consider standardizing a minimal set of Networking API extensions that CSPs must provide. On the other hand, the mandatory set of API extensions follows directly from the mandated features, e.g. the `external-net`, `provider`, and `router` extensions, which must all be available if an external provider network is required. So, just specifying mandatory features rather than a certain set of extensions may actually be preferable, as it removes redundancy and thus the potential for inconsistency. From 3308a9e66e6cc11d6d60b0b1da8935221a15e1f0 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Fri, 13 Sep 2024 19:32:04 +0200 Subject: [PATCH 18/21] Use "Networking RBAC" to refer to the RBAC feature of the API Signed-off-by: Konrad Gube --- .../scs-xxxx-v1-provider-network-standard.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index c089a380d..f1845b879 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -10,7 +10,7 @@ track: IaaS Many use-cases of IaaS require virtual servers to be able to connect to network resources outside of the cloud environment, often to the internet. OpenStack supports this by allowing CSPs to map non-virtualized networks onto specific virtual networks, such that virtual routers and servers can connect to them. -Such networks will usually be created in a provider-managed project and then shared to user projects using the RBAC-feature of the Networking API. +Such networks will usually be created in a provider-managed project and then shared to user projects using access control features of the Networking API. Because they have to be set up by the cloud provider, networks of this type are generally referred to as _provider networks_, though that term can also be used in a broader sense to refer to other types of CSP-managed virtual networks. When setting up provider networks for external access, CSPs have a number of different options regarding usage restrictions and the allocation of IP addresses. @@ -30,7 +30,7 @@ The following terms are used throughout this document: | DHCP | Dynamic Host Configuration Protocol: Used to automatically configure hosts in a network with IP addresses, default routes, and other information such as DNS servers. | | Prefix | IP address prefix of a given bit-length N, written as _ADDRESS/N_. Divides addresses into a network and a host part, a shorter prefix allows more hosts but takes up more address space. | | NAT | Network Address Translation: mapping (usually public) IPv4 addresses onto a different (usually private) address space. May allows multiple hosts to share a public address by multiplexing TCP/UDP ports. | -| RBAC | Role-based Access Control: A mechanism in the Networking API to give projects limited access to resources owned by other projects. Typically used by CSPs to create provider networks. | +| Networking RBAC | A mechanism in the Networking API to give projects limited access to resources owned by other projects. Used by CSPs to create provider networks. Not to be confused with the role-based access control of OpenStack's policy mechanism. | | Shared Network | Virtual network that is shared between projects in a way that allows direct attachment of servers. | | External Network | Virtual network that is shared between projects in a way that only allows virtual routers to use it as external gateway. Typically used by CSPs to provide access to networks outside of the cloud environment. | | Provider Network | A CSP-managed virtual network made available to projects as either shared or external, typically connected to non-virtualized infrastructure. | @@ -50,8 +50,10 @@ This section will provide some general background on OpenStack provider networks In OpenStack, ownership of resources is generally tracked through projects, and, as per default policy, only members of a project have access to its resources This is also true for CSP-managed resources, such as provider networks, which have to be created in a CSP-internal project, and are initially only accessible in this project. -The Networking API's Role Based Access Control (RBAC) extension can then be used to share it with other projects. -RBAC rules for networks support the two actions `access_as_external` and `access_as_shared`, and can be created automatically on `openstack network create` with the options `--external` and `--share`. +They can then be made available to other projects by using the Networking API's Role-based Access Control (_RBAC_) extension. +(This is it's official name in the documentation, even though it isn't technically role-based. To avoid confusion with the actual RBAC of OpenStack's API authorization, it will in the following referred to as _Networking RBAC_.) + +Networking RBAC rules for virtual networks support the two actions `access_as_external` and `access_as_shared`, and can be created automatically on `openstack network create` with the options `--external` and `--share`. * `access_as_external` allows networks to be used as external gateway for virtual routers in the target projects. Such networks are in the following referred to as _external networks_. External networks have some special properties, such as allowing the creation of floating IPs, which will be discussed in the next section. @@ -156,9 +158,9 @@ The floating IP quota also offers a finer granularity for distributing IPs among IPv4 NAT can also be used in a dual stack setup alongside a routed IPv6 subnet. -#### Disable RBAC for Users +#### Disable Networking RBAC for Users -Per default policy, Neutron allows any user the creation of RBAC rules to share resources of their projects with other projects. +Per default policy, Neutron allows any user the creation of Networking RBAC rules to share resources of their projects with other projects. Only the use of the `*` wildcard target is limited to admin users. However, how a network was shared, and who shared it, is not immediately obvious from the perspective of the target project. @@ -168,7 +170,7 @@ And even though users can determine the project ID of networks by using `network Under these conditions, a malicious user could create a network with a misleading name, share it with target projects to trick them into using it like a provider network, and then intercept their traffic. For this attack to work, the attacker has to find out the target's project ID, and the target has to be sufficiently oblivious to the CSPs provider network setup. -CSPs can try to educate users on the correct provider networks to use, and can avoid leaking project IDs, but the best protection is to disable the creation of RBAC rules for non-admin users. +CSPs can try to educate users on the correct provider networks to use, and can avoid leaking project IDs, but the best protection is to disable the creation of Networking RBAC rules for non-admin users. #### Required API extensions @@ -198,7 +200,7 @@ If they do offer public IPv4 addresses, they **MUST** provide at least one publi CSPs **MUST** externally route any public IP addresses allocated from subnets of the standard provider network. CSPs **MUST** provide dynamic routing for all project-allocated public IP-prefixes via the standard provider network. -By default, users **SHOULD** be prohibited by policy from creating RBAC rules for networks in their projects, to prevent the creation of faux provider networks. +By default, users **SHOULD** be prohibited by policy from creating Networking RBAC rules, to prevent the creation of faux provider networks. ## Conformance Tests From 7eb49c62f5b392bff4e965c1e4a144f99f5f0dd2 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Wed, 18 Sep 2024 12:18:10 +0200 Subject: [PATCH 19/21] Add requirement for auto-allocation support. Signed-off-by: Konrad Gube --- .../scs-xxxx-v1-provider-network-standard.md | 22 +++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index f1845b879..147a5a3e6 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -158,6 +158,20 @@ The floating IP quota also offers a finer granularity for distributing IPs among IPv4 NAT can also be used in a dual stack setup alongside a routed IPv6 subnet. +#### Support for Auto-Allocation + +The `auto-allocated-topology` extension of the Networking API can be used to create a default network setup in a project with a single API call [^aa]. +It will create a project network with a subnet from either an IPv4 or IPv6 subnet pool (or one of each, if configured that way), as well as a router connecting it to a provider network. +The OpenStack Compute API also has support for this feature, allowing automatic network allocation on server creation. + +To function for a project, auto-allocation requires one external provider network and one shared subnet pool to be available, both with the `is_default` flag set to True. +It is possible to have both an IPv4 and an IPv6 subnet pool where `is_default` is set, in which case a dual-stack setup will be allocated. +The behavior is undefined when more than one network in the project is marked as default, or more than one subnet pool per address family. + +So, it is strongly advisable to only have one default defined for each. +It seems sensible to standardize on using SCS-mandated resources as auto-allocation defaults, as this is likely to be the expected behavior from users. +It can also be useful for both users and automated compliance tests to determine the defaults in the presence of multiple provider networks. + #### Disable Networking RBAC for Users Per default policy, Neutron allows any user the creation of Networking RBAC rules to share resources of their projects with other projects. @@ -190,12 +204,15 @@ This provider network will in the following be referred to as the _standard prov To avoid ambiguity, the standard provider network **SHOULD** be the only provider network available to projects by default. The standard provider network **MUST** be an external network, allowing it to be used as external gateway by virtual routers. +The standard provider network **MUST** have the `is_default` flag set to True, and it **MUST** be the only network to do so. The standard provider network **MAY** be a shared network, allowing direct attachment of virtual servers. If the standard provider network is a shared network, it **MUST** enable port security to prevent projects from interfering with each other. -If CSPs offer public IP addresses at all, they **MUST** provide a subnet pool for the allocation of at least one public /64 IPv6 prefix per project. +If CSPs offer public IP addresses at all, they **MUST** provide a shared subnet pool for the allocation of at least one public /64 IPv6 prefix per project. +This subnet pool **MUST** have the `is_default` flag set to True, and it **MUST** be the only IPv6 subnet pool to do so. If CSPs offer public IP addresses, they **SHOULD** also offer public IPv4 addresses. -If they do offer public IPv4 addresses, they **MUST** provide at least one public Floating IP per project, but **MAY** also provide a subnet pool for the allocation of public IPv4 prefixes to project networks. +If they do offer public IPv4 addresses, they **MUST** provide at least one public Floating IP per project, but **MAY** also provide a shared subnet pool for the allocation of public IPv4 prefixes to project networks. +If CSPs offer a subnet pool for the allocation of public IPv4 prefixes, it **MUST** have the `is_default` flag set to True, and it **MUST** be the only IPv4 subnet pool to do so. CSPs **MUST** externally route any public IP addresses allocated from subnets of the standard provider network. CSPs **MUST** provide dynamic routing for all project-allocated public IP-prefixes via the standard provider network. @@ -213,3 +230,4 @@ By default, users **SHOULD** be prohibited by policy from creating Networking RB [^pd]: [^pf]: [^ds]: +[^aa]: From 337b0cdbf10b9f27acf9c56777f05b7158ce4d40 Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Mon, 11 Nov 2024 11:32:50 +0100 Subject: [PATCH 20/21] Add reference for RBAC policy change Signed-off-by: Konrad Gube --- Standards/scs-xxxx-v1-provider-network-standard.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index 147a5a3e6..04e6574c0 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -218,6 +218,7 @@ CSPs **MUST** externally route any public IP addresses allocated from subnets of CSPs **MUST** provide dynamic routing for all project-allocated public IP-prefixes via the standard provider network. By default, users **SHOULD** be prohibited by policy from creating Networking RBAC rules, to prevent the creation of faux provider networks. +The necessary policy change to implement this restriction for the Neutron API can be found in the Networking RBAC documentation [^rbac]. ## Conformance Tests @@ -231,3 +232,4 @@ By default, users **SHOULD** be prohibited by policy from creating Networking RB [^pf]: [^ds]: [^aa]: +[^rbac]: From cc2286c457aa2bde0c87efd6af8a4d24d6ea01ce Mon Sep 17 00:00:00 2001 From: Konrad Gube Date: Mon, 11 Nov 2024 11:41:25 +0100 Subject: [PATCH 21/21] Update standard to require "external" IPs instead of public IPs Signed-off-by: Konrad Gube --- .../scs-xxxx-v1-provider-network-standard.md | 58 ++++++++++++------- 1 file changed, 38 insertions(+), 20 deletions(-) diff --git a/Standards/scs-xxxx-v1-provider-network-standard.md b/Standards/scs-xxxx-v1-provider-network-standard.md index 04e6574c0..327c4aebb 100644 --- a/Standards/scs-xxxx-v1-provider-network-standard.md +++ b/Standards/scs-xxxx-v1-provider-network-standard.md @@ -100,12 +100,14 @@ This seems to be more of a niche use-case, however, and may warrant the creation For public clouds, external access generally means access to (and from) the internet, with allocation of public IP addresses. Providing a standardized approach for the allocation of public IP addresses is the main motivation for this standard. -However, the SCS Standards are intended to be applicable not just to public clouds, but also to private or even air-gaped cloud environments. -One way to reconcile these requirements would be to find a common basis between public and private clouds, and then build the standard around that, scoping individual rules where necessary. +However, the SCS Standards are intended to be applicable not just to public clouds, but also to private or even air-gapped cloud environments which have different requirements for IP address allocation. +One way to reconcile these requirements would be to limit the scope the requirements of this standard to only apply to cloud environment that support the allocation of public IP addresses. -However, private and public clouds may have quite different requirements. -Public IPv4 addresses, for example, are sparse and expensive, so having tight quotas and support for NAT makes a lot of sense in a public cloud, but may be completely unnecessary in a private environment without public IPs. -So, rather than trying to find common ground between the networking requirements of all SCS clouds, the requirements of this standard will be scoped to only apply to cloud environment that support the allocation of public IP addresses. +There are common networking use-cases between public and private clouds, however, like the very common requirement to connect to virtual servers via SSH. +For users (or any automated tooling that users might deploy) to have a standardized way of connecting to virtual servers in their projects is very valuable, regardless of whether this access happens across the Internet or within an isolated internal network or VPN. + +So instead of standardizing allocation of public IP addresses for some clouds, it might be more useful to require the allocation of IP addresses that are reachable for API users in all clouds. +Public clouds can then have the additional requirement that those allocated addresses must also be public. #### IPv6 @@ -169,7 +171,7 @@ It is possible to have both an IPv4 and an IPv6 subnet pool where `is_default` i The behavior is undefined when more than one network in the project is marked as default, or more than one subnet pool per address family. So, it is strongly advisable to only have one default defined for each. -It seems sensible to standardize on using SCS-mandated resources as auto-allocation defaults, as this is likely to be the expected behavior from users. +It seems sensible to standardize on using SCS-mandated resources as auto-allocation defaults, as this is likely to be the behavior expected by users. It can also be useful for both users and automated compliance tests to determine the defaults in the presence of multiple provider networks. #### Disable Networking RBAC for Users @@ -199,23 +201,39 @@ So, just specifying mandatory features rather than a certain set of extensions m ## Standard -If CSPs offer public IP addresses to projects, they **MUST** provide a provider network to every project to allocate public addresses. -This provider network will in the following be referred to as the _standard provider network_. +### External Addresses + +We define an _external_ address as an IPv4 or IPv6 address that is accessible to a user of the SCS-compliant OpenStack API. +For public clouds, external addresses **MUST** be either IPv4 public addresses or IPv6 global unicast addresses (GUA). + +The CSP **MUST** offer allocation of external IPv6 addresses to user projects. +The CSP **SHOULD** offer allocation of external IPv4 addresses to user projects. + +### Standard Provider Network + +The CSP **MUST** provide every user project with a provider network that can route any external addresses that are allocated to the project. +For external addresses from pool-allocated subnets, this requires support for dynamic routing. +This provider network will in the following will be referred to as the _standard provider network_. + +The standard provider network **MUST** have the `is_default` flag set to `True`, and it **MUST** be the only provider network in the project with `is_default` set to `True`. To avoid ambiguity, the standard provider network **SHOULD** be the only provider network available to projects by default. The standard provider network **MUST** be an external network, allowing it to be used as external gateway by virtual routers. -The standard provider network **MUST** have the `is_default` flag set to True, and it **MUST** be the only network to do so. -The standard provider network **MAY** be a shared network, allowing direct attachment of virtual servers. -If the standard provider network is a shared network, it **MUST** enable port security to prevent projects from interfering with each other. - -If CSPs offer public IP addresses at all, they **MUST** provide a shared subnet pool for the allocation of at least one public /64 IPv6 prefix per project. -This subnet pool **MUST** have the `is_default` flag set to True, and it **MUST** be the only IPv6 subnet pool to do so. -If CSPs offer public IP addresses, they **SHOULD** also offer public IPv4 addresses. -If they do offer public IPv4 addresses, they **MUST** provide at least one public Floating IP per project, but **MAY** also provide a shared subnet pool for the allocation of public IPv4 prefixes to project networks. -If CSPs offer a subnet pool for the allocation of public IPv4 prefixes, it **MUST** have the `is_default` flag set to True, and it **MUST** be the only IPv4 subnet pool to do so. - -CSPs **MUST** externally route any public IP addresses allocated from subnets of the standard provider network. -CSPs **MUST** provide dynamic routing for all project-allocated public IP-prefixes via the standard provider network. +It **MAY** also be a shared network, allowing direct attachment of virtual servers. +If the standard provider network is a shared network, then it **MUST** be configured allocate external addresses for directly attached servers and ports, and it **MUST** enable port security to prevent projects from interfering with each other. + +### IPv6 Allocation + +The CSP **MUST** provide a shared subnet pool for the allocation of at least one /64 prefix for external IPv6 addresses per project. +This subnet pool **MUST** have the `is_default` flag set to `True`, and it **MUST** be the only shared IPv6 subnet pool in the project with `is_default` set to `True`. + +### IPv4 Allocation + +If the CSP offers external IPv4 addresses, they **MUST** provide at least one external Floating IP per project that can be allocated from the standard provider network. +The CSP **MAY** also provide a shared subnet pool for the allocation of prefixes for external IPv4 addresses to project networks. +If such a subnet pool is provided, it **MUST** have the `is_default` flag set to `True`, and it **MUST** be the only shared IPv4 subnet pool in the project with `is_default` set to `True`. + +### RBAC Restrictions By default, users **SHOULD** be prohibited by policy from creating Networking RBAC rules, to prevent the creation of faux provider networks. The necessary policy change to implement this restriction for the Neutron API can be found in the Networking RBAC documentation [^rbac].