This document introduces the concepts that you need to understand to configure
a Google Cloud external proxy Network Load Balancer.
The external proxy Network Load Balancer is a reverse proxy load balancer that distributes TCP
traffic coming from the internet to virtual machine (VM) instances in your
Google Cloud Virtual Private Cloud (VPC) network. When using an
external proxy Network Load Balancer, incoming TCP or SSL
traffic is terminated at the load balancer. A new connection then forwards
traffic to the closest available backend by using either TCP or SSL
(recommended). For more use cases, see [Proxy Network Load Balancer
overview](https://docs.cloud.google.com/load-balancing/docs/proxy-network-load-balancer).
External proxy Network Load Balancers let you use a single IP address for
all users worldwide. The load balancer automatically routes
traffic to the backends that are closest to the user.
In this example, SSL traffic from users in City A and City B is terminated
at the load balancing layer, and a separate connection is established
to the selected backend.
[](https://docs.cloud.google.com/static/load-balancing/images/ssl-lb-overview.svg) Cloud Load Balancing with SSL termination (click to enlarge).
> [!NOTE]
> **Note:** Although external proxy Network Load Balancers can support HTTPS traffic, you should use an external Application Load Balancer for HTTPS traffic instead. External Application Load Balancers support a number of HTTP-specific features, including routing by HTTP request path and balancing by request rate. For more details, see the [External Application Load Balancer
> overview](https://docs.cloud.google.com/load-balancing/docs/https).
## Modes of operation
You can configure an external proxy Network Load Balancer in the following modes:
- A **classic proxy Network Load Balancer** is implemented on globally distributed [Google Front Ends (GFEs)](https://docs.cloud.google.com/docs/security/infrastructure/design.md.txt#google-frontend-service). This load balancer can be configured to handle either TCP or SSL traffic by using either a target TCP proxy or a target SSL proxy respectively. With the Premium Tier, this load balancer can be configured as a global load balancing service. With Standard Tier, this load balancer is configured as a regional load balancing service. Classic proxy Network Load Balancers can also be used for other protocols that use SSL, such as WebSockets and IMAP over SSL.
- A **global external proxy Network Load Balancer** is implemented on globally distributed [GFEs](https://docs.cloud.google.com/docs/security/infrastructure/design.md.txt#google-frontend-service) and supports [advanced traffic management](https://docs.cloud.google.com/load-balancing/docs/tcp.md.txt#gxproxy-traffic-distribution) capabilities. This load balancer can be configured to handle either TCP or SSL traffic by using either a target TCP proxy or a target SSL proxy respectively. This load balancer is configured as a global load balancing service with the Premium Tier. Global external proxy Network Load Balancers can also be used for other protocols that use SSL, such as WebSockets and IMAP over SSL.
- A **regional external proxy Network Load Balancer** is implemented on the open source [Envoy
proxy](https://www.envoyproxy.io/) software stack. It can handle only TCP traffic. This load balancer is configured as a regional load balancing service that can use either Premium or Standard Tier.
### Identify the mode
To determine the mode of a load balancer, complete the following steps.
### Console
1. In the Google Cloud console, go to the **Load balancing** page.
[Go to Load balancing](https://console.cloud.google.com/networking/loadbalancing)
2. On the **Load Balancers** tab, the load balancer type, protocol, and
region are displayed. If the region is blank, then the load balancer
is global.
The following table summarizes how to identify the mode of the load balancer:
| Load balancer mode | Load balancer type | Access type | Region |
| Classic proxy Network Load Balancer | Network (Proxy classic) | External | None |
| Global external proxy Network Load Balancer | Network (Proxy) | External | None |
| Regional external proxy Network Load Balancer | Network (Proxy) | External | Specifies a region |
|---|---|---|---|
[Load balancer modes in the Google Cloud console]
### gcloud
1. Use the `gcloud compute forwarding-rules describe` command:
```
gcloud compute forwarding-rules describe FORWARDING_RULE_NAME
```
2. In the command output, check the load balancing scheme, region, and network
tier. The following table summarizes how to identify the mode of the load
balancer:
| Load balancer mode | Load balancing scheme | Forwarding rule | Network tier |
| Classic proxy Network Load Balancer | EXTERNAL | Global | Standard or Premium |
| Global external proxy Network Load Balancer | EXTERNAL_MANAGED | Global | Premium |
| Regional external proxy Network Load Balancer | EXTERNAL_MANAGED | Regional | Standard or Premium |
|---|---|---|---|
[Load balancer modes in the gcloud CLI output]
> [!IMPORTANT]
> **Important:** After you create a load balancer, you can't edit its mode. Instead, you must delete the load balancer and create a new one.
## Architecture
The following diagrams show the components of external proxy Network Load Balancers.
### Global
This diagram shows the components of a global external proxy Network Load Balancer deployment.
This architecture applies to both the global external proxy Network Load Balancer and
classic proxy Network Load Balancer in Premium Tier.
[](https://docs.cloud.google.com/static/load-balancing/images/ext-tcp-proxy.svg) Global external proxy Network Load Balancer components (click to enlarge).
### Regional
This diagram shows the components of a regional external proxy Network Load Balancer
deployment.
[](https://docs.cloud.google.com/static/load-balancing/images/ext-reg-tcp-proxy.svg) Regional external proxy Network Load Balancer components (click to enlarge).
The following are components of external proxy Network Load Balancers.
### Proxy-only subnet
> [!NOTE]
> **Note:** Proxy-only subnets are only required for regional external proxy Network Load Balancers.
The proxy-only subnet provides a set of IP addresses
that Google uses to run Envoy proxies on your behalf. You must create one
proxy-only subnet in each region of a VPC network where you use
load balancers. The `--purpose` flag for this proxy-only subnet is
set to `REGIONAL_MANAGED_PROXY`. All [regional Envoy-based load
balancers](https://docs.cloud.google.com/load-balancing/docs/proxy-only-subnets#envoy-lb) in the same
region and VPC network share a pool of Envoy proxies from the
same proxy-only subnet.
Backend VMs or endpoints of all load balancers in a region and a
VPC network receive connections from the proxy-only subnet.
Points to remember:
- Proxy-only subnets are *only* used for Envoy proxies, not your backends.
- The IP address of the load balancer is *not* located in the proxy-only subnet. The load balancer's IP address is defined by its external managed forwarding rule.
- Proxy-only subnets must be configured with a stack type of `IPV4_IPV6` to terminate incoming IPv6 traffic and `IPV4_ONLY` or `IPV4_IPV6` to terminate incoming IPv4 traffic.
### Forwarding rules and IP addresses
[Forwarding rules](https://docs.cloud.google.com/load-balancing/docs/forwarding-rule-concepts.md.txt) route traffic
by IP address, port, and protocol to a load balancing configuration that
consists of a target proxy and a backend service.
**IP address specification**. Each forwarding rule references a single IP
address that you can use in DNS records for your application. You can either
reserve a static IP address that you can use or let Cloud Load Balancing
assign one for you. We recommend that you reserve a static IP address.
Otherwise, you must update your DNS record with the newly assigned ephemeral IP
address whenever you delete a forwarding rule and create a new one.
**Port specification** . External forwarding rules used in the definition of this
load balancer can reference exactly *one* port from 1-65535. If you want to
support multiple consecutive ports, you need to configure multiple forwarding
rules. Multiple forwarding rules can be configured with the same virtual IP
address and different ports; therefore, you can proxy multiple applications with
separate custom ports to the same TCP proxy virtual IP address. For more
details, see [Port specifications for forwarding
rules](https://docs.cloud.google.com/load-balancing/docs/forwarding-rule-concepts#port_specifications).
To support multiple consecutive ports, you have to configure multiple forwarding
rules. Multiple forwarding rules can be configured with the same virtual IP
address and different ports. Therefore, you can proxy multiple applications
with separate custom ports to the same TCP proxy virtual IP address.
The following table shows the forwarding rule requirements for
external proxy Network Load Balancers:
| Load balancer mode | Network service tier | Forwarding rule, IP address, and load balancing scheme | Routing from the internet to the load balancer frontend |
| Classic proxy Network Load Balancer | Premium Tier | [Global external forwarding rule](https://docs.cloud.google.com/compute/docs/reference/rest/v1/globalForwardingRules/insert.md.txt) [Global external IP address](https://docs.cloud.google.com/compute/docs/reference/rest/v1/globalAddresses/insert.md.txt) Load balancing scheme: `EXTERNAL` | Requests routed to the GFEs that are closest to the client on the internet. |
| Classic proxy Network Load Balancer | Standard Tier | [Regional external forwarding rule](https://docs.cloud.google.com/compute/docs/reference/rest/v1/forwardingRules/insert.md.txt) [Regional external IP address](https://docs.cloud.google.com/compute/docs/reference/rest/v1/addresses/insert.md.txt) Load balancing scheme: `EXTERNAL` | Requests routed to a GFE in the load balancer's region. |
| Global external proxy Network Load Balancer | Premium Tier | [Global external forwarding rule](https://docs.cloud.google.com/compute/docs/reference/rest/v1/globalForwardingRules/insert.md.txt) [Global external IP address](https://docs.cloud.google.com/compute/docs/reference/rest/v1/globalAddresses/insert.md.txt) Load balancing scheme: `EXTERNAL_MANAGED` | Requests routed to the GFEs that are closest to the client on the internet. |
| Regional external proxy Network Load Balancer | Premium and Standard Tier | [Regional external forwarding rule](https://docs.cloud.google.com/compute/docs/reference/rest/v1/forwardingRules/insert.md.txt) [Regional external IP address](https://docs.cloud.google.com/compute/docs/reference/rest/v1/addresses/insert.md.txt) Load balancing scheme: `EXTERNAL_MANAGED` | Requests routed to the Envoy proxies in the same region as the load balancer. |
|---|---|---|---|
[Forwarding rule requirements by load balancer mode and network tier]
#### Forwarding rules and VPC networks
This section describes how forwarding rules used by external Application Load Balancers are
associated with VPC networks:
| Load balancer mode | VPC network association |
| Global external proxy Network Load Balancer Classic proxy Network Load Balancer | No associated VPC network. The forwarding rule always uses an [IP address](https://docs.cloud.google.com/load-balancing/docs/tcp.md.txt#fw-rule-ip-address-table) that is outside the VPC network. Therefore, the forwarding rule has no associated VPC network. |
| Regional external proxy Network Load Balancer | The forwarding rule's VPC network is the network where the proxy-only subnet has been created. You specify the network when you create the forwarding rule. Depending on whether you use an IPv4 address or an IPv6 address range, there is always an explicit or implicit VPC network associated with the forwarding rule. - Regional external IPv4 addresses always exist outside of VPC networks. However, when you create the forwarding rule, you're required to specify the VPC network where the proxy-only subnet has been created. Hence, the forwarding rule has an explicit network association. - Regional external IPv6 address ranges always exist inside a VPC network. When you create the forwarding rule, you're required to specify the subnet from which the IP address range is taken. This subnet must be in the same region and VPC network where a proxy-only subnet has been created. Thus, there is an implied network association. **Network and subnet requirements.** For IPv6 traffic, your network and subnets must meet the following configuration requirements: - VPC network: you must use a custom mode VPC network configured with the `--enable-ula-internal-ipv6` flag. - Forwarding rule subnet: this subnet must be a dual-stack (`IPv4_IPv6`) or `IPV6_ONLY` subnet with the `ipv6-access-type` set to `EXTERNAL`. **IPv6 address allocation options.** The forwarding rule must reference a `/96` range of IPv6 addresses from the subnet's `/64` external IPv6 address range. The load balancer accepts traffic sent to any address within the allocated `/96` range. You can assign this `/96` IPv6 address range using one of the following methods: - Specifying a [reserved external IPv6 address](https://docs.cloud.google.com/vpc/docs/reserve-static-external-ip-address.md.txt#reserve_new_static), either automatically allocated or a custom `/96` range. - Specifying a custom ephemeral IPv6 address. - Letting Google Cloud automatically assign an ephemeral IPv6 address. **Limitations.** To specify a custom ephemeral IPv6 address, you must use the Google Cloud CLI or the API. The Google Cloud console doesn't support specifying custom ephemeral IPv6 addresses for forwarding rules.
|
|---|---|
[VPC network association by load balancer mode]
### Target proxies
External proxy Network Load Balancers terminate connections from the client and create
new connections to the backends. The [target
proxy](https://docs.cloud.google.com/load-balancing/docs/target-proxies) routes these new connections to
the backend service.
Depending on the type of traffic your application needs to handle, you can
configure an external proxy Network Load Balancer with either a target TCP proxy or a target SSL
proxy.
- **Target TCP proxy**: Configure the load balancer with a target TCP proxy if you're expecting TCP traffic.
- **Target SSL proxy** : Configure the load balancer with a target SSL proxy if you're expecting *encrypted* client traffic. This type of load balancer is intended for non-HTTP(S) traffic only. For HTTP(S) traffic, we recommend that you use an [external Application Load Balancer](https://docs.cloud.google.com/load-balancing/docs/https.md.txt).
By default, the target proxy does not preserve the original client IP address
and port information. You can preserve this information by enabling the [PROXY
protocol](https://docs.cloud.google.com/load-balancing/docs/tcp/setting-up-tcp#proxy-protocol) on the target
proxy.
The following table shows the target proxy requirements for
external proxy Network Load Balancers:
| Load balancer mode | Network service tier | Target proxy | Reference |
| Classic proxy Network Load Balancer | Premium Tier | `targetTcpProxies` or `targetSslProxies` | Target proxy references a single backend service. |
| Classic proxy Network Load Balancer | Standard Tier | `targetTcpProxies` or `targetSslProxies` | Target proxy references a single backend service. |
| Global external proxy Network Load Balancer | Premium Tier | `targetTcpProxies` or `targetSslProxies` | Target proxy references a single backend service. |
| Regional external proxy Network Load Balancer | Premium and Standard Tier | `regionTargetTcpProxies` | Target proxy references either a single backend service or one or more TLS routes. |
|---|---|---|---|
[Target proxy requirements by load balancer mode and network tier]
### SSL certificates
*SSL certificates are only required if you're deploying a global external proxy Network Load Balancer
and classic proxy Network Load Balancer with a target SSL proxy.*
External proxy Network Load Balancers using target SSL proxies require private keys and
SSL certificates as part of the load balancer configuration.
- Google Cloud provides two *configuration methods* for assigning private
keys and SSL certificates to target SSL proxies: Compute Engine SSL
certificates and Certificate Manager. For a description of each
configuration, see [Certificate configuration
methods](https://docs.cloud.google.com/load-balancing/docs/ssl-certificates#config-tech) in the SSL
certificates overview.
- Google Cloud provides two *certificate types* : Self-managed and
Google-managed. For a description of each type, see [Certificate
types](https://docs.cloud.google.com/load-balancing/docs/ssl-certificates#certificate-types) in the SSL
certificates overview.
### TLS routes
A [TLS
route](https://docs.cloud.google.com/service-mesh/docs/reference/network-services/rest/v1/projects.locations.tlsRoutes)
resource lets you define how traffic is [routed to backend services based on SNI
hostnames](https://docs.cloud.google.com/load-balancing/docs/tcp#sni-based-routing).
If the load balancer's target TCP proxy is not referenced by a TLS route, then
only the single default backend service is used, and the value of SNI hostname
sent by the client or its absence isn't relevant.
TLS routes are only available for regional external proxy Network Load Balancers. They aren't supported for classic proxy Network Load Balancers and global external proxy Network Load Balancers.
You can attach a TLS route configuration to a load balancer's target proxy by
using the [gcloud network-services
tls-routes](https://cloud.google.com/sdk/gcloud/reference/network-services/tls-routes)
commands.
The following table shows the TLS routes APIs required by external proxy Network Load Balancers:
| Load balancer mode | TLS route | Reference |
| Regional external proxy Network Load Balancer | [Regional `tlsRoutes`](https://docs.cloud.google.com/service-mesh/docs/reference/network-services/rest/v1/projects.locations.tlsRoutes.md.txt) | Each TLS route can reference one or more backend services. |
|---|---|---|
[TLS route APIs by load balancer mode]
### Backend services
[Backend services](https://docs.cloud.google.com/load-balancing/docs/backend-service.md.txt) direct incoming traffic
to one or more attached backends. Each backend is composed of an
[instance group](https://docs.cloud.google.com/compute/docs/instance-groups.md.txt) or
[network endpoint group](https://docs.cloud.google.com/load-balancing/docs/negs.md.txt) and
information about the backend's serving capacity. Backend serving capacity can be
based on CPU or requests per second (RPS).
Each load balancer has at least one backend service resource. If you use TLS
routes to configure routing, you
can configure multiple backend services for your load balancer. Without TLS
routes, you're limited to only a single backend service per load balancer.
The backend service specifies the [health
check](https://docs.cloud.google.com/load-balancing/docs/health-checks) to be performed for the available
backends.
Changes made to the backend service are not instantaneous. It can
take several minutes for changes to propagate to GFEs.
To ensure minimal interruptions to your users, you can enable connection
draining on backend services. Such interruptions might happen when a backend
is terminated, removed manually, or removed by an autoscaler. To learn more about
using connection draining to minimize service interruptions, see
[Enabling connection draining](https://docs.cloud.google.com/load-balancing/docs/enabling-connection-draining.md.txt).
For more information about the backend service resource, see
[Backend services overview](https://docs.cloud.google.com/load-balancing/docs/backend-service.md.txt).
The following table specifies the different backends supported on the backend service of external proxy Network Load Balancers:
| Load balancer mode | Backend type --- Instance groups | Backend type --- Zonal NEGs | Backend type --- Internet NEGs | Backend type --- Serverless NEGs | Backend type --- Hybrid NEGs | Backend type --- Private Service Connect NEGs | Backend type --- GKE |
| Classic proxy Network Load Balancer | Yes | [Use standalone zonal NEGs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg.md.txt) | No | No | Yes | No | Yes |
| Global external proxy Network Load Balancer | Yes | Yes`GCE_VM_IP_PORT` type endpoints | No | No | Yes | Yes | Yes |
| Regional external proxy Network Load Balancer | Yes | Yes`GCE_VM_IP_PORT` type endpoints | YesRegional NEGs only | No | Yes | [Add a Private Service Connect NEG](https://docs.cloud.google.com/vpc/docs/access-apis-managed-services-private-service-connect-backends.md.txt#add-neg-ext-tcp-proxy-lb) | No |
|---|---|---|---|---|---|---|---|
[Supported backends on a backend service by load balancer mode]
**Notes:**
- **For Global external proxy Network Load Balancer:** Global external proxy Network Load Balancers support IPv4 and IPv6 (dual stack) instance groups and zonal NEG backends with `GCE_VM_IP_PORT` endpoints.
#### Backends and VPC networks
**For global external proxy Network Load Balancer and classic proxy Network Load Balancer backends**,
all backend instances from instance group backends and all backend endpoints
from NEG backends must be located in the same project. However, an instance
group backend or a NEG can use a different VPC network in
that project. The different VPC networks don't need to be
connected using VPC Network Peering because GFEs communicate directly
with backends in their respective VPC networks.
**For regional external proxy Network Load Balancer backends**, the following applies:
- For instance groups, zonal NEGs, and hybrid connectivity NEGs, all backends
must be located in the same project and region as the backend service.
However, a load balancer can reference a backend that uses a different
VPC network in the same project as the backend service.
Connectivity between the load balancer's
VPC network and the backend VPC network
can be configured using either VPC Network Peering, Cloud VPN
tunnels, Cloud Interconnect VLAN attachments, or a Network Connectivity Center
framework.
**Backend network definition**
- For zonal NEGs and hybrid NEGs, you explicitly specify the VPC network when you create the NEG.
- For managed instance groups, the VPC network is defined in the instance template.
- For unmanaged instance groups, the instance group's VPC network is set to match the VPC network of the `nic0` interface for the first VM added to the instance group.
**Backend network requirements**
Your backend's network must satisfy *one of the following* network
requirements:
- The backend's VPC network must exactly match the
forwarding rule's VPC network.
- The backend's VPC network must be connected to the
forwarding rule's VPC network using
VPC Network Peering. You must configure subnet route exchanges to
allow communication between the proxy-only subnet in the forwarding rule's
VPC network and the subnets used by the backend instances
or endpoints.
**Backend IPv6 subnet requirements**
The IP version used for the frontend connection is independent of the backend
connection. If the proxy-only subnet is configured as dual-stack (IPV4_IPV6),
the proxy-only subnet can communicate with backends using either IPv4 or
IPv6. If the proxy-only subnet is single-stack (IPV4_ONLY), it can only
communicate with backends using IPv4.
If your backend instances are handling IPv6 traffic, the backend subnet can
be configured with a stack type of `IPV4_ONLY` or
`IPV4_IPV6` (dual-stack). If the backend subnet's stack type
includes IPv6, you must explicitly set the subnet's ipv6-access-type to
`EXTERNAL`.
- Both the backend's VPC network and the forwarding rule's VPC network must be [VPC
spokes](https://docs.cloud.google.com/network-connectivity/docs/network-connectivity-center/concepts/vpc-spokes-overview) attached to the same [NCC
hub](https://docs.cloud.google.com/network-connectivity/docs/network-connectivity-center/concepts/overview). Import and export filters must allow communication between the proxy-only subnet in the forwarding rule's VPC network and the subnets used by backend instances or endpoints.
- For all other backend types, all backends must be located in the same
VPC network and region.
#### Backends and network interfaces
If you use instance group backends, packets are always delivered to `nic0`. If
you want to send packets to non-`nic0` interfaces (either [vNICs or
Dynamic Network Interfaces](https://docs.cloud.google.com/vpc/docs/multiple-interfaces-concepts#nic-types)), use
NEG backends instead.
If you use zonal NEG backends, packets are sent to whatever network interface is
represented by the endpoint in the NEG. The NEG endpoints must be in the same
VPC network as the NEG's explicitly defined VPC
network.
#### Protocol for communicating with the backends
When you configure a backend service for an external proxy Network Load Balancer, you set the
protocol that the backend service uses to communicate with the backends.
- For classic proxy Network Load Balancers, you can choose either TCP or SSL.
- For global external proxy Network Load Balancers, you can choose either TCP or SSL.
- For regional external proxy Network Load Balancers, you can use TCP.
The load balancer uses only the protocol that you specify, and does not attempt
to negotiate a connection with the other protocol.
### Firewall rules
The following firewall rules are required:
- For classic proxy Network Load Balancers, an ingress `allow` firewall rule to permit
traffic from GFEs to reach your backends.
- For global external proxy Network Load Balancers, an ingress `allow` firewall rule to permit
traffic from GFEs to reach your backends.
- For regional external proxy Network Load Balancers, an ingress firewall rule to permit traffic
from the [proxy-only subnet](https://docs.cloud.google.com/load-balancing/docs/tcp.md.txt#proxy-only-subnet) to reach your backends.
- An ingress `allow` firewall rule to permit traffic from the health check probe
ranges to reach your backends. For more information about health check probes
and why it's necessary to allow traffic from them, see
[Probe IP ranges and firewall rules](https://docs.cloud.google.com/load-balancing/docs/health-check-concepts.md.txt#ip-ranges).
Firewall rules are implemented at the VM instance level, not at the GFE proxies
level. You cannot use firewall rules to prevent traffic from
reaching the load balancer.
The ports for these firewall rules must be configured as follows:
- Allow traffic to the destination port for each backend service's health check.
- For instance group backends: determine the ports to be configured by the mapping between the backend service's [named port](https://docs.cloud.google.com/load-balancing/docs/backend-service.md.txt#named_ports) and the port numbers associated with that named port on each instance group. Port numbers can vary between instance groups assigned to the same backend service.
- For `GCE_VM_IP_PORT NEG` zonal NEG backends: allow traffic to the port numbers of the endpoints.
The following table summarizes the required source IP address ranges for the
firewall rules:
| Load balancer mode | Health check source ranges | Request source ranges |
| Global external proxy Network Load Balancer | - `35.191.0.0/16` - `130.211.0.0/22` For IPv6 traffic to the backends: - `2600:2d00:1:b029::/64` | The source of GFE traffic depends on the backend type: - Instance groups and zonal NEGs (`GCE_VM_IP_PORT`): - `130.211.0.0/22` - `35.191.0.0/16` For IPv6 traffic to the backends: - `2600:2d00:1:1::/64` |
| Classic proxy Network Load Balancer | - `35.191.0.0/16` - `130.211.0.0/22` | These ranges apply to health check probes and requests from the GFE. |
| Regional external proxy Network Load Balancer | - `35.191.0.0/16` - `130.211.0.0/22` For IPv6 traffic to the backends: - `2600:2d00:1:b029::/64` | These ranges apply to health check probes. |
|---|---|---|
[Required firewall rule source IP address ranges by load balancer mode]
**Notes:**
- **For Regional external proxy Network Load Balancer (hybrid NEGs):** Allowing traffic from Google's health check probe ranges isn't required for hybrid NEGs. However, if you're using a combination of hybrid and zonal NEGs in a single backend service, you need to allow traffic from the [Google
health check probe ranges](https://docs.cloud.google.com/load-balancing/docs/health-check-concepts#ip-ranges) for the zonal NEGs.
- **For Regional external proxy Network Load Balancer (internet NEGs):** For regional internet NEGs, health checks are optional. Traffic from load balancers using *regional* internet NEGs originates from the [proxy-only subnet](https://docs.cloud.google.com/load-balancing/docs/proxy-only-subnets.md.txt) and is then NAT-translated (by using Cloud NAT) to either manually or automatically allocated NAT IP addresses. This traffic includes both health check probes and user requests from the load balancer to the backends. For details, see [Regional NEGs:
Use a Cloud NAT gateway](https://docs.cloud.google.com/load-balancing/docs/negs/internet-neg-concepts#nat-support).
> [!IMPORTANT]
> **Important:** Make sure that you allow packets from the full health check ranges. If your firewall rule allows packets from only a subset of the ranges, you might see health check failures because the load balancer can't communicate with your backends. This causes connection timeouts.
### Source IP addresses
The source IP address for packets, as seen by the backends, is *not* the
Google Cloud external IP address of the load balancer. In other words,
there are two TCP connections.
**For the classic proxy Network Load Balancers and global external proxy Network Load Balancers** :
- Connection 1, from original client to the load balancer (GFE):
- **Source IP address**: the original client (or external IP address if the client is behind a NAT gateway or a forward proxy).
- **Destination IP address**: your load balancer's IP address.
- Connection 2, from the load balancer (GFE) to the backend VM or endpoint:
- **Source IP address** : an IP address in one of the ranges specified in
[Firewall rules](https://docs.cloud.google.com/load-balancing/docs/tcp.md.txt#firewall-rules).
- **Destination IP address**: the internal IP address of the backend VM or
container in the VPC network.
**For the regional external proxy Network Load Balancers** :
- Connection 1, from original client to the load balancer (proxy-only subnet):
- **Source IP address**: the original client (or external IP address if the client is behind a NAT gateway or a forward proxy).
- **Destination IP address**: your load balancer's IP address.
- Connection 2, from the load balancer (proxy-only subnet) to the backend VM or
endpoint:
- **Source IP address** : an IP address in the [proxy-only
subnet](https://docs.cloud.google.com/load-balancing/docs/tcp#proxy-only-subnet) that is shared among all the Envoy-based load
balancers deployed in the same region and network as the load balancer.
- **Destination IP address**: the internal IP address of the backend VM or
container in the VPC network.
### Open ports
External proxy Network Load Balancers are reverse proxy load balancers. The load
balancer terminates incoming connections, and then opens new connections from
the load balancer to the backends. These load balancers are implemented by using
Google Front End (GFE) proxies worldwide.
GFEs have several open ports to support other Google services that run on the
same architecture. When you run a port scan, you might see other open ports for
other Google services running on GFEs.
Running a port scan on the IP address of a GFE-based load balancer isn't useful
from an auditing perspective for the following reasons:
- A port scan (for example, with `nmap`) generally expects no response packet or
a TCP RST packet when performing TCP SYN probing. GFEs send SYN-ACK
packets in response to SYN probes only for ports on which you have configured
a forwarding rule. GFEs only send packets to your backends in response to
packets sent to your load balancer's IP address *and* the destination port
configured on its forwarding rule. Packets that are sent to a different IP
address or port aren't sent to your backends.
- Packets sent to the IP address of your load balancer can be answered by any
GFE in Google's fleet; however, scanning a load balancer IP address and
destination port combination only interrogates a single GFE per
TCP connection. The IP address of your load balancer isn't assigned to a
single device or system. Thus, scanning the IP address of a GFE-based load
balancer doesn't scan all the GFEs in Google's fleet.
With that in mind, the following are some more effective ways to audit the
security of your backend instances:
- A security auditor should inspect the forwarding rules configuration for the
load balancer's configuration. The forwarding rules define the destination
port for which your load balancer accepts packets and forwards
them to the backends. For GFE-based load balancers, [each external forwarding
rule can only reference a single destination TCP
port](https://docs.cloud.google.com/load-balancing/docs/forwarding-rule-concepts#port_specifications).
- A security auditor should inspect the firewall rule configuration applicable
to backend VMs. The firewall rules that you set block traffic from the GFEs
to the backend VMs but don't block incoming traffic to the GFEs. For best
practices, see the [firewall rules section](https://docs.cloud.google.com/load-balancing/docs/tcp.md.txt#firewall-rules).
## Shared VPC architecture
Regional external proxy Network Load Balancers and classic proxy Network Load Balancers support
deployments that use Shared VPC networks.
Shared VPC lets you maintain a clear separation of responsibilities
between network administrators and service developers. Your development teams
can focus on building services in service projects, and the network
infrastructure teams can provision and administer load balancing. If you're not
already familiar with Shared VPC, read the
[Shared VPC overview documentation](https://docs.cloud.google.com/vpc/docs/shared-vpc.md.txt).
The following table summarizes how load balancing components are distributed across projects in a Shared VPC deployment:
| IP address | Forwarding rule | Target proxy | Backend components |
|---|---|---|---|
| An [external IP address](https://docs.cloud.google.com/vpc/docs/reserve-static-external-ip-address.md.txt#reserve_new_static) must be defined in the same project as the load balancer. | The external forwarding rule must be defined in the same project as the backend instances (the service project). | The target TCP or SSL proxy must be defined in the same project as the backend instances. | For classic proxy Network Load Balancers, a global backend service must be defined in the same project as the backend instances. These instances must be in instance groups attached to the backend service as backends. Health checks associated with backend services must be defined in the same project as the backend service. For regional external proxy Network Load Balancers, the backend VMs are typically located in a service project. A regional backend service and health check must be defined in that service project. |
[Component project placement in a Shared VPC deployment]
> [!NOTE]
> **Note:** Global external proxy Network Load Balancers do not support deployments that use Shared VPC networks.
## Traffic distribution
When you add a backend instance group or NEG to a backend service, you specify a
[load balancing
mode](https://docs.cloud.google.com/sdk/gcloud/reference/compute/backend-services/add-backend#--balancing-mode),
which defines a method that measures the backend load and target capacity.
For external proxy Network Load Balancers, the balancing mode can be `CONNECTION`
or `UTILIZATION`:
- If the load balancing mode is `CONNECTION`, the load is spread based on the total number of connections that the backend can handle.
- If the load balancing mode is `UTILIZATION`, the load is spread based on the utilization of instances in an instance group. This balancing mode applies to VM instance group backends only.
The distribution of traffic across backends is determined by the balancing mode
of the load balancer.
### Classic proxy Network Load Balancer
For the classic proxy Network Load Balancer, the balancing mode is used to select the
most favorable backend (instance group or NEG). Traffic is then distributed in a
round robin fashion among instances or endpoints within the backend.
#### How connections are distributed
A classic proxy Network Load Balancer can be configured as a global load balancing
service with [Premium Tier](https://docs.cloud.google.com/network-tiers/docs/overview.md.txt#premium_tier), and as a
regional service in the [Standard
Tier](https://docs.cloud.google.com/network-tiers/docs/overview#standard_tier).
The balancing mode and choice of target determine backend *fullness* from the
perspective of each zonal `GCE_VM_IP_PORT` NEG, zonal instance group, or zone of
a regional instance group. Traffic is then distributed within a zone by using
consistent hashing.
**For Premium Tier**
- You can have only one backend service, and the backend service can
have backends in multiple regions. For global load balancing, you deploy your
backends in multiple regions, and the load balancer automatically directs
traffic to the region closest to the user. If a region is at capacity, the
load balancer automatically directs new connections to another region with
available capacity. Existing user connections remain in the current region.
- Google advertises your load balancer's IP address from all points of
presence, worldwide. Each load balancer IP address is global anycast.
- If you configure a backend service with backends in multiple regions, Google
Front Ends (GFEs) attempt to direct requests to healthy backend instance
groups or NEGs in the region closest to the user.
**For Standard Tier**
- Google advertises your load balancer's IP address from points of presence
associated with the forwarding rule's region. The load balancer uses a
regional external IP address.
- You can only configure backends in the same region as the forwarding rule.
The load balancer only directs requests to healthy backends in that one
region.
### Global external proxy Network Load Balancer
For the global external proxy Network Load Balancer, traffic distribution is based on the load
balancing mode and the load balancing locality policy.
The balancing mode determines the weight and fraction of traffic to be
sent to each group (instance group or NEG). The load balancing locality policy
(`LocalityLbPolicy`) determines how backends within the group are load balanced.
When a backend service receives traffic, it first directs traffic to a backend
(instance group or NEG) according to the backend's balancing mode. After a
backend is selected, traffic is then distributed among instances or endpoints in
that backend group according to the load balancing locality policy.
For more information, see the following:
- [Balancing mode](https://docs.cloud.google.com/load-balancing/docs/backend-service.md.txt#balancing-mode).
- [Load balancing locality policy (global backend service API
documentation)](https://docs.cloud.google.com/compute/docs/reference/rest/beta/backendServices).
#### How connections are distributed
A global external proxy Network Load Balancer can be configured as a global load balancing
service with [Premium Tier](https://docs.cloud.google.com/network-tiers/docs/overview.md.txt#premium_tier)
The balancing mode and choice of target determine backend *fullness* from the
perspective of each zonal `GCE_VM_IP_PORT` NEG, or zonal instance group.
Traffic is then distributed within a zone by using consistent hashing.
- You can have only one backend service, and the backend service can
have backends in multiple regions. For global load balancing, you deploy your
backends in multiple regions, and the load balancer automatically directs
traffic to the region closest to the user. If a region is at capacity, the
load balancer automatically directs new connections to another region with
available capacity. Existing user connections remain in the current region.
- Google advertises your load balancer's IP address from all points of
presence, worldwide. Each load balancer IP address is global anycast.
- If you configure a backend service with backends in multiple regions, Google
Front Ends (GFEs) attempt to direct requests to healthy backend instance
groups or NEGs in the region closest to the user.
### Regional external proxy Network Load Balancer
For regional external proxy Network Load Balancers, traffic distribution is based on the load
balancing mode and the load balancing locality policy.
The balancing mode determines the weight and fraction of traffic that should be
sent to each backend (instance group or NEG). The load balancing locality policy
(`LocalityLbPolicy`) determines how backends within the group are load balanced.
When a backend service receives traffic, it first directs traffic to a backend
(instance group or NEG) according to its balancing mode. After a
backend is selected, traffic is then distributed among instances or endpoints in
that backend group according to the load balancing locality policy.
For more information, see the following:
- [Balancing modes](https://docs.cloud.google.com/load-balancing/docs/backend-service.md.txt#balancing-mode)
- [Load balancing locality policy (regional backend service API
documentation)](https://docs.cloud.google.com/compute/docs/reference/rest/beta/regionBackendServices)
### SNI-based routing with TLS routes
> [!NOTE]
> **Note:** SNI-based routing is supported only for regional external proxy Network Load Balancers.
SNI-based routing lets your TCP proxy Network Load Balancers route traffic to specific
backend services based on the Server Name Indication (SNI) hostname provided
during the TLS handshake. SNI is a TLS extension that allows a client to specify
the domain name it wants to reach before the encrypted tunnel is established.
The SNI hostname is transmitted in an unencrypted format within the `ClientHello`
message so that the load balancer can inspect this value and route traffic to
the appropriate backend service.
You can use a TLS route configuration resource to map specific SNI hostnames to
backend services. When SNI-based routing is configured, the load balancer
enables *TLS passthrough*, where the encrypted byte stream is passed directly to
the backend without being decrypted by the load balancer.
This feature unlocks some key use-cases such as:
- Use a single global anycast IP address and port to serve multiple applications, reducing IPv4 address consumption.
- Supports end-to-end encrypted deployments where certificates can remain on the backends, ensuring that traffic is never unencrypted in transit.
- Allows platform administrators to manage the core infrastructure while service owners independently manage their specific routes and backends.
- Enables routing to specific shards for stateful non-HTTP applications (such as MongoDB) based on the SNI provided by the client.
#### How traffic is distributed when TLS routes are used
When SNI-based routing is configured, traffic is routed to backend services as
follows:
1. The load balancer listens on the IP and port defined in the forwarding rule.
2. When a client initiates a TLS connection, the load balancer intercepts the `ClientHello` message and extracts the SNI hostname.
3. The extracted SNI is compared against the hostnames defined in the configured
TLS routes. The load balancer uses the following logic to determine
which backend service should receive the traffic:
1. If the client does not use the TLS protocol, the load balancer closes the connection.
2. If the client does not send any SNI hostname name in its `ClientHello` message, the load balancer closes the connection.
3. If the client sends an SNI hostname that is not a valid DNS name, the load balancer closes the connection.
4. If the client sends an SNI hostname that doesn't match any SNI hostnames associated with a TLS route, the load balancer closes the connection.
5. In all other situations: The load balancer selects a backend service
using the following matching process:
Matching is done by the longest suffix (longest in terms of the number
of subdomains). To illustrate the matching method, consider a TLS route
whose SNI is `*.foo.com`, another TLS route whose SNI is `*.bar.foo.com`,
and the third TLS route whose SNI is `baz.bar.foo.com`.
- If the client's SNI hostname is `baz.bar.foo.com`, the load balancer selects a backend service which is referenced by the TLS route whose SNI is `baz.bar.foo.com`.
- If the client's SNI hostname is `qux.bar.foo.com`, the load balancer selects a backend service which is referenced by the TLS route whose SNI is `*.bar.foo.com`.
- If the client's SNI hostname is `qux.qux.foo.com`, the load balancer selects a backend service which is referenced by the TLS route whose SNI is `*.foo.com`.
Once a match is found, the encrypted byte stream is proxied directly to the
assigned backend service without terminating the TLS connection (for TLS
passthrough).
#### Limitations
The following limitations apply when you use TLS routes to configure SNI-based
routing:
- A target TCP proxy can either reference a single backend service or one or more TLS routes. You can't configure both.
- Proxy Network Load Balancers can't detect HTTP connection coalescing, which might
lead to misrouted HTTP requests.
By default, browsers already coalesce HTTP connections. For example, when a
browser opens a connection to `www.example.com` and the server presents a
certificate for `*.example.com` during TLS handshake, the browser reuses this
connection for all requests to `*.example.com`, as long as the hostname
resolves to the same IP address. This is in accordance with [RFC
7540](https://www.rfc-editor.org/rfc/rfc7540).
With SNI-based routing, once a connection is established, it is then
permanently bound to its destination backend. Consider a deployment where
`*.example.com` is served by one HTTPS backend with a `*.example.com` TLS
certificate, and `my.example.com` is served by another HTTPS backend. If a
connection to `*.example.com` is established first, then subsequent HTTP
requests to `my.example.com` are coalesced onto the same connection, using the
`*.example.com` HTTPS backend.
### Session affinity
Session affinity lets you configure the load balancer's backend service to send
all requests from the same client to the same backend, as long as the backend is
healthy and has capacity.
External proxy Network Load Balancers offer the following types of session affinity:
- **None**
A session affinity setting of `NONE` does *not* mean that there is no
session affinity. It means that no session affinity option is explicitly configured.
Hashing is always performed to select a backend. And a session affinity setting of
`NONE` means that the load balancer uses a 5-tuple hash to select a backend. The 5-tuple
hash consists of the source IP address, the source port, the protocol, the destination IP address,
and the destination port.
A session affinity of `NONE` is the default value.
- **Client IP affinity**
*Client IP* session affinity (`CLIENT_IP`) is a 2-tuple hash created from the
source and destination IP addresses of the packet. Client IP affinity forwards
all requests from the same client IP address to the same backend, as long as
that backend has capacity and remains healthy.
When you use client IP affinity, keep the following in mind:
- The packet destination IP address is only the same as the load balancer forwarding rule's IP address if the packet is sent directly to the load balancer.
- The packet source IP address might not match an IP address associated with the original client if the packet is processed by an intermediate NAT or proxy system before being delivered to a Google Cloud load balancer. In situations where many clients share the same effective source IP address, some backend VMs might receive more connections or requests than others.
Keep the following in mind when configuring session affinity:
- Don't rely on session affinity for authentication or security purposes.
Session affinity can break whenever the
number of serving and healthy backends changes. For more details, see [Losing
session affinity](https://docs.cloud.google.com/load-balancing/docs/tcp#losing_session_affinity).
- The default values of the `--session-affinity` and `--subsetting-policy`
flags are both `NONE`, and only one of them at a time can be set to a
different value.
**Losing session affinity**
All session affinity options require the following:
- The selected backend instance or endpoint must remain configured as a backend. Session affinity can break when one of the following events occurs:
- You remove the selected instance from its instance group.
- Managed instance group autoscaling or autohealing removes the selected instance from its managed instance group.
- You remove the selected endpoint from its NEG.
- You remove the instance group or NEG that contains the selected instance or endpoint from the backend service.
- The selected backend instance or endpoint must remain healthy. Session affinity can break when the selected instance or endpoint fails health checks.
- For Global external proxy Network Load Balancers and Classic proxy Network Load Balancers, session affinity can break if a different first-layer Google Front End (GFE) is used for subsequent requests or connections after the change in routing path. A different first-layer GFE might be selected if the routing path from a client on the internet to Google changes between requests or connections.
All session affinity options have the following additional requirements:
- The instance group or NEG that contains the selected instance or endpoint
must not be full as defined by its [target capacity](https://docs.cloud.google.com/load-balancing/docs/backend-service.md.txt#target-capacity). (For
regional managed instance groups, the zonal component of the instance group
that contains the selected instance must not be full.) Session affinity can
break when the instance group or NEG is full and other instance groups or
NEGs are not. Because fullness can change in unpredictable ways when using
the `UTILIZATION` balancing mode, you should use the `RATE` or `CONNECTION`
balancing mode to minimize situations when session affinity can break.
- The *total number* of configured backend instances or endpoints must remain
constant. When at least one of the following events occurs, the number of
configured backend instances or endpoints changes, and session affinity can
break:
- Adding new instances or endpoints:
- You add instances to an existing instance group on the backend service.
- Managed instance group autoscaling adds instances to a managed instance group on the backend service.
- You add endpoints to an existing NEG on the backend service.
- You add non-empty instance groups or NEGs to the backend service.
- Removing any instance or endpoint, *not just the selected instance or
endpoint*:
- You remove any instance from an instance group backend.
- Managed instance group autoscaling or autohealing removes any instance from a managed instance group backend.
- You remove any endpoint from a NEG backend.
- You remove any existing, non-empty backend instance group or NEG from the backend service.
- The *total number* of healthy backend instances or endpoints must remain
constant. When at least one of the following events occurs, the number of
healthy backend instances or endpoints changes, and session affinity can
break:
- Any instance or endpoint passes its health check, transitioning from unhealthy to healthy.
- Any instance or endpoint fails its health check, transitioning from healthy to unhealthy or timeout.
### Failover
Failover for external proxy Network Load Balancers works as follows:
- If a backend becomes unhealthy, traffic is automatically redirected to healthy backends within the same region.
- If all backends within a region are unhealthy, traffic is distributed to healthy backends in other regions (global and classic modes only).
- If all backends are unhealthy, the load balancer drops traffic.
## Load balancing for GKE applications
If you are building applications in Google Kubernetes Engine, you can use standalone
NEGs to load balance traffic directly to containers. With standalone NEGs, you
are responsible for creating the
[Service](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/service.md.txt) object that creates the NEG,
and then associating the NEG with the backend service so that the load balancer
can connect to the Pods.
For related documentation, see [Container-native load balancing through standalone zonal
NEGs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg).
## Limitations
- You can't create a regional external proxy Network Load Balancer in Premium Tier using the
Google Cloud console. Additionally, only [regions
supporting Standard Tier](https://docs.cloud.google.com/network-tiers/docs/overview#regions_supporting_standard_tier) are available for these load balancers in the
Google Cloud console. Use either the gcloud CLI
or the API instead.
- Google Cloud does not make any guarantees on the lifetime of TCP
connections when you use external proxy Network Load Balancers. Clients should be
resilient to dropped connections, both due to broader internet issues
and due to regularly scheduled restarts of the proxies managing the
connections. If a proxy is restarted, the `proxyStatus status` field in
[Proxy Network Load Balancer logs](https://docs.cloud.google.com/load-balancing/docs/tcp/tcp-ssl-proxy-logging-monitoring.md.txt#proxy-status)
displays an associated status message.
- The following limitations apply only to classic proxy Network Load Balancers and
global external proxy Network Load Balancer that are deployed with an SSL target proxy:
- Classic proxy Network Load Balancers and global external proxy Network Load Balancers do not
support client certificate-based authentication, also known as mutual TLS
authentication.
- Classic proxy Network Load Balancers and global external proxy Network Load Balancers support
only lowercase characters in domains in a common name (`CN`) attribute or a
subject alternative name (`SAN`) attribute of the certificate. Certificates
with uppercase characters in domains are returned only when set as the
[primary certificate](https://docs.cloud.google.com/load-balancing/docs/ssl-certificates.md.txt#multiplessl) in
the target proxy.
## What's next
- [Set up a classic proxy Network Load Balancer (TCP proxy)](https://docs.cloud.google.com/load-balancing/docs/tcp/setting-up-tcp.md.txt).
- [Set up a classic proxy Network Load Balancer (SSL proxy)](https://docs.cloud.google.com/load-balancing/docs/ssl/setting-up-ssl.md.txt).
- [Set up a global external proxy Network Load Balancer (TCP proxy)](https://docs.cloud.google.com/load-balancing/docs/tcp/set-up-global-ext-proxy-tcp.md.txt).
- [Set up a global external proxy Network Load Balancer (SSL proxy)](https://docs.cloud.google.com/load-balancing/docs/tcp/set-up-global-ext-proxy-ssl.md.txt).
- [Set up a regional external proxy Network Load Balancer (TCP proxy)](https://docs.cloud.google.com/load-balancing/docs/tcp/set-up-ext-reg-tcp-proxy-migs.md.txt).
- [Proxy Network Load Balancer logging and monitoring](https://docs.cloud.google.com/load-balancing/docs/tcp/tcp-ssl-proxy-logging-monitoring.md.txt).
- [Using SSL policies](https://docs.cloud.google.com/load-balancing/docs/use-ssl-policies.md.txt).
- [GFE locations](https://docs.cloud.google.com/load-balancing/docs/locations.md.txt).