Ingress and DNS configuration

This guide covers how to configure ingress and DNS for KubeVirt-based Hosted Control Plane (HCP) clusters.

How KubeVirt ingress works

On KubeVirt HCP clusters, the guest cluster's default IngressController defaults to the NodePortService endpoint publishing strategy. This means the guest cluster's router pods are exposed through a NodePort Service (router-nodeport-default in the openshift-ingress namespace) that listens on dynamically assigned ports on each guest VM's network interface.

There are two modes for routing external traffic to these NodePorts:

When baseDomainPassthrough is auto-enabled

When creating a KubeVirt HostedCluster without specifying a baseDomain, the HyperShift webhook automatically enables baseDomainPassthrough:

• If spec.dns.baseDomain is empty, the webhook sets spec.platform.kubevirt.baseDomainPassthrough = true
• If you provide an explicit baseDomain, the webhook does not enable baseDomainPassthrough, and you are responsible for configuring ingress manually

Default: baseDomainPassthrough

When baseDomainPassthrough is enabled (the default when no baseDomain is specified), HyperShift automatically configures all ingress routing infrastructure on the management cluster. No manual LoadBalancer or DNS setup is required.

What HyperShift creates automatically

1. A wildcard passthrough Route on the management cluster with TLSTerminationPassthrough and WildcardPolicySubdomain. This Route matches all *.apps.<guest>.<mgmt-apps-domain> requests and forwards them without terminating TLS.

2. A ClusterIP Service on the management cluster with an empty selector (no pod selector). The Service's target port is set to the guest router's HTTPS NodePort.

3. EndpointSlices managed by the Machine controller, pointing to the VM's machineNetwork IPs (not pod IPs) on the correct NodePort. These are automatically updated when VMs are added, removed, or live-migrated.

Resulting DNS domain

The guest cluster's base domain is auto-detected as a subdomain of the management cluster's *.apps domain. For example:

• Management cluster apps domain: *.apps.mgmt-cluster.example.com
• Guest cluster named guest: *.apps.guest.apps.mgmt-cluster.example.com

How the default ingress passthrough works

The default ingress passthrough is implemented in the infra cluster namespace where the KubeVirt VMs run with:

• A selector-less ClusterIP Service (default-ingress-passthrough-service-<id>) exposing port 443.
• EndpointSlices for that Service, one per worker VM, pointing at the VM internal IPs and the port where the guest routers listen.
• A wildcard passthrough Route (default-ingress-passthrough-route-<id>) for *.apps.<guest>.<infra base domain> targeting that Service.

The port targeted on the VMs depends on the guest default IngressController endpointPublishingStrategy:

• NodePortService (default for KubeVirt): the HTTPS nodePort of the openshift-ingress/router-nodeport-default Service in the guest cluster.

• HostNetwork: the hostNetwork.httpsPort (defaults to 443). This strategy can be selected at creation time through spec.operatorConfiguration.ingressOperator.endpointPublishingStrategy in the HostedCluster, for example:

  spec:
    operatorConfiguration:
      ingressOperator:
        endpointPublishingStrategy:
          type: HostNetwork
          hostNetwork:
            httpPort: 80
            httpsPort: 443
            statsPort: 1936
            protocol: TCP
  

Other endpoint publishing strategies (e.g. LoadBalancerService) are not supported by the default ingress passthrough; use the custom baseDomain behavior described below instead.

Prerequisites

The management cluster must allow wildcard DNS routes:

oc patch ingresscontroller -n openshift-ingress-operator default \
  --type=json \
  -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {"wildcardPolicy": "WildcardsAllowed"}}]'

Custom baseDomain (without baseDomainPassthrough)

When you provide an explicit baseDomain at creation time, HyperShift does not enable baseDomainPassthrough and does not create any ingress routing infrastructure on the management cluster. You are fully responsible for configuring:

1. A LoadBalancer Service on the management cluster
2. An EndpointSlice pointing to the VM machineNetwork IPs
3. A wildcard DNS record for *.apps.<cluster-name>.<baseDomain>

The traffic flow for this configuration is:

Client
  └─> *.apps.<cluster>.<baseDomain>     (DNS wildcard)
       └─> LoadBalancer VIP              (MetalLB / external LB)
            └─> VM machineNetwork IP     (EndpointSlice target)
                 └─> NodePort            (guest router)
                      └─> guest Route    (application)

This process involves four steps:

1. Cluster creation
2. LoadBalancer and EndpointSlice creation
3. Wildcard DNS configuration
4. Verification

Step 1 - Deploy the HostedCluster with a custom baseDomain

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

export CLUSTER_NAME=example
export PULL_SECRET="$HOME/pull-secret"
export MEM="6Gi"
export CPU="2"
export WORKER_COUNT="2"
export BASE_DOMAIN=hypershift.lab

hcp create cluster kubevirt \
--name $CLUSTER_NAME \
--node-pool-replicas $WORKER_COUNT \
--pull-secret $PULL_SECRET \
--memory $MEM \
--cores $CPU \
--base-domain $BASE_DOMAIN

This creates a HostedCluster with ingress wildcard *.apps.example.hypershift.lab.

The HostedCluster will remain in Partial progress until the LoadBalancer and DNS are configured:

1
2
3
4

oc get --namespace clusters hostedclusters

NAME            VERSION   KUBECONFIG                       PROGRESS   AVAILABLE   PROGRESSING   MESSAGE
example                   example-admin-kubeconfig         Partial    True        False         The hosted control plane is available

Step 2 - Set up the LoadBalancer and EndpointSlice

1. Retrieve the guest cluster NodePorts

export CLUSTER_KUBECONFIG="${CLUSTER_NAME}-kubeconfig"
hcp create kubeconfig --name $CLUSTER_NAME > $CLUSTER_KUBECONFIG

export HTTP_NODEPORT=$(oc --kubeconfig $CLUSTER_KUBECONFIG get services \
  -n openshift-ingress router-nodeport-default \
  -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}')

export HTTPS_NODEPORT=$(oc --kubeconfig $CLUSTER_KUBECONFIG get services \
  -n openshift-ingress router-nodeport-default \
  -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}')

echo "HTTP NodePort: $HTTP_NODEPORT"
echo "HTTPS NodePort: $HTTPS_NODEPORT"

2. Retrieve the VM machineNetwork IPs

export HCP_NAMESPACE="clusters-${CLUSTER_NAME}"

oc get vmi -n $HCP_NAMESPACE -o json | \
  jq -r '.items[] | "\(.metadata.name)\t\(.status.interfaces[] | select(.name != "default" and .ipAddress != null and .ipAddress != "") | .ipAddress | split("/")[0])"'

This command filters out the pod network interface (default) and strips any CIDR suffix from the IP address. If your VMs use a different interface layout, list all interfaces with oc get vmi -n $HCP_NAMESPACE -o yaml and adjust the filter accordingly.

Save the VM IPs for use in the EndpointSlice below. For example:

example-workers-abc12-xyz34    192.168.216.50
example-workers-abc12-xyz56    192.168.216.51

3. Create the LoadBalancer Service (no selector)

cat << EOF | oc apply -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    app: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}-apps
  namespace: ${HCP_NAMESPACE}
spec:
  ports:
  - name: https-443
    port: 443
    protocol: TCP
    targetPort: ${HTTPS_NODEPORT}
  - name: http-80
    port: 80
    protocol: TCP
    targetPort: ${HTTP_NODEPORT}
  type: LoadBalancer
EOF

Note that the Service has no selector field. Traffic routing is handled by the EndpointSlice created in the next step.

4. Create the EndpointSlice

Replace the IP addresses below with the VM machineNetwork IPs retrieved in step 2:

cat << EOF | oc apply -f -
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: ${CLUSTER_NAME}-apps-endpoints
  namespace: ${HCP_NAMESPACE}
  labels:
    kubernetes.io/service-name: ${CLUSTER_NAME}-apps
    endpointslice.kubernetes.io/managed-by: manual
addressType: IPv4
ports:
- name: https-443
  port: ${HTTPS_NODEPORT}
  protocol: TCP
- name: http-80
  port: ${HTTP_NODEPORT}
  protocol: TCP
endpoints:
- addresses:
  - "192.168.216.50"
- addresses:
  - "192.168.216.51"
EOF

Step 3 - Set up a wildcard DNS record for *.apps

Configure a wildcard DNS record that references the LoadBalancer Service's external address:

1. Get the external address. Depending on the load balancer provider, either .ip (IP-based, e.g., MetalLB, GCE) or .hostname (DNS-based, e.g., AWS ELB) is populated:

   export EXTERNAL_IP=$(oc -n $HCP_NAMESPACE get service ${CLUSTER_NAME}-apps \
     -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   export EXTERNAL_HOSTNAME=$(oc -n $HCP_NAMESPACE get service ${CLUSTER_NAME}-apps \
     -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   

2. Configure a wildcard *.apps.<cluster_name>.<base_domain>. DNS entry. The DNS record must be routable both from outside the cluster and from inside the guest VMs (see the Troubleshooting section for hairpin issues).

   • If $EXTERNAL_IP is set, create a wildcard A record:

     *.apps.example.hypershift.lab.  IN  A  192.168.20.30
     

   • If $EXTERNAL_HOSTNAME is set instead, create a wildcard CNAME record:

     *.apps.example.hypershift.lab.  IN  CNAME  a1b2c3-1234.us-east-1.elb.amazonaws.com.
     

   Verify DNS resolves correctly:

   dig +short test.apps.example.hypershift.lab
   

Step 4 - Verify the HostedCluster status

Once the LoadBalancer and DNS are in place, the HostedCluster progress should move from Partial to Completed:

1
2
3
4

oc get --namespace clusters hostedclusters

NAME            VERSION   KUBECONFIG                       PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
example         4.14.0    example-admin-kubeconfig         Completed   True        False         The hosted control plane is available

Optional MetalLB Configuration Steps

LoadBalancer type services are required. If MetalLB is in use, here are some example steps outlining how to configure MetalLB after installing MetalLB using CLI.

1. Create a MetalLB instance:

   oc create -f - <<EOF
   apiVersion: metallb.io/v1beta1
   kind: MetalLB
   metadata:
     name: metallb
     namespace: metallb-system
   EOF
   

2. Create address pool with an available range of IP addresses within the node network:

   oc create -f - <<EOF
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     name: metallb
     namespace: metallb-system
   spec:
     addresses:
     - 192.168.216.200-192.168.216.220
   EOF
   

3. Advertise the address pool using L2 protocol:

   oc create -f - <<EOF
   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: l2advertisement
     namespace: metallb-system
   spec:
     ipAddressPools:
      - metallb
   EOF
   

Troubleshooting

CanaryChecksRepetitiveFailures with custom baseDomain

When using a custom baseDomain (without baseDomainPassthrough), the ingress operator may report Degraded with errors like:

CanaryChecksRepetitiveFailures: Canary route checks for the default ingress
controller are failing. Last 1 error messages:
error sending canary HTTP request: http: server gave HTTP response to HTTPS client

or:

connection refused

Diagnostic steps

1. Verify DNS resolution from inside the guest VMs. The canary check runs from inside the guest cluster, so DNS must resolve correctly from within the VMs:

   oc --kubeconfig $CLUSTER_KUBECONFIG debug node/<any-guest-node> -- \
     chroot /host nslookup canary-openshift-ingress-canary.apps.${CLUSTER_NAME}.${BASE_DOMAIN}
   

   Compare this IP with the LoadBalancer VIP. If they differ, DNS is misconfigured.

2. Verify the LoadBalancer endpoints use machineNetwork IPs, not pod IPs. Check the EndpointSlice:

   oc get endpointslice -n $HCP_NAMESPACE -l kubernetes.io/service-name=${CLUSTER_NAME}-apps -o yaml
   

   The IP addresses in the EndpointSlice must be the VM machineNetwork IPs (the same IPs returned by the oc get vmi -o json | jq command in Step 2), not the virt-launcher pod IPs. If the EndpointSlice contains pod network IPs (typically in a different CIDR than the machineNetwork), the guest router NodePort will not be reachable and connections will be refused.

3. Test LoadBalancer VIP reachability from inside the guest. Curl the LoadBalancer VIP from within a guest VM:

   oc --kubeconfig $CLUSTER_KUBECONFIG debug node/<any-guest-node> -- \
     chroot /host curl -vk --connect-timeout 5 https://<EXTERNAL_IP>:443
   

   If this returns connection refused but the same curl works from outside the guest VMs, the issue is VIP return-path routing — the VMs are sending traffic to a VIP that routes back to themselves, but the return path is broken (asymmetric routing). Configure split-horizon DNS so that guest VMs resolve *.apps directly to their own machineNetwork IPs instead of the external VIP. See the JSON patch DNS override in the recipe for an automated approach.