VR only starts to work at 100% after migration

[n4l5u0r]

Hello, I have an issue on a VR which is created as part of default: DefaultNetworkOfferingforKubernetesService.
when I create it and launch CKS in it I have an issue of never achieving Running state of the CKS cluster which stays in Startingstate.

If I launch CKS without specifying a network it creates it by itself and same issue.

But once I migrate the created VR from one KVM to another then everything is smooth and works perfectly, I achieve the Runningstate instantly no matter the configuration of the post deployment of the CKS on this network.

So I don't understand what is wrong with the initial VR startup which looks like not having the full config until a migration happens.
It looks like it is a network misconfiguration as if deploy the VR without any migration and establish a ssh tunnel between the KVM hosting the VR and the mgmt 8080 endpoint then it works without going though a migration.

I am on 4.20.0.2 and using 1.33.1-calico-Kubernetes-Binaries-ISO

So it looks like the VR does not have the full network topology at start and receive it at migration... is it possible to have a workaround without using and ssh tunnel neither using migration.

[weizhouapache]

@n4l5u0r
if you use vlan, have you configured vlan trunking on upstream router ?

[weizhouapache]

ok, so it looks like not a vlan misconfiguration issue of guest network

can you check if the public IP of CKS nodes are reachable from management server when VR is running on various host ? ACS configures the CKS nodes via the Public IP.

ssh -i ~cloud/.ssh/id_rsa -p 2222 cloud@<Public ip address of CKS cluster>

just to verify if the issue with public network

[n4l5u0r]

ok, so it looks like not a vlan misconfiguration issue of guest network

can you check if the public IP of CKS nodes are reachable from management server when VR is running on various host ? ACS configures the CKS nodes via the Public IP.

ssh -i ~cloud/.ssh/id_rsa -p 2222 cloud@<Public ip address of CKS cluster>

just to verify if the issue with public network

It is accessible from KVM hosts not MGMT hosts as the mgmt network is isolated on 10.10.0.0/20 and the public network is on 10.40.0.0/20.

Have just a proxy on port 8080 from public network to mgmt network to reach the endpoint.url to update the CKS status. This endpoint is triggered properly once the VR has been migrated once.

Our current setup:

[Mgmt hosts <- VIP]<-[haproxy<-KVM hosts<-VIP]<-VR

HAPROXY switch the 8080 http traffic from 10.40.0.0/20 to 10.10.0.0/20

VIP though keepalived on each host

[weizhouapache]

Our current setup:

[Mgmt hosts <- VIP]<-[haproxy<-KVM hosts<-VIP]<-VR

HAPROXY switch the 8080 http traffic from 10.40.0.0/20 to 10.10.0.0/20

VIP though keepalived on each host

hm, this setup looks like not very good for production.

[n4l5u0r]

Our current setup:

[Mgmt hosts <- VIP]<-[haproxy<-KVM hosts<-VIP]<-VR

HAPROXY switch the 8080 http traffic from 10.40.0.0/20 to 10.10.0.0/20

VIP though keepalived on each host

hm, this setup looks like not very good for production.

Not production target as keepalived will be replaced by lb but there is still an issue with VR not working properly on start with CKS.

What are your recommendations to have the best architecture to have it working for CKS but also secure. Because this endpoint.url from public to mgmt is really a security risk which we would like to avoid.

[weizhouapache]

What are your recommendations to have the best architecture to have it working for CKS but also secure. Because this endpoint.url from public to mgmt is really a security risk which we would like to avoid.

the CKS to mgmt is needed only when you use cloudstack-kubernetes-provider or cloudstack-csi-driver after CKS is setup.
the first issue you need to solve it the connection from mgmt to CKS, it must be stable.

IMHO you should look at the haproxy configuration. maybe try with other balance algorithms.
I cannot help you more on it

[n4l5u0r]

@weizhouapache do you know where can I look to see what is happening ? the mangement logs don't report any issue on this point... Is it more on KVM side ? or within the VR itself maybe is it possible to identify the issues post boot ?

[weizhouapache]

@weizhouapache do you know where can I look to see what is happening ? the mangement logs don't report any issue on this point... Is it more on KVM side ? or within the VR itself maybe is it possible to identify the issues post boot ?

I think the issue is not with cloudstack management server and VR, but your network configuration.

[n4l5u0r]

hi @weizhouapache one question regarding the initial provisionning: is it normal to have these logs on initial boot of the VR or any VM:

Especially these ones : libvirt: QEMU Driver error : Domain not found: no domain with matching name 'r-4-VM' and libvirt: Storage Driver error : Storage volume not found: no storage vol with matching name 'd1f85858-e00a-4971-92a1-13ed2b92103c'

2025-12-10 12:08:56,936 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool d7c0ff52-f8df-3d44-bd21-a5814655ac1d from libvirt

==> /var/log/cloudstack/agent/agent.err <==
libvirt: Storage Driver error : Storage volume not found: no storage vol with matching name 'd1f85858-e00a-4971-92a1-13ed2b92103c'

==> /var/log/cloudstack/agent/agent.log <==
2025-12-10 12:08:57,099 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool d7c0ff52-f8df-3d44-bd21-a5814655ac1d from libvirt
2025-12-10 12:08:57,300 WARN  [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-5:[]) (logid:) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2025-12-10 12:08:57,300 WARN  [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-5:[]) (logid:) Groovy scripting engine is not initialized. Data transformation skipped.

==> /var/log/cloudstack/agent/agent.err <==
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'r-4-VM'

==> /var/log/cloudstack/agent/agent.log <==
2025-12-10 12:08:58,216 WARN  [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-5:[]) (logid:) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2025-12-10 12:08:58,216 WARN  [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-5:[]) (logid:) Groovy scripting engine is not initialized. Data transformation skipped.

==> /var/log/cloudstack/agent/security_group.log <==
2025-12-10 12:09:10,815 - Executing command: get_rule_logs_for_vms

==> /var/log/cloudstack/agent/agent.log <==
2025-12-10 12:09:32,210 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:) Trying to fetch storage pool d7c0ff52-f8df-3d44-bd21-a5814655ac1d from libvirt
2025-12-10 12:09:32,213 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:) Asking libvirt to refresh storage pool d7c0ff52-f8df-3d44-bd21-a5814655ac1d

[weizhouapache]

r-4-VM

it looks this vr is not running on the host, so stopping it throwed some warnings.
I think you can ignore the warnings and errors

[n4l5u0r]

Hello @weizhouapache

just want to share this behavior which I really don't understand :

1. When creating an isolated network for CKS with underlying VR I fails to pass the Starting state of CKS to Running state.
2. Then when migrating the VR the CKS switch to Running state.
3. Delete the CKS to have an empty network.
4. But then rebuilding entirely the network by replacing the VR with Clear option with a new one any newly created CKS works instantly and switch to Rinning state without any issue and without any migration.

[weizhouapache]

Hello @weizhouapache

just want to share this behavior which I really don't understand :

1. When creating an isolated network for CKS with underlying VR I fails to pass the Starting state of CKS to Running state.
2. Then when migrating the VR the CKS switch to Running state.
3. Delete the CKS to have an empty network.
4. But then rebuilding entirely the network by replacing the VR with Clear option with a new one any newly created CKS works instantly and switch to Rinning state without any issue and without any migration.

I do not understand either, as I do not know your configurations in details.

I believe it is caused by your network configurations.

[mwaag]

Hi,
i stumbled upon this since i have a very similar issue but with some differences.

First some differences in setup:

• Version 4.20.3.0
• Hypervisor: VMWare vCentred ESX Cluster 8.x (but only 1 Workload-Host, as this is a testing-environment, management-server is on a dedicated ESX-Cluster but managed from the same vCentre)
• connection from mgmt to CKS is stable (we run into this problem before ourself, ssh -i ~cloud/.ssh/id_rsa -p 2222 cloud@ does work)

Here my issue:

• when using the default DefaultNetworkOfferingforKubernetesService Network Offering (or just dont specify the target-network at deployment) my cluster never reaches Running-State and seems to be stuck in Starting state. Wether the kubeconfig is coming available via webinterface nor does this cluster ever run into error state. The problem i see here is, there is no successful creation of port-forwarding of 6443 to the control node, when i try to add it manually, i get the error: Request failed. (431)
  There are multiple services used ip 192.168.8.20. (case 1).

• But using an isolated network (that has the needed services enabled) within a vpc do not raise this issue. The cluster is getting deployed fine and reports running state successfully, kubeconfig gets downloadable via webinterface as well when all nodes are deployed (case 2).

Naturally there are some differences between these offerings, here are my listings of both offerings, can you spot the cause?

list networkofferings id=e19d2436-5935-4943-a1d3-1540ac4fb41c (DefaultNetworkOfferingforKubernetesService)
{
"count": 1,
"networkoffering": [
{
"availability": "Required",
"conservemode": true,
"created": "2024-06-25T16:42:04+0200",
"displaytext": "Network Offering used for CloudStack Kubernetes service",
"egressdefaultpolicy": true,
"fornsx": false,
"fortungsten": false,
"forvpc": false,
"guestiptype": "Isolated",
"hasannotations": false,
"id": "e19d2436-5935-4943-a1d3-1540ac4fb41c",
"internetprotocol": "IPv4",
"isdefault": true,
"ispersistent": false,
"name": "DefaultNetworkOfferingforKubernetesService",
"networkrate": 200,
"service": [
{
"name": "Vpn",
"provider": [
{
"name": "VirtualRouter"
}
]
},
{
"name": "PortForwarding",
"provider": [
{
"name": "VirtualRouter"
}
]
},
{
"capability": [
{
"canchooseservicecapability": false,
"name": "SupportedSourceNatTypes",
"value": "peraccount"
},
{
"canchooseservicecapability": false,
"name": "RedundantRouter",
"value": "false"
}
],
"name": "SourceNat",
"provider": [
{
"name": "VirtualRouter"
}
]
},
{
"name": "Dns",
"provider": [
{
"name": "VirtualRouter"
}
]
},
{
"capability": [
{
"canchooseservicecapability": false,
"name": "ElasticIp",
"value": "false"
},
{
"canchooseservicecapability": false,
"name": "AssociatePublicIP",
"value": "true"
}
],
"name": "StaticNat",
"provider": [
{
"name": "VirtualRouter"
}
]
},
{
"name": "UserData",
"provider": [
{
"name": "VirtualRouter"
}
]
},
{
"name": "Dhcp",
"provider": [
{
"name": "VirtualRouter"
}
]
},
{
"capability": [
{
"canchooseservicecapability": false,
"name": "SupportedLBIsolation",
"value": "dedicated"
},
{
"canchooseservicecapability": false,
"name": "ElasticLb",
"value": "false"
},
{
"canchooseservicecapability": false,
"name": "InlineMode",
"value": "false"
},
{
"canchooseservicecapability": false,
"name": "VmAutoScaling",
"value": "true"
}
],
"name": "Lb",
"provider": [
{
"name": "VirtualRouter"
}
]
},
{
"name": "Firewall",
"provider": [
{
"name": "VirtualRouter"
}
]
}
],
"serviceofferingid": "95c4a416-9c7e-467e-9e4a-e237ac4318b9",
"specifyasnumber": false,
"specifyipranges": false,
"specifyvlan": false,
"state": "Enabled",
"supportsinternallb": false,
"supportspublicaccess": false,
"supportsstrechedl2subnet": false,
"traffictype": "Guest"
}
]
}

list networkofferings id=33c2d2a0-f230-4db8-9b29-bef8f0fa34de (the custom one)
{
"count": 1,
"networkoffering": [
{
"availability": "Optional",
"conservemode": false,
"created": "2024-06-26T12:08:30+0200",
"details": {
"internetProtocol": "IPv4"
},
"displaytext": "NAT for VPC with Loadbalancing",
"egressdefaultpolicy": true,
"fornsx": false,
"fortungsten": false,
"forvpc": true,
"guestiptype": "Isolated",
"hasannotations": false,
"id": "33c2d2a0-f230-4db8-9b29-bef8f0fa34de",
"internetprotocol": "IPv4",
"isdefault": false,
"ispersistent": false,
"maxconnections": 4096,
"name": "NAT for VPC with Loadbalancing",
"networkrate": 500,
"service": [
{
"name": "Vpn",
"provider": [
{
"name": "VpcVirtualRouter"
}
]
},
{
"name": "PortForwarding",
"provider": [
{
"name": "VpcVirtualRouter"
}
]
},
{
"capability": [
{
"canchooseservicecapability": false,
"name": "SupportedSourceNatTypes",
"value": "peraccount"
},
{
"canchooseservicecapability": false,
"name": "RedundantRouter",
"value": "false"
}
],
"name": "SourceNat",
"provider": [
{
"name": "VpcVirtualRouter"
}
]
},
{
"name": "Dns",
"provider": [
{
"name": "VpcVirtualRouter"
}
]
},
{
"capability": [
{
"canchooseservicecapability": false,
"name": "ElasticIp",
"value": "false"
},
{
"canchooseservicecapability": false,
"name": "AssociatePublicIP",
"value": "false"
}
],
"name": "StaticNat",
"provider": [
{
"name": "VpcVirtualRouter"
}
]
},
{
"name": "UserData",
"provider": [
{
"name": "VpcVirtualRouter"
}
]
},
{
"name": "Dhcp",
"provider": [
{
"name": "VpcVirtualRouter"
}
]
},
{
"capability": [
{
"canchooseservicecapability": false,
"name": "SupportedLBIsolation",
"value": "dedicated"
},
{
"canchooseservicecapability": false,
"name": "ElasticLb",
"value": "false"
},
{
"canchooseservicecapability": false,
"name": "InlineMode",
"value": "false"
},
{
"canchooseservicecapability": false,
"name": "VmAutoScaling",
"value": "true"
}
],
"name": "Lb",
"provider": [
{
"name": "VpcVirtualRouter"
}
]
},
{
"name": "NetworkACL",
"provider": [
{
"name": "VpcVirtualRouter"
}
]
}
],
"serviceofferingid": "95c4a416-9c7e-467e-9e4a-e237ac4318b9",
"specifyasnumber": false,
"specifyipranges": false,
"specifyvlan": false,
"state": "Enabled",
"supportsinternallb": false,
"supportspublicaccess": false,
"supportsstrechedl2subnet": false,
"traffictype": "Guest"
}
]
}