After reboot host connection issue

[TheKunalSen]

my setup was working well untill i rebooted all the servers and now two of my hosts are in alert mode and cant connect to the management but the the third one got connected

i am getting logs like this in two of my hosts

2026-06-17 12:16:19,938 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Process agent startup answer, agent [id: 0, uuid: , name: ] connected to the server
2026-06-17 12:16:19,940 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Startup Response Received: agent [id: 0, uuid: , name: ]
2026-06-17 12:16:20,086 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:4a2bdde0) Attempting to create storage pool 81ecd9fd-9cdf-34cb-b99d-51d0e2cf3a0c (NetworkFilesystem) in libvirt
2026-06-17 12:16:20,088 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:4a2bdde0) Found existing defined storage pool 81ecd9fd-9cdf-34cb-b99d-51d0e2cf3a0c, using it.
2026-06-17 12:16:20,095 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:4a2bdde0) Trying to fetch storage pool 81ecd9fd-9cdf-34cb-b99d-51d0e2cf3a0c from libvirt
2026-06-17 12:16:22,053 INFO  [cloud.agent.Agent] (Agent-Handler-1:[]) (logid:) Connected to the host: 172.16.17.201
2026-06-17 12:16:25,145 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Detected agent disconnect event, running through 0 disconnect hooks
2026-06-17 12:16:25,145 INFO  [cloud.agent.Agent] (Agent-Handler-1:[]) (logid:) Lost connection to host: 172.16.17.201. Attempting reconnection while we still have 0 commands in progress.
2026-06-17 12:16:25,147 INFO  [utils.nio.NioClient] (Agent-Handler-1:[]) (logid:) NioClient connection closed
2026-06-17 12:16:30,147 INFO  [cloud.agent.Agent] (Agent-Handler-1:[]) (logid:) Reconnecting to host: 172.16.17.201
2026-06-17 12:16:30,148 INFO  [utils.nio.NioClient] (Agent-Handler-1:[]) (logid:) Connecting to 172.16.17.201:8250
2026-06-17 12:16:30,148 INFO  [utils.nio.NioClient] (Agent-Handler-1:[]) (logid:) Connected to 172.16.17.201:8250
2026-06-17 12:16:30,149 INFO  [utils.nio.Link] (Agent-Handler-1:[]) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2026-06-17 12:16:30,240 INFO  [utils.nio.NioClient] (Agent-Handler-1:[]) (logid:) SSL: Handshake done
2026-06-17 12:16:30,312 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching CPU speed from command "lscpu".
2026-06-17 12:16:30,349 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Command [lscpu | grep -i 'Model name' | head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g'] resulted in the value [2100] for CPU speed.
2026-06-17 12:16:30,349 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching host CPU arch
2026-06-17 12:16:30,361 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Host uses control group [cgroup2fs].
2026-06-17 12:16:30,361 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Calculating the max shares of the host.
2026-06-17 12:16:30,361 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) The max shares of the host is [50400].
2026-06-17 12:16:30,431 WARN  [utils.script.Script] (Agent-Handler-1:[]) (logid:) Execution of process [12494] for command [/bin/bash -c virt-v2v --version ] failed.
2026-06-17 12:16:30,431 WARN  [utils.script.Script] (Agent-Handler-1:[]) (logid:) Process [12494] for command [/bin/bash -c virt-v2v --version ] encountered the error: [127].
2026-06-17 12:16:30,457 WARN  [utils.script.Script] (Agent-Handler-1:[]) (logid:) Execution of process [12514] for command [/bin/bash -c ovftool --version ] failed.
2026-06-17 12:16:30,457 WARN  [utils.script.Script] (Agent-Handler-1:[]) (logid:) Process [12514] for command [/bin/bash -c ovftool --version ] encountered the error: [127].
2026-06-17 12:16:32,018 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Attempting to create storage pool 2eb31580-54fb-4573-8401-083fbe23dcb7 (Filesystem) in libvirt
2026-06-17 12:16:32,042 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Found existing defined storage pool 2eb31580-54fb-4573-8401-083fbe23dcb7, using it.
2026-06-17 12:16:32,042 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Trying to fetch storage pool 2eb31580-54fb-4573-8401-083fbe23dcb7 from libvirt
2026-06-17 12:16:33,107 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Process agent startup answer, agent [id: 0, uuid: , name: ] connected to the server
2026-06-17 12:16:33,109 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Startup Response Received: agent [id: 0, uuid: , name: ]
2026-06-17 12:16:33,250 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:902a941f) Attempting to create storage pool 81ecd9fd-9cdf-34cb-b99d-51d0e2cf3a0c (NetworkFilesystem) in libvirt
2026-06-17 12:16:33,270 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:902a941f) Found existing defined storage pool 81ecd9fd-9cdf-34cb-b99d-51d0e2cf3a0c, using it.
2026-06-17 12:16:33,276 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:902a941f) Trying to fetch storage pool 81ecd9fd-9cdf-34cb-b99d-51d0e2cf3a0c from libvirt
2026-06-17 12:16:35,241 INFO  [cloud.agent.Agent] (Agent-Handler-1:[]) (logid:) Connected to the host: 172.16.17.201
2026-06-17 12:16:38,325 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Detected agent disconnect event, running through 0 disconnect hooks
2026-06-17 12:16:38,325 INFO  [cloud.agent.Agent] (Agent-Handler-1:[]) (logid:) Lost connection to host: 172.16.17.201. Attempting reconnection while we still have 0 commands in progress.
2026-06-17 12:16:38,327 INFO  [utils.nio.NioClient] (Agent-Handler-1:[]) (logid:) NioClient connection closed
2026-06-17 12:16:43,328 INFO  [cloud.agent.Agent] (Agent-Handler-1:[]) (logid:) Reconnecting to host: 172.16.17.201
2026-06-17 12:16:43,328 INFO  [utils.nio.NioClient] (Agent-Handler-1:[]) (logid:) Connecting to 172.16.17.201:8250
2026-06-17 12:16:43,329 INFO  [utils.nio.NioClient] (Agent-Handler-1:[]) (logid:) Connected to 172.16.17.201:8250
2026-06-17 12:16:43,329 INFO  [utils.nio.Link] (Agent-Handler-1:[]) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2026-06-17 12:16:43,420 INFO  [utils.nio.NioClient] (Agent-Handler-1:[]) (logid:) SSL: Handshake done
2026-06-17 12:16:43,471 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching CPU speed from command "lscpu".
2026-06-17 12:16:43,510 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Command [lscpu | grep -i 'Model name' | head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g'] resulted in the value [2100] for CPU speed.
2026-06-17 12:16:43,510 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching host CPU arch
2026-06-17 12:16:43,521 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Host uses control group [cgroup2fs].
2026-06-17 12:16:43,521 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Calculating the max shares of the host.
2026-06-17 12:16:43,521 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) The max shares of the host is

its like connecting and disconnecting

[weizhouapache]

@TheKunalSen
check management-server.log ?

[weizhouapache]

@TheKunalSen
could you restart the management service ?

[TheKunalSen]

I did restart the management service but still that compute 4 got connected easily but the compute 1 and 3 is still the same

[TheKunalSen]

the thing is when i tried to delet the compute 3 host it gave me an error

026-06-17 13:45:10,924 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-86:[ctx-8119c015]) (logid:0af5c09a) Total over provisioned capacity of the pool StoragePool {"id":6,"name":"primary","poolType":"NetworkFilesystem","uuid":"81ecd9fd-9cdf-34cb-b99d-51d0e2cf3a0c"} is (115.1981 TB) 126661639012352
2026-06-17 13:45:10,924 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-86:[ctx-8119c015]) (logid:0af5c09a) Successfully set Capacity - (115.1981 TB) 126661639012352 for capacity type - 3 , DataCenterId - 3, Pool - StoragePool {"id":6,"name":"primary","poolType":"NetworkFilesystem","uuid":"81ecd9fd-9cdf-34cb-b99d-51d0e2cf3a0c"}, PodId null
2026-06-17 13:45:10,925 ERROR [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-86:[ctx-8119c015]) (logid:0af5c09a) Monitor StoragePoolMonitor says there is an error in the connect process for 8 due to Cannot invoke "org.apache.cloudstack.storage.datastore.db.StoragePoolVO.isShared()" because "pool" is null java.lang.NullPointerException: Cannot invoke "org.apache.cloudstack.storage.datastore.db.StoragePoolVO.isShared()" because "pool" is null
        at com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:187)
        at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:770)
        at com.cloud.agent.manager.AgentManagerImpl.connectHostAgent(AgentManagerImpl.java:1403)
        at com.cloud.agent.manager.AgentManagerImpl.sendReadyAndGetAttache(AgentManagerImpl.java:1365)
        at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1419)
        at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1515)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-06-17 13:45:10,926 WARN  [c.c.u.d.GlobalLock] (AgentConnectTaskPool-86:[ctx-8119c015]) (logid:0af5c09a) Global lock re-entrance detected
2026-06-17 13:45:10,927 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-86:[ctx-8119c015]) (logid:0af5c09a) Acquired lock on host Host {"id":8,"name":"compute3","type":"Routing","uuid":"c64a6f6b-3b43-43c7-b6b5-9b2510a056b0"}, to process agent disconnection
2026-06-17 13:45:10,927 INFO  [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-86:[ctx-8119c015]) (logid:0af5c09a) Host AgentAttache {"_id":8,"_name":"compute3","_uuid":"c64a6f6b-3b43-43c7-b6b5-9b2510a056b0"} is disconnecting with event AgentDisconnected
2026-06-17 13:45:10,927 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-86:[ctx-8119c015]) (logid:0af5c09a) The next status of agent Host {"id":8,"name":"compute3","type":"Routing","uuid":"c64a6f6b-3b43-43c7-b6b5-9b2510a056b0"} is Alert, current status is Connecting
2026-06-17 13:45:10,927 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-86:[ctx-8119c015]) (logid:0af5c09a) Deregistering link for AgentAttache {"_id":8,"_name":"compute3","_uuid":"c64a6f6b-3b43-43c7-b6b5-9b2510a056b0"} with state Alert
2026-06-17 13:45:10,927 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-8

[weizhouapache]

You may want to check the storage_pool and storage_pool_host_ref tables for inconsistencies.

For example, a storage pool may be marked as removed in storage_pool, while corresponding records still exist in storage_pool_host_ref.

[TheKunalSen]

yes this was the issue

there were two null uuid rows which i deleted then the hosts got connected automatically