Ceph RBD No remote endpoint to send command

[miloserdoff]

Hi, I'm using Apache Cloudstack 4.22.1.0. I have Primary Storage with Ceph RBD. When I click on the Browser, I have no data output from the storage.

Here is the log from Cloudstack Management:

2026-07-01 07:37:41,069 ERROR [c.c.a.ApiServer] (qtp1404565079-56:[ctx-b4b08029, ctx-289639e4]) (logid:088e725b) unhandled exception executing api command: [Ljava.lang.String;@2641e182 com.cloud.utils.exception.CloudRuntimeException: No remote endpoint to send command
        at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listObjectsInStore(StorageBrowserImpl.java:183)
        at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listPrimaryStoreObjects(StorageBrowserImpl.java:139)
        at org.apache.cloudstack.api.command.admin.storage.ListStoragePoolObjectsCmd.execute(ListStoragePoolObjectsCmd.java:73)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:875)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:689)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:417)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:194)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:191)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:145)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:840)

[miloserdoff]

@weizhouapache Can you help me?

[weizhouapache]

@miloserdoff
according to https://www.shapeblue.com/cloudstack-storage-browser/, only NFS is supported.

@vishesh92 is it still the same now ?

[miloserdoff]

@weizhouapache No, I have another Apache Cloudstack cluster where Primary Storage also uses RBD, and everything works there.

Until recently, everything was working fine on the cluster, where a problem has now arisen.

[vishesh92]

@miloserdoff Is SSVM running?

[miloserdoff]

@vishesh92 Yes, SSVMs are running. The hosts are running too.

[miloserdoff]

@vishesh92 P.S. I have a second primary storage on NFS, and the error doesn't occur there. The error only occurs on RBD.

[weizhouapache]

@weizhouapache No, I have another Apache Cloudstack cluster where Primary Storage also uses RBD, and everything works there.

Until recently, everything was working fine on the cluster, where a problem has now arisen.

this seems not to be the files on ceph rbd, but in a linux server ...

[miloserdoff]

@weizhouapache What should I double-check to fix the "No remote endpoint to send command" error?

[weizhouapache]

@weizhouapache What should I double-check to fix the "No remote endpoint to send command" error?

@miloserdoff
can you share the management-server.log in the period ?

[miloserdoff]

@weizhouapache

2026-07-01 12:19:07,438 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 21-: Sending 180
2026-07-01 12:19:07,650 DEBUG [c.c.a.ApiServlet] (qtp1404565079-1062:[ctx-5961431f]) (logid:5323db80) ===START===  10.10.3.8 -- GET  path=&id=b3bbaf2d-dadc-39d0-b4d4-47f59838b7a9&page=1&pagesize=10&command=listStoragePoolObjects&response
=json&
2026-07-01 12:19:07,650 DEBUG [c.c.a.ApiServlet] (qtp1404565079-1062:[ctx-5961431f]) (logid:5323db80) Two factor authentication is already verified for the user 2, so skipping
2026-07-01 12:19:07,652 DEBUG [c.c.a.ApiServer] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"e5525160-9533-11f0-827b-525400bd7713"}]' is allow
ed to perform API calls: 0.0.0.0/0,::/0
2026-07-01 12:19:07,653 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) Account for user id e552a874-9533-11f0-827b-525400bd7713 is Root Admin or Domain Admin, all APIs
are allowed.
2026-07-01 12:19:07,653 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-07-01 12:19:07,653 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-07-01 12:19:07,655 ERROR [c.c.a.ApiServer] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) unhandled exception executing api command: [Ljava.lang.String;@3a02048f com.cloud.utils.exception.CloudRuntimeException: N
o remote endpoint to send command
        at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listObjectsInStore(StorageBrowserImpl.java:183)
        at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listPrimaryStoreObjects(StorageBrowserImpl.java:139)
        at org.apache.cloudstack.api.command.admin.storage.ListStoragePoolObjectsCmd.execute(ListStoragePoolObjectsCmd.java:73)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:875)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:689)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:417)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:194)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:191)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:145)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-07-01 12:19:07,656 DEBUG [c.c.a.ApiServlet] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) ===END===  10.10.3.8 -- GET  path=&id=b3bbaf2d-dadc-39d0-b4d4-47f59838b7a9&page=1&pagesize=10&command=listStoragePoolObje
cts&response=json&
2026-07-01 12:19:08,483 INFO  [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc6c645d]) (logid:8b9e4af6) Found 1 inactive management server node based on timestamp
2026-07-01 12:19:08,483 INFO  [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc6c645d]) (logid:8b9e4af6) management server node msid: 90520743147283, name: node, service ip: 192.168.200.20, version: 4.21.0.0
2026-07-01 12:19:08,483 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc6c645d]) (logid:8b9e4af6) Peer scan is finished. profiler: Done. Duration: 0ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo
: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms
2026-07-01 12:19:08,886 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"node","_uuid":"a02fa8e7-0e09-448e-9acb-6a95aafeb42f"}
2026-07-01 12:19:08,887 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"node","type":"Routing","uuid":"a02fa8e7-0e09-448e-9acb
-6a95aafeb42f"}
2026-07-01 12:19:08,888 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"node","type":"Routing","uuid":"a02fa8e7-0e09-448e-9acb-6a95aafeb42f"}, number of re
cords in report: 21. VMs: [1792:PowerOn, 1826:PowerOn, 1827:PowerOn, 1636:PowerOn, 1893:PowerOn, 1737:PowerOn, 41:PowerOn, 1833:PowerOn, 266:PowerOn, 1834:PowerOn, 1835:PowerOn, 235:PowerOn, 1905:PowerOn, 337:PowerOn, 84:PowerOn, 1911:Po
werOn, 1339:PowerOn, 1820:PowerOn, 1886:PowerOn, 1822:PowerOn, 1663:PowerOn]]
2026-07-01 12:19:08,890 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"node","type":"Routing","uuid":"a02fa8e7-0e09-448e-9acb-6a95aafeb
42f"}
2026-07-01 12:19:08,912 DEBUG [c.c.a.ApiServlet] (qtp1404565079-361:[ctx-490c2445]) (logid:63484896) ===START===  10.10.3.8 -- GET  storageid=b3bbaf2d-dadc-39d0-b4d4-47f59838b7a9&command=listConfigurations&response=json&
2026-07-01 12:19:08,912 DEBUG [c.c.a.ApiServlet] (qtp1404565079-361:[ctx-490c2445]) (logid:63484896) Two factor authentication is already verified for the user 2, so skipping
2026-07-01 12:19:08,915 DEBUG [c.c.a.ApiServer] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"e5525160-9533-11f0-827b-525400bd7713"}]' is allowe
d to perform API calls: 0.0.0.0/0,::/0
2026-07-01 12:19:08,915 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Account for user id e552a874-9533-11f0-827b-525400bd7713 is Root Admin or Domain Admin, all APIs a
re allowed.
2026-07-01 12:19:08,915 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-07-01 12:19:08,915 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-07-01 12:19:08,928 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [kvm.storage.pool.io.policy] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,931 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [sp.alternative.endpoint] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,931 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [sp.cluster.location] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,932 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [sp.enable.alternative.endpoint] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,932 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storage.overprovisioning.factor] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,932 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storage.pool.client.max.connections] in Config.java. Perhaps moved to ConfigDepo
t.
2026-07-01 12:19:08,933 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storage.pool.client.timeout] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,933 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storage.pool.connected.clients.limit] in Config.java. Perhaps moved to ConfigDep
ot.
2026-07-01 12:19:08,933 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storage.pool.disk.wait] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,934 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storpool.delete.after.interval] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,934 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [use.https.to.upload] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,934 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [use.storage.replication] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,935 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [vmware.create.full.clone] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,936 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [volume.check.and.repair.leaks.before.use] in Config.java. Perhaps moved to Confi
gDepot.
2026-07-01 12:19:08,936 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [volume.resize.allowed.beyond.allocation] in Config.java. Perhaps moved to Config
Depot.

[weizhouapache]

@miloserdoff
the logs said no host can be found to list the files on the primary storage. but the hosts look ok, I am confused.

I doubt if browsering ceph rbd is supported. IMHO, the files in your screenshot seems to be files on a kvm host, not on ceph rbd.

[miloserdoff]

@weizhouapache I ran an experiment. I disabled all primary storage except the problematic one.

I created a virtual machine and got an error:

Logs:

2026-07-02 05:50:53,403 WARN  [c.c.a.AlertManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) alertType=[8] dataCenterId=[1] podId=[null] clusterId=[null] message=[Failed to deploy Vm VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Error","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}, on Host null with Id: null].
2026-07-02 05:50:53,405 WARN  [c.c.a.AlertManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Failed to deploy Vm VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Error","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}, on Host null with Id: null] and content [Failed to deploy Vm VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Error","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}, on Host null with Id: null].
2026-07-02 05:50:53,406 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) Updating resource Type = user_vm count for Account with id = 2 Operation = decreasing Amount = 1
2026-07-02 05:50:53,406 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) Updating resource Type = cpu count for Account with id = 2 Operation = decreasing Amount = 32
2026-07-02 05:50:53,407 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) Updating resource Type = memory count for Account with id = 2 Operation = decreasing Amount = 65536
2026-07-02 05:50:53,410 INFO  [o.a.c.a.c.a.v.DeployVMCmdByAdmin] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) No destination found for a deployment for VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Stopped","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}: No destination found for a deployment for VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Stopped","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}
2026-07-02 05:50:53,410 DEBUG [o.a.c.a.c.a.v.DeployVMCmdByAdmin] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) No destination found for a deployment for VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Stopped","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"} com.cloud.exception.InsufficientServerCapacityException: No destination found for a deployment for VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Stopped","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}Scope=interface com.cloud.dc.DataCenter; id=1
        at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:227)
        at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachineUnchecked(UserVmManagerImpl.java:5832)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5896)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5724)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5307)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5296)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy202.startVirtualMachine(Unknown Source)
        at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:98)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-07-02 05:50:53,411 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801]) (logid:f92282f1) Complete async job-11801, jobStatus: FAILED, resultCode: 533, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"533","errortext":"No destination found for a deployment for VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Stopped","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}"}

[miloserdoff]

@weizhouapache

P.S. An error also appears during migration:

2026-07-02 06:21:58,985 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820, ctx-6e96672a]) (logid:40dab182) Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
2026-07-02 06:21:58,985 DEBUG [c.c.s.VolumeApiServiceImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820, ctx-6e96672a]) (logid:40dab182) Failed to migrate volume com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1450)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy248.migrateVolume(Unknown Source)
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3869)
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:5393)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
        at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:5415)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy257.handleVmWorkJob(Unknown Source)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-07-02 06:21:58,985 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820, ctx-6e96672a]) (logid:40dab182) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
2026-07-02 06:21:58,985 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820, ctx-6e96672a]) (logid:40dab182) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
2026-07-02 06:21:58,986 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Done with run of VM work job: com.cloud.vm.VmWorkMigrateVolume for VM 1822, job origin: 11819
2026-07-02 06:21:58,986 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Unable to complete AsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkMigrateVolume","cmdInfo":"rO0ABXNyACBjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVZvbHVtZfgl82-871PmAgAESgAKZGVzdFBvb2xJZFoAC2xpdmVNaWdyYXRlSgAIdm9sdW1lSWRMABFuZXdEaXNrT2ZmZXJpbmdJZHQAEExqYXZhL2xhbmcvTG9uZzt4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHAAAAAAAAAAAgAAAAAAAAACAAAAAAAABx50ABRWb2x1bWVBcGlTZXJ2aWNlSW1wbAAAAAAAAAABAAAAAAAAAAQoc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAAk","cmdVersion":0,"completeMsid":null,"created":"Thu Jul 02 06:21:57 UTC 2026","id":11820,"initMsid":209369746774169,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"6cb9ff89-f8f4-4539-9888-5b23226fda64"}, job origin: 11819 com.cloud.utils.exception.CloudRuntimeException: Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3876)
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:5393)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
        at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:5415)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy257.handleVmWorkJob(Unknown Source)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-07-02 06:21:58,987 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Complete async job-11820, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvblZNT3AAAAACAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANJSZXNvdXJjZSBbU3RvcmFnZVBvb2w6MV0gaXMgdW5yZWFjaGFibGU6IFZvbHVtZSBbeyJuYW1lIjoiUk9PVC0xODIyIiwidXVpZCI6ImQ4Yjk0YTJkLWUwMGEtNDk4Mi1iOTc5LTVkYTRhYzk5ZmYwNSJ9XSBtaWdyYXRpb24gZmFpbGVkIGR1ZSB0byBbTm8gcmVtb3RlIGVuZHBvaW50IHRvIHNlbmQgY29tbWFuZCwgY2hlY2sgaWYgaG9zdCBvciBzc3ZtIGlzIGRvd24_XS51cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAACBzcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAAhCAAZmb3JtYXRJAApsaW5lTnVtYmVyTAAPY2xhc3NMb2FkZXJOYW1lcQB-AAVMAA5kZWNsYXJpbmdDbGFzc3EAfgAFTAAIZmlsZU5hbWVxAH4ABUwACm1ldGhvZE5hbWVxAH4ABUwACm1vZHVsZU5hbWVxAH4ABUwADW1vZHVsZVZlcnNpb25xAH4ABXhwAQAADyR0AANhcHB0ACZjb20uY2xvdWQuc3RvcmFnZS5Wb2x1bWVBcGlTZXJ2aWNlSW1wbHQAGVZvbHVtZUFwaVNlcnZpY2VJbXBsLmphdmF0ABhvcmNoZXN0cmF0ZU1pZ3JhdGVWb2x1bWVwcHNxAH4ADAEAABURcQB-AA5xAH4AD3EAfgAQcQB-ABFwcHNxAH4ADAL____-cHQALWpkay5pbnRlcm5hbC5yZWZsZWN0Lk5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbHQAHU5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbC5qYXZhdAAHaW52b2tlMHQACWphdmEuYmFzZXQABzE3LjAuMTlzcQB-AAwCAAAATXBxAH4AFHEAfgAVdAAGaW52b2tlcQB-ABdxAH4AGHNxAH4ADAIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxAH4AGnEAfgAXcQB-ABhzcQB-AAwCAAACOXB0ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAacQB-ABdxAH4AGHNxAH4ADAEAAABmcQB-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAABUncQB-AA5xAH4AD3EAfgAQcQB-ACRwcHNxAH4ADAL____-cHEAfgAUcQB-ABVxAH4AFnEAfgAXcQB-ABhzcQB-AAwCAAAATXBxAH4AFHEAfgAVcQB-ABpxAH4AF3EAfgAYc3EAfgAMAgAAACtwcQB-ABxxAH4AHXEAfgAacQB-ABdxAH4AGHNxAH4ADAIAAAI5cHEAfgAfcQB-ACBxAH4AGnEAfgAXcQB-ABhzcQB-AAwBAAABWHEAfgAOdAAob3JnLnNwcmluZ2ZyYW1ld29yay5hb3Auc3VwcG9ydC5Bb3BVdGlsc3QADUFvcFV0aWxzLmphdmF0AB5pbnZva2VKb2lucG9pbnRVc2luZ1JlZmxlY3Rpb25wcHNxAH4ADAEAAADGcQB-AA50ADxvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5mcmFtZXdvcmsuUmVmbGVjdGl2ZU1ldGhvZEludm9jYXRpb250AB9SZWZsZWN0aXZlTWV0aG9kSW52b2NhdGlvbi5qYXZhdAAPaW52b2tlSm9pbnBvaW50cHBzcQB-AAwBAAAAo3EAfgAOcQB-AC9xAH4AMHQAB3Byb2NlZWRwcHNxAH4ADAEAAABhcQB-AA50AD9vcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5pbnRlcmNlcHRvci5FeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3J0ACBFeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3IuamF2YXEAfgAacHBzcQB-AAwBAAAAunEAfgAOcQB-AC9xAH4AMHEAfgAzcHBzcQB-AAwBAAAA13EAfgAOdAA0b3JnLnNwcmluZ2ZyYW1ld29yay5hb3AuZnJhbWV3b3JrLkpka0R5bmFtaWNBb3BQcm94eXQAF0pka0R5bmFtaWNBb3BQcm94eS5qYXZhcQB-ABpwcHNxAH4ADAD_____cHQAFGpkay5wcm94eTMuJFByb3h5MjU3cHEAfgAkdAAKamRrLnByb3h5M3BzcQB-AAwBAAAAY3EAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAK6cQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAxcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADhxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGdxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4ATHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADVxAH4ADnEAfgBPcQB-AEx0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC5xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgBIcQB-AElwcHNxAH4ADAEAAAKGcQB-AA5xAH4AQ3EAfgBEcQB-AElwcHNxAH4ADAIAAAIbcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgBNcQB-ABdxAH4AGHNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ASXEAfgAXcQB-ABhzcQB-AAwCAAAEcHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAXcQB-ABhzcQB-AAwCAAACe3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AF5xAH4ASXEAfgAXcQB-ABhzcQB-AAwCAAADSHB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ASXEAfgAXcQB-ABhzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
2026-07-02 06:21:58,987 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Publish async job-11820 complete on message bus
2026-07-02 06:21:58,987 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Wake up jobs related to job-11820
2026-07-02 06:21:58,987 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Update db status for job-11820
2026-07-02 06:21:58,988 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Wake up jobs joined with job-11820 and disjoin all subjobs created from job- 11820
2026-07-02 06:21:58,990 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Done executing com.cloud.vm.VmWorkMigrateVolume for job-11820
2026-07-02 06:21:58,990 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Remove job-11820 from job monitoring
2026-07-02 06:21:58,997 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-11:[ctx-cc30a767, job-11819]) (logid:40dab182) Unexpected exception while executing org.apache.cloudstack.api.command.admin.volume.MigrateVolumeCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3876)
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:5393)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
        at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:5415)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy257.handleVmWorkJob(Unknown Source)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-07-02 06:21:58,998 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-cc30a767, job-11819]) (logid:40dab182) Complete async job-11819, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?]."}

[miloserdoff]

@weizhouapache @vishesh92 My Primary Storage RBD has the ID b3bbaf2d-dadc-39d0-b4d4-47f59838b7a9

But it's not in virsh pool-list --all. Is that how it should be?

virsh pool-list --all
 Name                                   State    Autostart
------------------------------------------------------------
 5e2af39c-6d03-33ff-abc6-5662920ed5ff   active   no
 7079a70b-dad4-4c66-b900-416980aefd16   active   no
 vadm                                   active   yes