After removing a container-group Software from a device, the services contained in that software still remain in the UI

[emibcn]

From Support ticket CST-3187.

In some circumstances, after removing a container-group Software from a device, the services contained in that software (containers) still remain in the UI. This can be mitigated by unassigning the software using the cli:

DEVICE_NAME="MY-DEVICE-ID"
c8y devices list \
    --name "${DEVICE_NAME}" \
    | c8y devices get --raw \
    | jq -r '.childAdditions.references[] | .managedObject.id' \
    | c8y inventory delete

The circumstances in which it has been detected to happen is (verified):

• Remove a software
• Immediately after, stop the tedge-mapper-c8y.service in the device:

  systemctl stop tedge-mapper-c8y.service

• Wait for all the software to be removed (for example, watching the logs).
• Restart the service:

  systemctl start tedge-mapper-c8y.service

  The operation is marked as finished in the very same moment the mapper is started.
• The software has been removed from the UI/inventory, but the containers not.

[reubenmiller]

@emibcn thanks for the ticket. Can you provide details about the version of the plugin you're using and the tedge version?

Normally the container plugin service should delete the container services for you (as container are transient services), however it seems that this isn't always the case...one reason would be that at the time the container is removed, if the device is not connected to the internet then the service deletion won't be retried when the device goes online again.

Currently tedge doesn't have an api to propagate the deletion of services as there isn't an mqtt api on the Cumulocity side yet (but we have requested one).

[reubenmiller]

Ah wait. I just read the you stop the tedge-mapper-service! this would explain everything as the tedge-mapper-c8y provides the local c8y proxy service that is used by the container plugin to delete the service. So if it is stopped then the delete service api call will fail and not be retried.

[emibcn]

Hi @reubenmiller ,
We have these tedge versions (I believe they are the latest ones in Ubuntu Noble):

• tedge/noble,now 1.7.1
• tedge-container-plugin-ng/noble,now 2.6.2

About this:

Ah wait. I just read the you stop the tedge-mapper-service! this would explain everything as the tedge-mapper-c8y provides the local c8y proxy service that is used by the container plugin to delete the service. So if it is stopped then the delete service api call will fail and not be retried.

We detected that the service was disconnected from the MQTT, even it being UP. To fix it, we re-connected the whole connection (tedge reconnect). After that, the only way to reproduce the non-updating/removing services from the cloud was by stopping the mapping service in the middle of the operation.

Either way, we would expect the final status to get into the server (somehow) once the communication channel gets fixed (or until an ACK from the server is received).

[reubenmiller]

Yeah that could definitely be an improvement, though we're still working out how to do that efficiently and robustly as we don't want to just continuously check the cloud and local state as that would increase the overall bandwidth used on the device.

[reubenmiller]

We detected that the service was disconnected from the MQTT, even it being UP.

Though I would think that this root cause should be investigated more then apply the fix which causes other non-intended side-effects.

Out of interest, what mosquitto version are you using? As there is a known bug in mosquitto > 2.0.11 to < 2.2.0 where messages can get lost.

[emibcn]

Hi again,

Though I would think that this root cause should be investigated more then apply the fix which causes other non-intended side-effects.

This is a development device with lots of test commands being executed there. It would be complicated to find out which one was the responsible.

Out of interest, what mosquitto version are you using? As there is a known bug in mosquitto > 2.0.11 to < 2.2.0 where messages can get lost.

libmosquitto1/noble,now 2.0.18-1build3 amd64 [installed,automatic]
mosquitto-clients/noble,now 2.0.18-1build3 amd64 [installed]
mosquitto/noble,now 2.0.18-1build3 amd64 [installed]

Those are the latest available versions in Ubuntu Noble. It looks like they might be affected by the known bug you just mentioned. do you have any link to the bug? Do you have any workaround?

[reubenmiller]

Hi again,

Though I would think that this root cause should be investigated more then apply the fix which causes other non-intended side-effects.

This is a development device with lots of test commands being executed there. It would be complicated to find out which one was the responsible.

Out of interest, what mosquitto version are you using? As there is a known bug in mosquitto > 2.0.11 to < 2.2.0 where messages can get lost.

libmosquitto1/noble,now 2.0.18-1build3 amd64 [installed,automatic]
mosquitto-clients/noble,now 2.0.18-1build3 amd64 [installed]
mosquitto/noble,now 2.0.18-1build3 amd64 [installed]

Those are the latest available versions in Ubuntu Noble. It looks like they might be affected by the known bug you just mentioned. do you have any link to the bug? Do you have any workaround?

We have a ticket in the thin-edge.io project which links to the mosquitto bugs, thin-edge/thin-edge.io#3185. We spurred the fix on the mosquitto repository, but it only made it into the 2.1.0 (which introduce a lot of changes so adoption is very slow). We're hoping for a mosquitto 2.0.23 bugfix release, but there isn't any sign of it.

Unfortunately there is no good workaround for the situation.

[reubenmiller]

@emibcn Though back to the original problem (the services not being deleted in the cloud after the container-group is removed by a software remove operation), can you confirm that there is some indication that the delete service api call failed in the tedge-container-plugin.service service?

If you're using systemd, then you can look for log entries around the time of the deletion (though you'll probably have to apply your own filters):

sudo journalctl -u tedge-container-plugin.service -n 1000

[emibcn]

Hi again @reubenmiller ,

can you confirm that there is some indication that the delete service api call failed in the tedge-container-plugin.service service?

Sure:

Mar 23 10:05:38 vm tedge-container[1007]: 2026/03/23 10:05:38 INFO Publishing MQTT Message. topic=te/device/main/service/tedge-agent/signal/sync_log_upload payload={} qos=0 retained=false
Mar 23 10:05:38 vm tedge-container[1007]: 2026/03/23 10:05:38 INFO Processing update request
Mar 23 10:05:38 vm tedge-container[1007]: 2026/03/23 10:05:38 INFO Found entities. total=7
Mar 23 10:05:38 vm tedge-container[1007]: 2026/03/23 10:05:38 INFO Reading containers
Mar 23 10:05:38 vm tedge-container[1007]: 2026/03/23 10:05:38 INFO Registering containers
Mar 23 10:05:38 vm tedge-container[1007]: 2026/03/23 10:05:38 INFO Updating digital twin information
Mar 23 10:05:38 vm tedge-container[1007]: 2026/03/23 10:05:38 INFO Checking for any stale services
Mar 23 10:05:38 vm tedge-container[1007]: 2026/03/23 10:05:38 INFO Removing stale service. topic-id=device/main/service/[REDACTED]
Mar 23 10:05:38 vm tedge-container[1007]: 2026/03/23 10:05:38 INFO Publishing MQTT Message. topic=te/device/main/service/[REDACTED]/status/health payload="" qos=1 retained=true
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 INFO Removing service from the cloud topic=te/device/main/service/[REDACTED]
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 INFO Deleting service by external ID. name=019a4e22-86e1-73e8-a4a6-6c9ce897aa68:device:main:service:[REDACTED]
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 Headers: map[Accept:[application/json] User-Agent:[go-client] X-Application:[go-client]]
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 Sending request: GET http://{host}/c8y/identity/externalIds/c8y_Serial/019a4e22-86e1-73e8-a4a6-6c9ce897aa68:device:main:service:[REDACTED]
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 ERROR: Request failed. Get "http://127.0.0.1:8001/c8y/identity/externalIds/c8y_Serial/019a4e22-86e1-73e8-a4a6-6c9ce897aa68:device:main:service:[REDACTED]": dial tcp 127.0.0.1:8001: connect: connection refused
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 WARN Failed to delete managed object. err="Get \"http://127.0.0.1:8001/c8y/identity/externalIds/c8y_Serial/019a4e22-86e1-73e8-a4a6-6c9ce897aa68:device:main:service:[REDACTED]\": dial tcp 127.0.0.1:8001: connect: connection refused"
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 Headers: map[Accept:[application/json] User-Agent:[go-client] X-Application:[go-client]]
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 Sending request: GET http://{host}/c8y/identity/externalIds/c8y_Serial/019a4e22-86e1-73e8-a4a6-6c9ce897aa68
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 ERROR: Request failed. Get "http://127.0.0.1:8001/c8y/identity/externalIds/c8y_Serial/019a4e22-86e1-73e8-a4a6-6c9ce897aa68": dial tcp 127.0.0.1:8001: connect: connection refused
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 WARN Could not lookup device's managed object by its external id. err="Get \"http://127.0.0.1:8001/c8y/identity/externalIds/c8y_Serial/019a4e22-86e1-73e8-a4a6-6c9ce897aa68\": dial tcp 127.0.0.1:8001: connect: connection refused" externalId=019a4e22-86e1-73e8-a4a6-6c9ce897aa68
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 Headers: map[Accept:[application/json] User-Agent:[go-client] X-Application:[go-client]]
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 Sending request: GET http://{host}/c8y/inventory/managedObjects/childAdditions?pageSize=100&query=type+eq+%27c8y_Service%27+and+%28serviceType+eq+%27container%27+or+serviceType+eq+%27container-group%27%29
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 ERROR: Request failed. Get "http://127.0.0.1:8001/c8y/inventory/managedObjects/childAdditions?pageSize=100&query=type+eq+%27c8y_Service%27+and+%28serviceType+eq+%27container%27+or+serviceType+eq+%27container-group%27%29": dial tcp 127.0.0.1:8001: connect: connection refused
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 WARN Could not delete orphaned cloud services. err="Get \"http://127.0.0.1:8001/c8y/inventory/managedObjects/childAdditions?pageSize=100&query=type+eq+%27c8y_Service%27+and+%28serviceType+eq+%27container%27+or+serviceType+eq+%27container-group%27%29\": dial tcp 127.0.0.1:8001: connect: connection refused"
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 INFO Publishing MQTT Message. topic=te/device/main/service/tedge-agent/signal/sync_log_upload payload={} qos=0 retained=false
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 INFO Processing update request
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 INFO Found entities. total=6
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 INFO Reading containers
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 INFO Registering containers
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 INFO Updating digital twin information
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 INFO Checking for any stale services
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 Headers: map[Accept:[application/json] User-Agent:[go-client] X-Application:[go-client]]
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 Sending request: GET http://{host}/c8y/identity/externalIds/c8y_Serial/019a4e22-86e1-73e8-a4a6-6c9ce897aa68
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 ERROR: Request failed. Get "http://127.0.0.1:8001/c8y/identity/externalIds/c8y_Serial/019a4e22-86e1-73e8-a4a6-6c9ce897aa68": dial tcp 127.0.0.1:8001: connect: connection refused
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 WARN Could not lookup device's managed object by its external id. err="Get \"http://127.0.0.1:8001/c8y/identity/externalIds/c8y_Serial/019a4e22-86e1-73e8-a4a6-6c9ce897aa68\": dial tcp 127.0.0.1:8001: connect: connection refused" externalId=019a4e22-86e1-73e8-a4a6-6c9ce897aa68
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 Headers: map[Accept:[application/json] User-Agent:[go-client] X-Application:[go-client]]
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 Sending request: GET http://{host}/c8y/inventory/managedObjects/childAdditions?pageSize=100&query=type+eq+%27c8y_Service%27+and+%28serviceType+eq+%27container%27+or+serviceType+eq+%27container-group%27%29
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 ERROR: Request failed. Get "http://127.0.0.1:8001/c8y/inventory/managedObjects/childAdditions?pageSize=100&query=type+eq+%27c8y_Service%27+and+%28serviceType+eq+%27container%27+or+serviceType+eq+%27container-group%27%29": dial tcp 127.0.0.1:8001: connect: connection refused
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 WARN Could not delete orphaned cloud services. err="Get \"http://127.0.0.1:8001/c8y/inventory/managedObjects/childAdditions?pageSize=100&query=type+eq+%27c8y_Service%27+and+%28serviceType+eq+%27container%27+or+serviceType+eq+%27container-group%27%29\": dial tcp 127.0.0.1:8001: connect: connection refused"
Mar 23 10:05:40 vm tedge-container[1007]: 2026/03/23 10:05:40 INFO Publishing MQTT Message. topic=te/device/main/service/tedge-agent/signal/sync_log_upload payload={} qos=0 retained=false

[reubenmiller]

I've created a feature improvement issue, #182 which we'll see what we can do.

[emibcn]

Hi again,

We have a ticket in the thin-edge.io project which links to the mosquitto bugs, thin-edge/thin-edge.io#3185. We spurred the fix on the mosquitto repository, but it only made it into the 2.1.0 (which introduce a lot of changes so adoption is very slow). We're hoping for a mosquitto 2.0.23 bugfix release, but there isn't any sign of it.

We install mosquitto only because tedge depends on it, so we don't use it for anything else. If tedge works well with the newer version and if we manage to install this newer version in Ubuntu Noble, tedge will work without issues, right?

If yes, we could try that path, as well. Any recommended version?

[reubenmiller]

Hi again,

We have a ticket in the thin-edge.io project which links to the mosquitto bugs, thin-edge/thin-edge.io#3185. We spurred the fix on the mosquitto repository, but it only made it into the 2.1.0 (which introduce a lot of changes so adoption is very slow). We're hoping for a mosquitto 2.0.23 bugfix release, but there isn't any sign of it.

We install mosquitto only because tedge depends on it, so we don't use it for anything else. If tedge works well with the newer version and if we manage to install this newer version in Ubuntu Noble, tedge will work without issues, right?

If yes, we could try that path, as well. Any recommended version?

mosquitto 2.1.0 should work for you, however you'd have to compile it yourself (which can be a bit more complicated)

[reubenmiller]

I've created an initial PR. #183, though I'm still testing it.

Just for our verification, can you confirm how you're stopping the tedge-mapper-c8y service during the processing? Is there a workflow you can provide so I can add it to the systems test to ensure everything works as expected?

[emibcn]

Hi again @reubenmiller ,

Just for our verification, can you confirm how you're stopping the tedge-mapper-c8y service during the processing?

Using SystemD tools:

systemctl stop tedge-mapper-c8y.service

Is there a workflow you can provide so I can add it to the systems test to ensure everything works as expected?

Hmmm, it involves steps in the web mixed with the stopping step above:

1. UI: Assign a software to the device.
2. UI: Wait for it to be fully installed.
3. UI: Remove the software from the device.
4. Fast: stop the service in the device:

   systemctl stop tedge-mapper-c8y.service

5. Watch the device logs until the error shows up.
6. Restart the service anytime: it will never re-send the message:

   systemctl start tedge-mapper-c8y.service

[reubenmiller]

Hi again @reubenmiller ,

Just for our verification, can you confirm how you're stopping the tedge-mapper-c8y service during the processing?

Using SystemD tools:

systemctl stop tedge-mapper-c8y.service

Is there a workflow you can provide so I can add it to the systems test to ensure everything works as expected?

Hmmm, it involves steps in the web mixed with the stopping step above:

1. UI: Assign a software to the device.
2. UI: Wait for it to be fully installed.
3. UI: Remove the software from the device.
4. Fast: stop the service in the device:
   systemctl stop tedge-mapper-c8y.service
5. Watch the device logs until the error shows up.
6. Restart the service anytime: it will never re-send the message:
   systemctl start tedge-mapper-c8y.service

ah I thought you were stopping the tedge-mapper-c8y.service within a workflow or something...but I'll try the manual approach for now.