Maintenance Window?

[Jeppedy]

I have a number of jobs that run through most of my Docker stacks. For that 20 minutes or so, they go offline and come back up, one at a time.
Couldn't we get a "maintenance window" for containers down, or just a "don't log issues or send alerts during a time period"?

What do you all do to work around missing this feature?

[rcourtman]

No maintenance window exists today. What comes closest is the quiet hours schedule on the Schedule tab of the alert settings, per day start and end times in your timezone, and notifications raised inside the window are held rather than sent. For containers your jobs bounce every night, that covers the recurring case. You can also disable state alerts or drop their severity per container.

An ad hoc pause, or a window scoped to specific containers rather than all notifications, does not exist yet, and I think it is a fair ask for exactly this pattern. Does your job schedule run at a fixed time each day, or does it float?

[luckman212]

Came here to ask this basically similar thing: I would like a way to control the enablement of Alerts via API. When I do maintenance on my Proxmox, shut down VMs, reboot nodes, update storage etc the alerts go crazy

I looked at the API docs but I could only find an endpoint /api/alerts/activate (no /api/alerts/deactivate ?)

Is there a way to do this? @rcourtman

[rcourtman]

v6.4.1 already supports timed maintenance through PUT /api/resources/{id}/operator-state with monitoring:write access, using maintenanceStartAt, maintenanceEndAt and maintenanceScope: "resource_and_descendants" to cover a node and its guests until expiry. This endpoint is missing from the API docs, and /api/alerts/activate is for notification activation rather than maintenance. Read the existing operator state and preserve unrelated settings when changing the maintenance fields, because PUT replaces the record, and be aware that maintenance clears matching active alerts rather than simply pausing their notifications.

[luckman212]

Thanks @rcourtman yes would be great if the docs covered this endpoint

I'll have a play with /api/resources/{id}/operator-state... so basically you're saying, iterate over all my nodes, and first GET the current values, adjust the maintenanceStartAt, maintenanceEndAt and maintenanceScope: "resource_and_descendants" fields and then PUT this back should start the maintenance window?

[rcourtman]

That sequence is correct, and you only need to update the nodes undergoing maintenance, since resource_and_descendants covers their guests. Set the start to now or slightly earlier and the end to a future time, and maintenance expires automatically. If GET returns operator_state_not_set, start with an empty record, but stop on other errors and remove any existing maintenanceRecurrence before setting a one-time window.

[luckman212]

@rcourtman awesome. I think I did it. It seems to be working!

[FabienD74]

Hi @luckman212 ,
Is it possible to "disable" all notifications and set it back ON afterward ?
I mean like the global ON/OFF switch in the GUI ?

[luckman212]

@FabienD74 I think that's what I've accomplished with my script.

The script iterates over all of the nodes in the /resources endpoint, collecting any that are of type agent or pbs. Then, it executes a GET on resources/{id}/operator-state to collect the state, and then a PUT on that same endpoint, changing the following (I use jq to generate the JSON, so the below is pseudocode):

{
  "maintenanceStartAt": "{now}",
  "maintenanceEndAt": "{now + 3600}",
  "maintenanceScope": "resource_and_descendants"
}

This creates a 1-hour maintenance window. I tested this and it appeared to work, but it was a simulation not an actual maintenance so no actual alarms were triggered. We will see during the next window if it actually succeeds.

If you want I can post my script somewhere...

[FabienD74]

@luckman212
Assuming there is no sensitive info in your script, ( URL / IP, and api key ) if you could share your script, so i can get inspired :-)
I'm also performing some automatic "apt update/ apt uprade/ reboot" on some LXC. ....
It would be great to have a bash script, so an LXC could create a maintenance window when performing a planned reboot ;-) ;-)

I'm currently looking at the api /api/resources but i don't find anything related to VM.... To me, they are just missing...

[FabienD74]

@rcourtman
If i call the api /resources without specifying any type then No agents are reported, and no VM are reported.

[FabienD74]

i created a tool to retreive API from pulse, so if i do
./test_api.sh /api/resources?type=agent,vm,system-container | jq -f jq_filter_resources.txt
I have a nice list

{
 "agent": [
   {
     "id": "agent-4b4a2815f8902fde",
     "type": "agent",
     "name": "proxmox0"
   },
   {
     "id": "agent-49615d3052f30057",
     "type": "agent",
     "name": "proxmox1"
   },
   . . . . . . . .
 "vm": [
   {
     "id": "vm-0fab1b1d7f677c5e",
     "type": "vm",
     "name": "vm-debian"
   },
   {
     "id": "vm-563d2fd41ee93e27",
     "type": "vm",
     "name": "vm-debian-01"
   },
   . . . .. .
 "lxc": [
   {
     "id": "system-container-e65f5b381725cf0c",
     "type": "system-container",
     "name": "ct-apt-cache"
   },
   {
     "id": "system-container-6905da5ffffa66f1",
     "type": "system-container",
     "name": "ct-dhcp-02"
   },

Now each system can search for it's own ID then play with /api/resources/{id}/operator-state

[luckman212]

@FabienD74 yeah I don't mind sharing my script. Hang on a minute I'll post it as a gist. It's already a shell script so you should be able to drop it right in to your workflow.

[luckman212]

@FabienD74 posted: https://github.com/luckman212/pulse-maintenance

[FabienD74]

NICE !! To achieve my goal i only had to change:
done < <(__get "resources?type=agent,vm,system-container" | jq --raw-output '.data[] | "\(.id)\t\(.type)\t\(.name)"')

Then pass as new argument and add a little IF

__maint() {
  local op=${1:-off}
  local lv_system=${2}
  while IFS=$'\t' read -r id type name ; do
  if [ "${name}" == "${lv_system}" ] ; then

PS: I just need to fine-tuned to retreive the hostname ( not FQDN, handle upper/lower cases ) ... then i can stat to deploy it!