Proxmox Kit¶

id: proxmox · sections: Datacenter, Nodes, Machines, Storage, Topology, Observations, Actions, Configuration · state: first prototype, verified on a live cluster.

Watching Proxmox VE clusters over their HTTP API, and — if the owner adds a second token — switching guests on and off. The reason for the kit is one and practical: Proxmox knows everything about its own installation and answers almost none of an administrator's questions whole. "Which machines have no copy", "what fills this storage", "what dies with this node", "which guest was working hardest when that node went red" — each of them lies across two or three of its own screens, and therefore has no answer at all.

What it does¶

One request describes the cluster¶

GET /cluster/resources on any node returns the nodes, the guests, the storages and the networks of the whole cluster at once, in hundredths of a second. Hence the main decision: read everything, every time, rather than invent economies — a full pass costs about a second, and no selective polling would repay the complexity.

A connection is a cluster, not a node: one name, one token, several addresses. Any node answers for the whole cluster while it is alive, so the connection walks its addresses until somebody answers, and the one that answered is tried first next time. Otherwise the monitoring goes blind at exactly the moment it is needed.

Two intervals, and history in three steps¶

A cluster is asked every five minutes when nothing is wrong and every minute when something is — both settings of the connection, and the switch can be turned off. "The five minutes after a node falls over are the only five minutes anybody looks at." What counts as alarmed is the same written list that raises alerts, and the screen says why it is polling faster.

History keeps three steps: how many days as taken, how many folded to the hour, how many folded to the day, all three chosen by whoever owns the disk. Folding, not deletion — the shape of last March survives at a twelfth of the size — and a folded row carries the peak beside the average, because an hourly mean hides exactly what history is read for: a node that sat at 30 % and touched 98 % twice looks calm by the average.

Three arrangements are offered — 2/30/180, 7/90/365, 30/365/1095 days — and a fourth typed by hand, priced by the same arithmetic rather than recomputed in the browser. Zero means "this step does not exist"; ten years is the ceiling, trimmed rather than accepted in silence. The screen says what the history weighs now and what it would weigh, measured on this installation rather than taken from a constant somebody once measured elsewhere.

Backups, asked of the storages¶

The kit asks the storages, not the jobs: a job says what should happen, only the storage knows what did. The main question is "which machines have no copy at all" — Proxmox's backup screen lists jobs, so a guest created after the job was written is simply absent from a screen nobody re-reads. A fresh copy and a working job are separate facts, and so is backup=0 on one disk of a machine that is otherwise copied properly.

Dependencies and topology¶

Links are computed and stored rather than recomputed per request, so a link that disappeared between two polls is visible too, and both directions are asked: "what breaks if I reboot this node" and "what has to be fixed for this machine to come back". A local storage is one node's storage, not one storage with that name: local-zfs on pve00 and local-zfs on pve03 are two different disks, and joining them by name draws a hub the whole cluster converges on — a picture of something that does not exist. A shared storage stays one object, however many nodes report it.

The guest agent¶

Asked for names: a machine that reports its own hostname, system and addresses is one the platform recognises as already known — on the reference cluster, 21 of 27 running machines. Only reading commands are sent; the allowed list is in the code and anything outside it is refused before the network, because monitoring able to run a command in 38 machines is not monitoring. The agent is polled in a slow lane, once an hour, and what it said last time stays in force. "Set up and silent" is its own state, in words: answers, switched on, silent, not switched on — and a dash for containers, which will never have one.

Power, on the second token¶

start, shutdown and reboot are one press. stop and reset cut the power and require the machine's name typed back — the same gate the core uses for deletions, and for the same reason. The record of an action is written before the call, so a request that killed a container half way is still in the journal as "sent, outcome unknown".

A migration is offered together with what is in its way: Proxmox itself says which nodes are possible and which local disks would have to come along, and a container is not migrated while it runs — said before the press. A template has no power: it is not a machine switched off but a thing that does not run at all, so it has no Start button, though moving it is still offered. Automatic rules are deliberately narrow — one guest or one tag, one condition, one action, a floor between firings and a ceiling per day: "two firings a day is a repair; twenty is a fight with something". A rule may start, shut down or reboot; cutting the power is a switch of its own on the rule, off unless somebody turned it on, because the name a person has to type back is a name a rule supplies to itself.

Observations¶

An observation is not an alert. An alert is "broken now" and goes to the platform; an observation is "this will cost money", "that is unusual for this machine", "these two facts do not add up", and it is read on Tuesday. Each carries the numbers it was derived from. Twenty-five rules, one thought each: machines never copied; a disk excluded from the backup of a machine that has backups; copies older than the rule allows; every copy in one basket; strict HA rules with no live node left; solid-state disks near the end of their write life; a storage on course to be full, and when; certificates running out; tasks the cluster keeps failing; replication behind; a node that has promised more memory than it has; machines off for a month still holding their disks; old snapshots; detached unusedN volumes; memory a machine has never touched; machines given more cores, memory or disk than they have ever used; thin storage without discard; a kvm64 processor; which guest was busy when its node was; machines whose uptime keeps going back to nothing; security updates waiting; nodes on the no-subscription repository and the nightly apt failure that follows; a node firewall switched off; guest interfaces with no firewall at all; and privileged containers, worse with nesting on.

There are three answers to one. "Not now" is a date, not a cancellation — it comes back by itself. "This machine needs no copy" is a property of the machine, kept in a table the poll does not rewrite, and the rules stop counting it in the numbers as well as in the advice: otherwise a screen saying "25 machines without a copy" is read by somebody who knows three of them do not count, and then no number on it is believed. Such decisions are one table and a word rather than a column per idea, and they carry an author and a date. Dismissing is possible, deleting is not: an observation waved away every week is a bad rule, and that is the only way it becomes visible.

The right size for the load¶

Judged over the longest window there is history for — a year, half a year, a quarter, a month — and the window is written into the line, because a machine quiet for a month and busy once a quarter is a machine with a quarterly job. Peaks, not averages, and too little history means silence rather than advice from noise. Three resources, three lines: cores can be taken back while the machine runs, memory usually cannot, and the disk means moving the load to a smaller machine — so cores and memory are counted into what would be released, and the disk is said in words and left out of that sum.

The dashboard¶

Assembled rather than laid out: every block moves, widens and goes away, and where a person left them is where they will be. An unknown block in a saved layout is ignored and a new one is added at the end, so an upgrade adds a card rather than resetting somebody's screen.

What it needs¶

A reading token, and the check says what it can do rather than whether it worked — a token created with privilege separation on authenticates perfectly and reads nothing, which looks like an empty cluster rather than a rights problem.

Right	What it is for
`Sys.Audit`	the cluster, its nodes, their disks and their tasks	needed
`VM.Audit`	what each machine is made of	needed
`Datastore.Audit`	the storages and the copies on them	needed
`VM.GuestAgent.Audit`	asking a machine what it calls itself	helps
`VM.PowerMgmt`	starting, stopping and restarting a machine	control token
`VM.Migrate`	moving a machine to another node	control token, helps

A right held over part of the estate — one pool, one machine — is reported as partly, not rounded to yes; and what the token can do that the kit never asks of it is listed too, because the one moment anybody narrows a working token is while looking at a screen that says what it is carrying.

A second token for power, or none at all. Everything the kit does by itself is reading. An installation that never added the second token has monitoring physically unable to stop a production machine — not through a bug in this code, not through a mistake in a rule, not through whoever got a shell in a container. That is a property, not an omission.
The certificate, confirmed by a person. Proxmox signs itself and no wish will change that. There is no "do not verify" switch and there will not be one; instead the fingerprint is shown when the connection is added, a person compares it with their own Proxmox, and from then on any other certificate at that address is a refusal rather than a warning.
A secret in the platform's store. The kit's tables hold the name of an entry, never a value.

What it measures¶

Metric	Help
`nodes_total`, `nodes_online`	Nodes in the cluster, whether or not they are watched; nodes answering right now
`quorate`	Whether the cluster has quorum — 1 or 0. Without it nothing starts anywhere, however healthy the nodes look
`guests_total`, `guests_running`	Machines and containers defined; of those, running now
`node_cpu_ratio`, `node_memory_ratio`	How busy a node's processors are and how much of its memory is in use
`guest_cpu_ratio`, `guest_memory_ratio`	The same two per machine, by `vmid`. A machine that is switched off reads zero — the series stays, because a series that vanishes means "the measurement broke", which is not what "switched off" means
`storage_used_ratio`	How full a storage is — the number that decides whether anything can still be written
`ceph_ok`	Whether Ceph says `HEALTH_OK` — 1 or 0. Absent where there is no Ceph
`guests_without_copy`	Machines with no backup anywhere the kit can see — the number the kit was written for
`backup_age_seconds`	Age of the newest copy of the machine that has gone longest without one
`observations_open`	Things noticed and not dealt with, by how serious they are
`poll_age_seconds`	How long ago this cluster was last asked anything — the series that says the monitoring itself has stopped

Nodes and machines are both platform resources of kind host, told apart by what each publishes about itself: only a node carries hypervisor, only a machine carries vmid. So a node's card draws the node's own processor and memory and leads into Nodes, a machine's card draws its own and leads into Machines — it cannot be done by node name, or every machine's card would draw somebody else's hypervisor as its own.

What it refuses to do, and why¶

Act without the second token. "this kit reads with one token; starting and stopping machines needs a second."
Connect to an unconfirmed certificate. A token cannot even be checked before the fingerprint is pinned, because "unreachable" and "wrong certificate" would otherwise be the same answer.
Cut the power on one press. stop and reset need the machine's name typed back.
Start a template. "it runs nothing, so there is nothing about it to start or stop." A button that always errors teaches people not to trust buttons.
Run anything inside a guest. The guest agent can execute programs; only the named reading commands are ever sent.
Let a rule fight. Past its floor or its daily ceiling a rule stops.
Drop a person's choice when the licence cannot be read. "The core is unreachable" is not the same answer as "the free level": the choice of watched nodes is left exactly as it was. Where the ceiling is read and really is lower, the journal says how many were watched, how many are allowed and which were kept.
Hide the nodes it does not watch. An unwatched node is still in the list, still has a name, is still in the count and still in the topology. The free level takes away history, not the knowledge that a machine exists.
Let a number and a decision disagree. A machine marked as needing no copy leaves the advice and the counts at the same moment.

Known limitations¶

No creating or deleting guests. Backups of virtual machines, then creating LXC containers, are the next steps; neither exists now.
No editing configuration. The kit advises changing a processor type or switching on discard, and does not do it.
No console — that is a separate kit with a separate right.
No metrics database of its own. The history is in the platform's database and the platform draws the charts.
The "full on the 3rd" line is a straight line through a fortnight of readings — storage does not grow linearly and everybody knows it — and it is said only where the line points within three months.
Verified on one cluster — Ceph and two Proxmox Backup Server storages. Behaviour on larger estates has not been measured.

Free level¶

One cluster, two of its nodes, three automatic rules. The shape is deliberate: every feature is free — the dashboard, the observations, the guest agent, the power buttons, the whole history — and the key buys scale: how many nodes are watched and how many rules act on their own. A homelab is served whole; seven production nodes are a business.

Which two nodes are watched is the operator's choice; the kit chooses only when nothing has been chosen. Of a node nobody watches the platform knows two things — that it exists, and its name. Not its status, not its hardware, not what it carries: its machines are not listed, its local storages are not shown, no observation is made about it, nothing about it is offered for a map. Shared storages and Ceph stay: a distributed thing belongs to the cluster. History is not bounded by time on any level; what is bounded is which nodes grow one, and readings from before a node was unticked are withheld rather than thrown away.

From the manifest: monitoring, control, advice and history are all free; proxmox_clusters: 1, proxmox_nodes: 2, proxmox_rules: 3.

Configuration¶

AG_PROXMOX_TIMEOUT (20), AG_PROXMOX_POLL_CALM (300), AG_PROXMOX_POLL_ALARMED (60), AG_PROXMOX_KEEP_DAYS (365), AG_PROXMOX_FREE_CLUSTERS (1), AG_PROXMOX_FREE_NODES (2), AG_PROXMOX_FREE_RULES (3), AG_PROXMOX_CPU_PCT (80), AG_PROXMOX_MEM_PCT (90), AG_PROXMOX_STORAGE_PCT (85), AG_PROXMOX_BACKUP_AGE_HOURS (36), AG_PROXMOX_PORT (8109) — in kits/proxmox/app/config.py. The thresholds are starting points — every one of them is a setting — and the two intervals belong to the connection rather than to the installation.