Skip to content

Git Kit

id: git · sections: Monitoring, Sources, Found, Projects, Schedules, Copies, Verification, Moves, Reports · state: first release.

Copies of git repositories, and moving them to another host. The reason for the kit: "a git host is a single point of failure for everything an estate knows how to build, and the copy most people have of it is a working copy of one branch on somebody's laptop."

What it does

Sources: what the host holds, and what of it is copied

A source is a GitLab or GitHub the kit may ask what it holds — the whole host, or one group or organisation. The kit writes the answer down and compares it with what it copies, and the difference is the number the source exists for: forty-seven there, thirty-one copied, sixteen without a copy. That number is a metric, so an alert can be built on the repository created last Tuesday that nobody added. Found is the same list across every source.

  • Taken on in bulk, with the settings chosen once on the source; one batch can override the storage or the way in. A source can also take new repositories on by itself as they appear.
  • The token is asked before it is saved. "what can it do" comes back as rights the kit needs, rights that help, and rights the token carries that the kit never uses. A GitHub fine-grained token reports no scopes at all, and that is said as "the host does not say", not as "the token has nothing".
  • A host is looked at hourly by itself, or on a cadence somebody set. Forgetting a source forgets the host, not the repositories: every one taken on from it stays a project.

The copy

  • A mirror, not a clone: every branch, tag and note, and it can be pushed back onto an empty host. octocat/Hello-World has 3 branches and 3569 refs; a clone of the default branch would have one.
  • What the copy lacks is part of the report, in the headline itself. LFS (a mirror without it holds every pointer and none of the files) and submodules (a restore lands an empty directory) are asked about whether or not they are switched on. A copy may be incomplete; it may not look complete.
  • Two ways in: https with a token, ssh with a key. A token never lands in a URL — it reaches git through GIT_ASKPASS in a directory that lives for one fetch. Plain http is refused, and so is an address with a credential inside it (https://user:token@…): the token goes into a secret, the address stays clean. One saved before this was refused keeps working and is shown as https://***@….
  • The kit's own key or a person's. On request the kit makes an ed25519 pair and shows the public half to add as a deploy key; the private half goes into the platform's secrets and is returned by nothing. A person's own key is allowed, and the screen says what that costs.
  • Host keys are learned once and then required. A different key later is a refusal; a rebuilt host is forgotten on purpose, from the project.
  • Check is one ls-remote, not a fetch, and its refusal is named: credential, address, network or host key.

Archives and verification

  • A bundle, a tar or both, chosen per project. A bundle is git's own format and can be verified; a tar keeps the mirror byte for byte, hooks and LFS objects included. Either goes into a platform storage as a stream, compressed and encrypted, under the platform's retention.
  • Verification reads the copy back out of the storage, unpacks it, recomputes every object (git fsck) and compares every ref with what the mirror held when the archive was cut. git bundle verify alone only says the file agrees with itself. A copy nobody has read back is shown as an unasked question, not as a failure.
  • The whole round — copy, archive, verify — is one job, and one cadence can name a whole host: a hundred repositories are one schedule, not three hundred. One repository failing does not stop the others.
  • The kit offers retention templates of its own (nightly, weekly, a year of months) rather than a database's hourly ones.

Moves: repositories to another host

A move puts repositories into another GitLab, GitHub, Gitea or a plain git address, so that people work there from now on.

  • Into repositories that exist, or made on GitLab on the first push, each keeping its whole path under the group named: backend/api becomes team/backend/api, and the subgroups on the way are made, private.
  • Checked ref by ref after the push. "git push returned zero" and "the other side has these branches" are different statements; a repository that arrived short is failed, and the list says what is missing.
  • In pieces when a proxy refuses the size. A push refused with HTTP 413 — an nginx client_max_body_size in front of the other host — goes over ref by ref, and a ref that is too large by its history in steps that halve until they fit. Only a single commit larger than the limit cannot be helped, and the refusal names the commit.
  • Resumable, with a log. Carry on takes what is not over yet; Go over everything again is for a far side that was redone. Every try of every repository stays in the log.
  • Synchronised on a schedule if wanted: each run fetches every repository from the old host and pushes it to the new one.
  • Nothing is deleted anywhere: the source is left as it is, and removing on the far side what is gone here is a switch that is off.

Where a secret is used

The platform's Secrets screen asks the kit where each secret is written down — which projects, sources and moves name it, and which ssh projects depend on the kit's own key — so deleting one says what stops working before it does.

What it needs

  • A git host reachable from the kit over https or ssh, and a credential kept as a platform secret — the kit's tables hold the name, never the value. A move needs a second one that may write: the token that reads a backup is never widened into one that writes.
  • The git-mirrors volume (/srv/mirrors): the mirrors are the copy and cannot be rebuilt from the database.
  • A platform storage for archives, named on the project when there is more than one: the kit does not pick.

What it measures

Metric Help
projects_total Repositories this kit keeps a copy of
copy_age_seconds Age of the last fetch that finished — or, where none ever has, how long the project has gone without one
copy_failures_24h, copies_24h Fetches that stopped part way, over a day; fetches finished over the last day
copy_seconds How long the last fetch took
copy_incomplete Things the repository has that this copy does not — LFS data, submodules. Zero means the copy is whole
mirror_bytes, archive_bytes What a project's copy occupies as folders, and what its newest archives occupy in the storage
archive_verified Whether the last bundle passed git bundle verify; no series for a tar
refs_total Branches, tags and notes in the copy
source_repositories, source_copied, source_uncopied What the host held at the last look, how much of it is copied, and how much nothing is copying
source_look_age_seconds How long since the host was last asked

What it refuses to do, and why

  • Delete the copy when a project is forgotten, unless asked in so many words — and never from under a job that is using it.
  • Fetch two repositories at once, or more than 32 submodules in a pass.
  • Archive what was never fetched: an empty file registered as a copy is the one failure a backup kit must not have.
  • Replace its own ssh key with a second click: every project using it would stop until the new key is on every host.
  • Move LFS objects: they do not travel with a push, and moving the rest would leave pointers where the files were. Such a repository is refused.
  • Promise what it does not do: moving issues and merge requests, closing or archiving the source after a move, and making repositories on GitHub or Gitea are refused when the move is planned.

Known limitations

  • A move carries the mirror only — branches, tags and notes. Issues, merge requests, releases and CI variables stay where they are.
  • Repositories are made on the far side only on GitLab; on GitHub and Gitea they must exist first.
  • No restore button. Restore is git clone from the downloaded bundle, or unpacking the tar.
  • A submodule on another host is reported as not copied: it is taken with the parent's credential.
  • LFS is the least exercised part: the fetch path exists, but no LFS-heavy repository has been behind it.

Free level

One source and ten of its repositories, backed up whole: copied on a schedule, archived into a storage, read back and verified, moved and synchronised — every feature of the kit is free. A small estate lives on one GitLab, and the free level is meant to back it up rather than demonstrate that it could. A key buys scale: a second source, the eleventh repository.

From the manifest: mirror, sources, schedules and archive are all free; sources: 1, projects: 10.

Configuration

AG_GIT_MIRRORS (/srv/mirrors), AG_GIT_PARALLEL (1), AG_GIT_FETCH_TIMEOUT (3600), AG_GIT_ARCHIVE_TIMEOUT (3600), AG_GIT_MEASURE (300), AG_GIT_LOOK (3600; 0 turns the hourly look off), AG_GIT_FORGE_TIMEOUT (30), AG_GIT_COMPRESS (1; 0 turns compression off), AG_GIT_PORT (8107) — in kits/git/app/config.py; AG_GIT_QUIET_WAIT (300) in kits/git/app/worker.py. The shared git library reads AG_GIT_LFS_TIMEOUT (3600), AG_GIT (git) and AG_KEYGEN (ssh-keygen).