Add update apply script and agent guide

2026-07-15 01:02:43 +02:00
parent 7a4c561f8e
commit 398c1ae6c3
2 changed files with 304 additions and 2 deletions
@@ -1,9 +1,219 @@
-# Agent notes
+# Agent notes for PVE Cloud Backup
+
+This project is a directly installed Proxmox backup web app. It runs from:
+
+```text
+/opt/pve-cloud-backup/
+  backend/    FastAPI app, worker, SQLite migrations, tests
+  frontend/   Vue 3 + TypeScript + Tailwind source
+  static/     generated frontend build; ignored by git
+  data/       runtime SQLite database; ignored by git
+  logs/       runtime logs; ignored by git
+  scripts/    install/update/uninstall/manual integration scripts
+  systemd/    source copies of service units
+```
+
+The live services are:
+
+- `pve-cloud-backup-web.service`
+- `pve-cloud-backup-worker.service`
+
+The web app listens on `0.0.0.0:8080`.
+
+## Core rules

 - Keep runtime configuration in SQLite only. Do not add `.env`, YAML, JSON, TOML, or other runtime config files.
+- Do not commit `/opt/pve-cloud-backup/data/`, `/logs/`, `/static/`, `.venv`, `node_modules`, caches, or generated artifacts.
 - Do not use Docker or add distributed-worker infrastructure.
 - Do not call `vzdump` directly. Use `pvesh` for backup operations.
 - Do not require a crypt remote. Plain OneDrive remotes and crypt remotes are both allowed.
- Do not use `rclone sync`. Use exact-object operations: `copyto`, existence check, and `deletefile`.
+- Do not use `rclone sync`.
+- Use exact-object rclone operations:
+  - upload with `copyto`;
+  - verify with `lsjson`;
+  - delete files with `deletefile`;
+  - remove empty per-backup folders with `rmdir`.
 - Never delete a local archive unless upload completed and the remote object was verified.
 - Mock `pvesh` and `rclone` in automated tests. Manual integration testing is limited to `scripts/integration-test-adguard.sh`.
+
+## Runtime data
+
+SQLite database:
+
+```text
+/opt/pve-cloud-backup/data/app.db
+```
+
+Current important settings live in SQLite, not files. Common live values on this node have included:
+
+- Proxmox node: `astrid`
+- Proxmox storage: `hitachi`
+- local dump path: `/mnt/pve/hitachi/dump`
+- rclone remote: `onedrive`
+- remote path: `pve-cloud-backup`
+- timezone: `Europe/Berlin`
+- manual test guest: `adguard`, VMID `110`, type `lxc`
+
+Do not hardcode those into normal app behavior. They are local operational context.
+
+## Backend map
+
+- `backend/app/main.py`
+  - FastAPI app, API routes, setup/settings validation, static frontend serving.
+- `backend/app/backup_service.py`
+  - Backup state machine, upload/delete/retention/recovery logic.
+- `backend/app/commands.py`
+  - Shell wrappers for `pvesh` and `rclone`.
+- `backend/app/jobs.py`
+  - Cron validation, timezone-aware next-run calculation, job CRUD helpers.
+- `backend/app/migrations.py`
+  - SQLite schema and default settings.
+- `backend/app/settings_store.py`
+  - SQLite settings serialization/parsing.
+- `backend/app/worker.py`
+  - Scheduler/worker entrypoint.
+- `backend/tests/`
+  - Unit tests. Keep external command usage mocked.
+
+## Frontend map
+
+- `frontend/src/App.vue`
+  - Single-file UI for setup, dashboard, guests, jobs, history, backup details, and settings.
+- `frontend/src/main.ts`
+  - Vue mount entrypoint.
+- `frontend/src/style.css`
+  - Tailwind/global styling.
+
+The built frontend goes to `/opt/pve-cloud-backup/static/` via `npm run build`; do not commit that directory.
+
+## Backup workflow
+
+Expected states:
+
+- `queued`
+- `pve_running`
+- `local_ready`
+- `uploading`
+- `remote_ready`
+- `local_deleting`
+- `completed`
+- `failed`
+- `deleting`
+- `deleted`
+
+Important behavior:
+
+- The worker atomically claims queued backups before starting Proxmox work.
+- `pve_running` must have a real Proxmox UPID before polling task status.
+- Local archive discovery must ignore `.log` and `.notes`; only real archive suffixes count.
+- Remote path shape is:
+
+```text
+<rclone_remote>:<rclone_remote_path>/<backup_id>/<archive_name>
+```
+
+- Deleting a backup removes the exact remote file and then attempts to remove the empty `<backup_id>` folder.
+- Deleting a backup already in `deleted` state purges the SQLite metadata row.
+- Deleting a job deletes all known non-active backups for that job first, then removes the job.
+- Retention runs after backup completion and hourly from the worker.
+
+## Applying updates after pulling changes
+
+After a `git pull`, apply the new code with:
+
+```bash
+cd /opt/pve-cloud-backup
+./scripts/apply-update.sh
+```
+
+The script:
+
+1. backs up `data/app.db` to `data/update-backups/`;
+2. stops web/worker services;
+3. creates/updates `backend/.venv`;
+4. installs backend requirements;
+5. optionally runs backend tests if `RUN_TESTS=1`;
+6. runs `npm ci` or `npm install`;
+7. builds frontend assets into `static/`;
+8. runs SQLite migrations;
+9. installs systemd units from `systemd/`;
+10. reloads systemd;
+11. restarts and health-checks services.
+
+Use this stricter variant when practical:
+
+```bash
+cd /opt/pve-cloud-backup
+git pull --ff-only
+RUN_TESTS=1 ./scripts/apply-update.sh
+```
+
+If the health check fails, inspect:
+
+```bash
+journalctl -u pve-cloud-backup-web.service -u pve-cloud-backup-worker.service --no-pager -n 100
+```
+
+## Validation commands
+
+Backend tests:
+
+```bash
+cd /opt/pve-cloud-backup/backend
+./.venv/bin/pytest
+```
+
+Frontend typecheck/build:
+
+```bash
+cd /opt/pve-cloud-backup/frontend
+npm run build
+```
+
+Service status:
+
+```bash
+systemctl is-active pve-cloud-backup-web.service pve-cloud-backup-worker.service
+```
+
+API smoke checks:
+
+```bash
+curl -fsS http://127.0.0.1:8080/api/setup/status
+curl -fsS http://127.0.0.1:8080/api/jobs | python3 -m json.tool
+curl -fsS http://127.0.0.1:8080/api/backups | python3 -m json.tool
+```
+
+Manual integration test:
+
+```bash
+API_URL=http://127.0.0.1:8080 /opt/pve-cloud-backup/scripts/integration-test-adguard.sh
+```
+
+Only run the manual integration test when the user explicitly wants a real Proxmox/rclone backup test.
+
+## Git workflow
+
+Remote:
+
+```text
+origin ssh://gitea@gitea.reversed.dev:2222/space/pve-cloud-backups.git
+```
+
+Default branch:
+
+```text
+main
+```
+
+Before committing, check:
+
+```bash
+git status --short --ignored
+```
+
+Confirm ignored runtime files stay ignored:
+
+```bash
+git check-ignore -v data/app.db logs/worker.log static/index.html frontend/node_modules/.package-lock.json backend/.venv/pyvenv.cfg
+```
@@ -0,0 +1,92 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+INSTALL_DIR="/opt/pve-cloud-backup"
+BACKEND_DIR="$INSTALL_DIR/backend"
+FRONTEND_DIR="$INSTALL_DIR/frontend"
+STATIC_DIR="$INSTALL_DIR/static"
+DATA_DIR="$INSTALL_DIR/data"
+LOG_DIR="$INSTALL_DIR/logs"
+BACKUP_DIR="$DATA_DIR/update-backups"
+WEB_SERVICE="pve-cloud-backup-web.service"
+WORKER_SERVICE="pve-cloud-backup-worker.service"
+
+if [[ "$(id -u)" -ne 0 ]]; then
+  echo "Run as root. The services need pvesh access and systemd control." >&2
+  exit 1
+fi
+
+if [[ ! -d "$INSTALL_DIR" ]]; then
+  echo "$INSTALL_DIR does not exist." >&2
+  exit 1
+fi
+
+command -v python3 >/dev/null || { echo "python3 is required" >&2; exit 1; }
+command -v npm >/dev/null || { echo "npm is required to build the Vue frontend" >&2; exit 1; }
+
+cd "$INSTALL_DIR"
+
+echo "Applying PVE Cloud Backup update from $INSTALL_DIR"
+
+mkdir -p "$BACKEND_DIR" "$FRONTEND_DIR" "$STATIC_DIR" "$DATA_DIR" "$LOG_DIR" "$BACKUP_DIR"
+
+if [[ -f "$DATA_DIR/app.db" ]]; then
+  stamp="$(date -u +%Y%m%dT%H%M%SZ)"
+  cp -a "$DATA_DIR/app.db" "$BACKUP_DIR/app.db.$stamp"
+  echo "Backed up SQLite database to $BACKUP_DIR/app.db.$stamp"
+fi
+
+echo "Stopping services if they are running..."
+systemctl stop "$WORKER_SERVICE" 2>/dev/null || true
+systemctl stop "$WEB_SERVICE" 2>/dev/null || true
+
+echo "Updating Python virtual environment..."
+if [[ ! -x "$BACKEND_DIR/.venv/bin/python" ]]; then
+  python3 -m venv "$BACKEND_DIR/.venv"
+fi
+"$BACKEND_DIR/.venv/bin/pip" install --upgrade pip
+"$BACKEND_DIR/.venv/bin/pip" install -r "$BACKEND_DIR/requirements.txt"
+
+if [[ "${RUN_TESTS:-0}" == "1" ]]; then
+  echo "Running backend tests because RUN_TESTS=1..."
+  (cd "$BACKEND_DIR" && "$BACKEND_DIR/.venv/bin/pytest")
+fi
+
+echo "Installing frontend dependencies and building static assets..."
+cd "$FRONTEND_DIR"
+if [[ -f package-lock.json ]]; then
+  npm ci
+else
+  npm install
+fi
+npm run build
+
+echo "Running database migrations..."
+cd "$BACKEND_DIR"
+"$BACKEND_DIR/.venv/bin/python" -c 'from app.migrations import run_migrations; run_migrations()'
+
+echo "Installing systemd service units..."
+install -m 0644 "$INSTALL_DIR/systemd/$WEB_SERVICE" "/etc/systemd/system/$WEB_SERVICE"
+install -m 0644 "$INSTALL_DIR/systemd/$WORKER_SERVICE" "/etc/systemd/system/$WORKER_SERVICE"
+systemctl daemon-reload
+systemctl enable "$WEB_SERVICE" "$WORKER_SERVICE" >/dev/null
+
+echo "Starting services..."
+systemctl restart "$WEB_SERVICE"
+systemctl restart "$WORKER_SERVICE"
+
+echo "Checking service state..."
+systemctl is-active "$WEB_SERVICE" "$WORKER_SERVICE"
+
+echo "Checking API health..."
+for _ in {1..20}; do
+  if curl -fsS http://127.0.0.1:8080/api/setup/status >/dev/null; then
+    echo "Update applied successfully."
+    exit 0
+  fi
+  sleep 1
+done
+
+echo "Services started, but API health check did not respond in time." >&2
+echo "Inspect with: journalctl -u $WEB_SERVICE -u $WORKER_SERVICE --no-pager -n 100" >&2
+exit 1