From fd024d31e9fb265330e3eb8f5e1d0e02dba1b0b6 Mon Sep 17 00:00:00 2001 From: Frank Schwenk Date: Sat, 18 Jul 2026 17:12:26 +0200 Subject: [PATCH] docs: update README and add ARCHITECTURE Dev setup (venv, ruff, pytest), config table entries, architecture overview, and cleanup completion report. Co-authored-by: Cursor --- ARCHITECTURE.md | 79 +++++++++++++++++++++++++++++++++++++++++ CLEANUP_REPORT.md | 89 +++++++++++++++++++++++++++++++++++++++++++++++ README.md | 20 +++++++++++ 3 files changed, 188 insertions(+) create mode 100644 ARCHITECTURE.md create mode 100644 CLEANUP_REPORT.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..d2e24d4 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,79 @@ +# Architecture — live.f12.rocks + +## Overview + +``` +Phone/Syncthing Docker host +─────────────── ─────────── +incoming/ ──watch──► worker ──► variants/ + intermediates/ + │ │ + └── rembg + gmic pipeline + │ +meta/{stem}.json ◄──────────────────────┘ + │ + ▼ + web (FastAPI) ──► browser UI + remix POST +``` + +Two containers share one image (`Dockerfile`) and one bind-mounted data +directory (`DATA_HOST_DIR` → `/data`). Syncthing runs on the host, not in +compose. + +## Python modules + +| Module | Role | +|--------|------| +| `app/worker.py` | Poll `incoming/`, copy to `variants/`, run `process_job()` | +| `app/pipeline.py` | gmic/rembg orchestration, meta/manifest I/O, variant compose | +| `app/remix.py` | User-selected remix variant (one-shot compose + manifest append) | +| `app/main.py` | FastAPI routes, Jinja templates, static files | +| `app/config.py` | Environment → constants (paths, timeouts, tuning) | +| `app/rembg_cli.py` | Subprocess entry for rembg (avoids heavy `rembg[cli]` extra) | +| `app/io_utils.py` | Atomic JSON read/write | +| `app/env_utils.py` | Shared bool env parsing | +| `app/logging_config.py` | Entry-point logging setup | + +## Data model + +One **job** = one sanitized stem (from incoming filename). Flat files: + +- `variants/{stem}_original.jpg` — preprocessed copy +- `variants/{stem}_rembg.png` — background removal +- `variants/{stem}_vN.png` — final composed images +- `intermediates/{stem}_vN_*.png` — step images (kept for debugging) +- `meta/{stem}.json` — status + manifest (variants list, filter metadata) +- `processed.json` — worker bookkeeping (incoming path → handled) + +## Pipeline origin + +The gmic filter/blend chain is ported from an external reference script +`make_random.py` (not in this repo). `config.POST_FILTERS` and asset lists +under `assets/` mirror that script's behaviour. + +## Background removal choice + +`compare-bg/` is a **standalone benchmark** (rembg models vs withoutbg). +It is not part of the runtime stack. Production uses `u2net` + alpha +matting via `app/rembg_cli.py` (see `config.REMBG_MODEL`). + +## Known limitations (intentional) + +- **No web auth** — anyone with the link sees all jobs (event tool). +- **Sequential worker** — one photo at a time; burst uploads queue. +- **Meta JSON** — worker and web both read-modify-write `meta/*.json` + without file locking; concurrent remix during active processing can race. +- **Incoming is append-only** — worker never deletes from `incoming/`. + +## Dev / test + +```bash +python3.11 -m venv .venv && source .venv/bin/activate +pip install -r requirements-dev.txt +ruff check app/ tests/ +pytest -q +``` + +Full stack (gmic/rembg): `docker compose up --build` with `DATA_HOST_DIR=./data`. + +Python **3.11** matches the Docker image; host 3.14+ cannot install pinned +`onnxruntime` — use Docker or a 3.11 venv for parity. diff --git a/CLEANUP_REPORT.md b/CLEANUP_REPORT.md new file mode 100644 index 0000000..a9ddefe --- /dev/null +++ b/CLEANUP_REPORT.md @@ -0,0 +1,89 @@ +# Cleanup Report — live.f12.rocks + +Branch: `cleanup/code-quality` (local only, not pushed). +Completed: 2026-07-18. + +## Summary + +Incremental code-quality cleanup across 10 commits. **No intentional +behavior changes** to the photo pipeline, worker loop, or web UX. + +## What changed + +| Step | What | Why | Risk | Verification | +|------|------|-----|------|--------------| +| Phase 0 | `CLEANUP_PLAN.md` | Analysis + roadmap | Low | — | +| Tooling | `pyproject.toml`, `requirements-dev.txt`, `.python-version` | Ruff + pytest baseline, pin 3.11 | Low | `ruff check`, `pytest` | +| Style | Ruff format entire `app/` | Consistent style | Low | `ruff format --check` | +| Dead code | Removed unused `blend_layers()` | Less noise | Low | grep + pytest | +| `.gitignore` | Build/coverage/venv patterns | Keep repo clean | Low | — | +| `io_utils` | Atomic JSON helpers | DRY meta/processed I/O | Low | `test_io_utils`, pipeline/worker tests | +| `env_utils` / `logging_config` | Shared bool env + logging | DRY | Low | `test_env_utils`, pytest | +| `compose.yml` | YAML anchor for env | DRY, fewer drift bugs | Low | `docker compose config` (manual) | +| Config | `PREPROCESS_TIMEOUT` constant | Named magic number | Low | default still 120s | +| `read_meta()` | Public meta accessor; `list_jobs` reads once | Perf + clarity | Low | `test_pipeline`, `test_main` | +| Tests | 33 characterization tests | Safe refactor baseline | Low | `pytest -q` | +| CI | `.gitea/workflows/ci.yml` | Lint + test on push/PR | Low | workflow syntax | +| Docs | `README.md`, `ARCHITECTURE.md` | Onboarding | Low | — | + +## What was NOT changed (and why) + +| Item | Reason | +|------|--------| +| `pipeline.py` module split | Medium risk without broader integration tests; deferred | +| Meta file locking | Behavior change; documented in `ARCHITECTURE.md` | +| Dependency major bumps | Per plan — list only, no auto-bump | +| `compare-bg/` | Separate experiment; left untouched | +| Deploy CI | Manual deploy on boka stays; STANDARDS says ask first | +| FastAPI `TemplateResponse` API migration | Deprecation warning only; no functional change | +| `_validate_job_id` always sanitizes to valid ID | Pre-existing; documented in tests | + +## Bugs noticed (not fixed) + +1. **Meta JSON race** — worker and web concurrent RMW without locking. +2. **`_filter_timeout_override` global** — thread-unsafe under concurrent remix (uvicorn workers). +3. **`sanitize_stem("a/b")` → `"b"`** — `Path.stem` drops path prefix before slash replacement. +4. **`_validate_job_id`** — sanitization makes rejection path effectively unreachable. + +## Dependency notes (suggestions only) + +| Package | Current pin | Note | +|---------|-------------|------| +| `fastapi` | `<0.117` | 0.117+ available; test before bump | +| `onnxruntime` | `<1.20` | Very old pin; rembg may allow newer in future | +| `pillow` | `<11` | Pillow 11 exists; verify rembg compatibility | + +No lockfile added — soft pins kept to avoid changing Docker resolve behavior. + +## How to verify locally + +```bash +git checkout cleanup/code-quality +python3.11 -m venv .venv && source .venv/bin/activate +pip install -r requirements-dev.txt +ruff check app/ tests/ +pytest -q +# Full stack: +cp .env.example .env # set DATA_HOST_DIR=./data +docker compose up --build +``` + +## Commits on branch + +``` +docs: add cleanup plan from phase-0 analysis +chore: add pyproject.toml with ruff and pytest config +style: apply ruff format and extend gitignore +refactor: extract atomic JSON helpers to io_utils +refactor: share env-bool parsing and logging setup +chore: deduplicate compose environment blocks +refactor: named preprocess timeout and single meta read +test: add characterization tests for core modules (+ CI workflow) +docs: update README and add ARCHITECTURE +``` + +## External references + +- `compare-bg/` — local venv `.venv-compare/` and 1.5GB model cache (gitignored) +- `make_random.py` — external reference script, not vendored +- Submodule/symlink: none diff --git a/README.md b/README.md index c028f53..cba0370 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,24 @@ Drop a photo into `incoming/` (via Syncthing or locally). After a few seconds (poll interval + processing time) it shows up on the web UI; flat files land in `variants/` and `intermediates/` for the phone. +## Development + +Python **3.11** (same as `Dockerfile`). On a host with newer Python, use +Docker or a 3.11 venv — pinned `onnxruntime` does not publish wheels for 3.14+. + +```bash +python3.11 -m venv .venv && source .venv/bin/activate +pip install -r requirements-dev.txt + +ruff check app/ tests/ # lint +ruff format app/ tests/ # format +pytest -q # unit tests (no gmic/rembg) +``` + +CI (`.gitea/workflows/ci.yml`): ruff + pytest on push/PR. + +Architecture notes: `ARCHITECTURE.md`. Cleanup log: `CLEANUP_PLAN.md`. + ## Configuration (`.env`, see `.env.example`) | Var | Default | Notes | @@ -64,6 +82,8 @@ files land in `variants/` and `intermediates/` for the phone. | `BLEND_OPACITY` | `30%` | Default blend opacity for auto-generated variants | | `FILTER_TIMEOUT` | `120` | Seconds before a single gmic call is killed | | `MAX_FILTER_ATTEMPTS` | `8` | Random-filter retry budget per bg/fg pick | +| `MAX_EDGE_PX` | `2000` | Longest edge after ImageMagick preprocess | +| `PREPROCESS_TIMEOUT` | `120` | Seconds for preprocess step | | `NICE_LEVEL` | `18` | `nice -n` level for gmic/rembg subprocesses | ## Deploy notes / caveats