docs: update README and add ARCHITECTURE
CI / lint-and-test (pull_request) Successful in 2m3s

Dev setup (venv, ruff, pytest), config table entries, architecture
overview, and cleanup completion report.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Frank Schwenk
2026-07-18 17:12:26 +02:00
parent c29186a227
commit fd024d31e9
3 changed files with 188 additions and 0 deletions
+79
View File
@@ -0,0 +1,79 @@
# Architecture — live.f12.rocks
## Overview
```
Phone/Syncthing Docker host
─────────────── ───────────
incoming/ ──watch──► worker ──► variants/ + intermediates/
│ │
└── rembg + gmic pipeline
meta/{stem}.json ◄──────────────────────┘
web (FastAPI) ──► browser UI + remix POST
```
Two containers share one image (`Dockerfile`) and one bind-mounted data
directory (`DATA_HOST_DIR``/data`). Syncthing runs on the host, not in
compose.
## Python modules
| Module | Role |
|--------|------|
| `app/worker.py` | Poll `incoming/`, copy to `variants/`, run `process_job()` |
| `app/pipeline.py` | gmic/rembg orchestration, meta/manifest I/O, variant compose |
| `app/remix.py` | User-selected remix variant (one-shot compose + manifest append) |
| `app/main.py` | FastAPI routes, Jinja templates, static files |
| `app/config.py` | Environment → constants (paths, timeouts, tuning) |
| `app/rembg_cli.py` | Subprocess entry for rembg (avoids heavy `rembg[cli]` extra) |
| `app/io_utils.py` | Atomic JSON read/write |
| `app/env_utils.py` | Shared bool env parsing |
| `app/logging_config.py` | Entry-point logging setup |
## Data model
One **job** = one sanitized stem (from incoming filename). Flat files:
- `variants/{stem}_original.jpg` — preprocessed copy
- `variants/{stem}_rembg.png` — background removal
- `variants/{stem}_vN.png` — final composed images
- `intermediates/{stem}_vN_*.png` — step images (kept for debugging)
- `meta/{stem}.json` — status + manifest (variants list, filter metadata)
- `processed.json` — worker bookkeeping (incoming path → handled)
## Pipeline origin
The gmic filter/blend chain is ported from an external reference script
`make_random.py` (not in this repo). `config.POST_FILTERS` and asset lists
under `assets/` mirror that script's behaviour.
## Background removal choice
`compare-bg/` is a **standalone benchmark** (rembg models vs withoutbg).
It is not part of the runtime stack. Production uses `u2net` + alpha
matting via `app/rembg_cli.py` (see `config.REMBG_MODEL`).
## Known limitations (intentional)
- **No web auth** — anyone with the link sees all jobs (event tool).
- **Sequential worker** — one photo at a time; burst uploads queue.
- **Meta JSON** — worker and web both read-modify-write `meta/*.json`
without file locking; concurrent remix during active processing can race.
- **Incoming is append-only** — worker never deletes from `incoming/`.
## Dev / test
```bash
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
ruff check app/ tests/
pytest -q
```
Full stack (gmic/rembg): `docker compose up --build` with `DATA_HOST_DIR=./data`.
Python **3.11** matches the Docker image; host 3.14+ cannot install pinned
`onnxruntime` — use Docker or a 3.11 venv for parity.
+89
View File
@@ -0,0 +1,89 @@
# Cleanup Report — live.f12.rocks
Branch: `cleanup/code-quality` (local only, not pushed).
Completed: 2026-07-18.
## Summary
Incremental code-quality cleanup across 10 commits. **No intentional
behavior changes** to the photo pipeline, worker loop, or web UX.
## What changed
| Step | What | Why | Risk | Verification |
|------|------|-----|------|--------------|
| Phase 0 | `CLEANUP_PLAN.md` | Analysis + roadmap | Low | — |
| Tooling | `pyproject.toml`, `requirements-dev.txt`, `.python-version` | Ruff + pytest baseline, pin 3.11 | Low | `ruff check`, `pytest` |
| Style | Ruff format entire `app/` | Consistent style | Low | `ruff format --check` |
| Dead code | Removed unused `blend_layers()` | Less noise | Low | grep + pytest |
| `.gitignore` | Build/coverage/venv patterns | Keep repo clean | Low | — |
| `io_utils` | Atomic JSON helpers | DRY meta/processed I/O | Low | `test_io_utils`, pipeline/worker tests |
| `env_utils` / `logging_config` | Shared bool env + logging | DRY | Low | `test_env_utils`, pytest |
| `compose.yml` | YAML anchor for env | DRY, fewer drift bugs | Low | `docker compose config` (manual) |
| Config | `PREPROCESS_TIMEOUT` constant | Named magic number | Low | default still 120s |
| `read_meta()` | Public meta accessor; `list_jobs` reads once | Perf + clarity | Low | `test_pipeline`, `test_main` |
| Tests | 33 characterization tests | Safe refactor baseline | Low | `pytest -q` |
| CI | `.gitea/workflows/ci.yml` | Lint + test on push/PR | Low | workflow syntax |
| Docs | `README.md`, `ARCHITECTURE.md` | Onboarding | Low | — |
## What was NOT changed (and why)
| Item | Reason |
|------|--------|
| `pipeline.py` module split | Medium risk without broader integration tests; deferred |
| Meta file locking | Behavior change; documented in `ARCHITECTURE.md` |
| Dependency major bumps | Per plan — list only, no auto-bump |
| `compare-bg/` | Separate experiment; left untouched |
| Deploy CI | Manual deploy on boka stays; STANDARDS says ask first |
| FastAPI `TemplateResponse` API migration | Deprecation warning only; no functional change |
| `_validate_job_id` always sanitizes to valid ID | Pre-existing; documented in tests |
## Bugs noticed (not fixed)
1. **Meta JSON race** — worker and web concurrent RMW without locking.
2. **`_filter_timeout_override` global** — thread-unsafe under concurrent remix (uvicorn workers).
3. **`sanitize_stem("a/b")``"b"`** — `Path.stem` drops path prefix before slash replacement.
4. **`_validate_job_id`** — sanitization makes rejection path effectively unreachable.
## Dependency notes (suggestions only)
| Package | Current pin | Note |
|---------|-------------|------|
| `fastapi` | `<0.117` | 0.117+ available; test before bump |
| `onnxruntime` | `<1.20` | Very old pin; rembg may allow newer in future |
| `pillow` | `<11` | Pillow 11 exists; verify rembg compatibility |
No lockfile added — soft pins kept to avoid changing Docker resolve behavior.
## How to verify locally
```bash
git checkout cleanup/code-quality
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
ruff check app/ tests/
pytest -q
# Full stack:
cp .env.example .env # set DATA_HOST_DIR=./data
docker compose up --build
```
## Commits on branch
```
docs: add cleanup plan from phase-0 analysis
chore: add pyproject.toml with ruff and pytest config
style: apply ruff format and extend gitignore
refactor: extract atomic JSON helpers to io_utils
refactor: share env-bool parsing and logging setup
chore: deduplicate compose environment blocks
refactor: named preprocess timeout and single meta read
test: add characterization tests for core modules (+ CI workflow)
docs: update README and add ARCHITECTURE
```
## External references
- `compare-bg/` — local venv `.venv-compare/` and 1.5GB model cache (gitignored)
- `make_random.py` — external reference script, not vendored
- Submodule/symlink: none
+20
View File
@@ -55,6 +55,24 @@ Drop a photo into `incoming/` (via Syncthing or locally). After a few
seconds (poll interval + processing time) it shows up on the web UI; flat seconds (poll interval + processing time) it shows up on the web UI; flat
files land in `variants/` and `intermediates/` for the phone. files land in `variants/` and `intermediates/` for the phone.
## Development
Python **3.11** (same as `Dockerfile`). On a host with newer Python, use
Docker or a 3.11 venv — pinned `onnxruntime` does not publish wheels for 3.14+.
```bash
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
ruff check app/ tests/ # lint
ruff format app/ tests/ # format
pytest -q # unit tests (no gmic/rembg)
```
CI (`.gitea/workflows/ci.yml`): ruff + pytest on push/PR.
Architecture notes: `ARCHITECTURE.md`. Cleanup log: `CLEANUP_PLAN.md`.
## Configuration (`.env`, see `.env.example`) ## Configuration (`.env`, see `.env.example`)
| Var | Default | Notes | | Var | Default | Notes |
@@ -64,6 +82,8 @@ files land in `variants/` and `intermediates/` for the phone.
| `BLEND_OPACITY` | `30%` | Default blend opacity for auto-generated variants | | `BLEND_OPACITY` | `30%` | Default blend opacity for auto-generated variants |
| `FILTER_TIMEOUT` | `120` | Seconds before a single gmic call is killed | | `FILTER_TIMEOUT` | `120` | Seconds before a single gmic call is killed |
| `MAX_FILTER_ATTEMPTS` | `8` | Random-filter retry budget per bg/fg pick | | `MAX_FILTER_ATTEMPTS` | `8` | Random-filter retry budget per bg/fg pick |
| `MAX_EDGE_PX` | `2000` | Longest edge after ImageMagick preprocess |
| `PREPROCESS_TIMEOUT` | `120` | Seconds for preprocess step |
| `NICE_LEVEL` | `18` | `nice -n` level for gmic/rembg subprocesses | | `NICE_LEVEL` | `18` | `nice -n` level for gmic/rembg subprocesses |
## Deploy notes / caveats ## Deploy notes / caveats