Frank Schwenk
c03f17488c
CI / lint-and-test (push) Failing after 1m18s
Replace Syncthing as the product upload path with a share-target app and POST /api/jobs. Web UI live-polls job progress. Co-authored-by: Cursor <cursoragent@cursor.com>
103 lines
4.1 KiB
Markdown
103 lines
4.1 KiB
Markdown
# Architecture — live.f12.rocks
|
|
|
|
## Overview
|
|
|
|
```
|
|
Organizer phone Docker host
|
|
────────────── ───────────
|
|
Share / pick
|
|
→ JPEG 2000px
|
|
→ POST /api/jobs ──► incoming/ ──watch──► worker
|
|
token │
|
|
└── rembg + gmic pipeline
|
|
│
|
|
meta/{stem}.json ◄───────────────────────────────────┘
|
|
│
|
|
├── web (HTML) ──► guest browsers + remix
|
|
└── GET /api/jobs/{id} ──► app polls, local notification,
|
|
save variants + intermediates
|
|
```
|
|
|
|
Two containers share one image (`Dockerfile`) and one bind-mounted data
|
|
directory (`DATA_HOST_DIR` → `/data`). The organizer app is in `android/`.
|
|
Syncthing is an optional fallback writer for `incoming/`, not required.
|
|
|
|
## Python modules
|
|
|
|
| Module | Role |
|
|
|--------|------|
|
|
| `app/worker.py` | Poll `incoming/`, copy to `variants/`, run `process_job()` |
|
|
| `app/pipeline.py` | gmic/rembg orchestration, meta/manifest I/O, variant compose |
|
|
| `app/remix.py` | User-selected remix variant (one-shot compose + manifest append) |
|
|
| `app/main.py` | FastAPI HTML routes, Jinja templates, static files |
|
|
| `app/api.py` | JSON API: health, list, detail, authenticated upload |
|
|
| `app/jobs.py` | Shared job payloads, labels, inbox filenames |
|
|
| `app/config.py` | Environment → constants (paths, timeouts, `UPLOAD_TOKEN`) |
|
|
| `app/rembg_cli.py` | Subprocess entry for rembg (avoids heavy `rembg[cli]` extra) |
|
|
| `app/io_utils.py` | Atomic JSON read/write |
|
|
| `app/env_utils.py` | Shared bool env parsing |
|
|
| `app/logging_config.py` | Entry-point logging setup |
|
|
| `app/images.py` | On-demand WebP srcset cache |
|
|
|
|
## Data model
|
|
|
|
One **job** = one sanitized stem (unique upload name). Flat files:
|
|
|
|
- `variants/{stem}_original.jpg` — preprocessed copy
|
|
- `variants/{stem}_rembg.png` — background removal
|
|
- `variants/{stem}_vN.png` — final composed images
|
|
- `intermediates/{stem}_vN_*.png` — step images (kept for debugging)
|
|
- `meta/{stem}.json` — status + stage + progress + manifest
|
|
- `processed.json` — worker bookkeeping (incoming path → handled)
|
|
|
|
Status values: `queued` → `processing` (stages `preprocess` / `rembg` /
|
|
`variant`) → `done` | `error`.
|
|
|
|
## API
|
|
|
|
| Method | Path | Auth | Notes |
|
|
|--------|------|------|-------|
|
|
| GET | `/api/health` | no | `{ok, upload}` |
|
|
| GET | `/api/jobs` | no | list + `active` flag |
|
|
| GET | `/api/jobs/{id}` | no | full payload + file URLs |
|
|
| POST | `/api/jobs` | Bearer / `X-Upload-Token` | multipart `file`, 202 + job summary |
|
|
| GET | `/jobs/{id}/files/{name}` | no | original file or `?w=` WebP |
|
|
|
|
Empty `UPLOAD_TOKEN` → POST returns 503. Wrong token → 401.
|
|
|
|
## Pipeline origin
|
|
|
|
The gmic filter/blend chain is ported from an external reference script
|
|
`make_random.py` (not in this repo). `config.POST_FILTERS` and asset lists
|
|
under `assets/` mirror that script's behaviour.
|
|
|
|
## Background removal
|
|
|
|
Production uses `u2net` + alpha matting via `app/rembg_cli.py`
|
|
(see `config.REMBG_MODEL`).
|
|
|
|
## Known limitations (intentional)
|
|
|
|
- **No web login** — anyone with the link sees all jobs (event tool).
|
|
- **Upload token** — organizer app only; not an account system.
|
|
- **Sequential worker** — one photo at a time; burst uploads queue.
|
|
- **Meta JSON** — worker and web share `meta/*.json` via locked
|
|
read-modify-write (`fcntl.flock`); variants are merged by id so
|
|
concurrent remix during processing does not clobber entries.
|
|
- **Incoming is append-only** — worker never deletes from `incoming/`.
|
|
|
|
## Dev / test
|
|
|
|
```bash
|
|
python3.11 -m venv .venv && source .venv/bin/activate
|
|
pip install -r requirements-dev.txt
|
|
ruff check app/ tests/
|
|
pytest -q
|
|
```
|
|
|
|
Android: see `android/README.md`. Full stack (gmic/rembg):
|
|
`docker compose up --build` with `DATA_HOST_DIR=./data`.
|
|
|
|
Python **3.11** matches the Docker image; host 3.14+ cannot install pinned
|
|
`onnxruntime` — use Docker or a 3.11 venv for parity.
|