feat: replace SFTP drop with Syncthing share path
Remove SFTPGo; mount event data from the Syncthing folder, watch incoming/, and name variants after the source stem. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
+6
-16
@@ -1,21 +1,11 @@
|
|||||||
# Copy to `.env` and fill in real values. Never commit `.env`.
|
# Copy to `.env` and fill in real values. Never commit `.env`.
|
||||||
|
|
||||||
# --- SFTPGo credentials -----------------------------------------------
|
# --- Host data path (Syncthing share root) ------------------------------
|
||||||
# Whoever runs the event camera/laptop uploads photos here.
|
# Prod default (compose.yml): /home/frank/sync.schwenk.online/data/livef12
|
||||||
SFTP_USER=livef12
|
# Layout under that path: incoming/ jobs/ processed.json
|
||||||
SFTP_PASSWORD=changeme
|
# Local override when the absolute path does not exist:
|
||||||
|
# DATA_HOST_DIR=./data
|
||||||
# --- Host paths / ports --------------------------------------------------
|
# DATA_HOST_DIR=/home/frank/sync.schwenk.online/data/livef12
|
||||||
# On the server this should be an absolute path, e.g.
|
|
||||||
# /home/frank/live.f12.rocks/data
|
|
||||||
DATA_HOST_DIR=./data
|
|
||||||
|
|
||||||
# Spec asked for 121212 which exceeds TCP max (65535). Use 12121.
|
|
||||||
SFTP_HOST_PORT=12121
|
|
||||||
|
|
||||||
# Minutes until SFTPGo closes an idle connection. Android/SSHJ clients
|
|
||||||
# often sit idle after upload; the default 15m shows up as a spurious EOF.
|
|
||||||
SFTPGO_IDLE_TIMEOUT=120
|
|
||||||
|
|
||||||
# --- Pipeline tuning (see make_random.py for background) -----------------
|
# --- Pipeline tuning (see make_random.py for background) -----------------
|
||||||
OUTPUT_COUNT=3
|
OUTPUT_COUNT=3
|
||||||
|
|||||||
+8
-1
@@ -1,7 +1,7 @@
|
|||||||
# Secrets
|
# Secrets
|
||||||
.env
|
.env
|
||||||
|
|
||||||
# Runtime data (bind-mounted, never belongs in git)
|
# Runtime data (bind-mounted / Syncthing share — never in git)
|
||||||
/data/
|
/data/
|
||||||
|
|
||||||
# Local rembg/withoutbg comparison scratch
|
# Local rembg/withoutbg comparison scratch
|
||||||
@@ -14,8 +14,15 @@
|
|||||||
# Python
|
# Python
|
||||||
__pycache__/
|
__pycache__/
|
||||||
*.pyc
|
*.pyc
|
||||||
|
*.pyo
|
||||||
.venv/
|
.venv/
|
||||||
|
.pytest_cache/
|
||||||
|
.mypy_cache/
|
||||||
|
.ruff_cache/
|
||||||
|
|
||||||
# OS / editor noise
|
# OS / editor noise
|
||||||
.DS_Store
|
.DS_Store
|
||||||
*.swp
|
*.swp
|
||||||
|
*~
|
||||||
|
.idea/
|
||||||
|
.vscode/
|
||||||
|
|||||||
@@ -4,7 +4,14 @@
|
|||||||
|
|
||||||
- Live on boka: `/home/frank/live.f12.rocks`
|
- Live on boka: `/home/frank/live.f12.rocks`
|
||||||
- Web: https://live.f12.rocks
|
- Web: https://live.f12.rocks
|
||||||
- SFTP: `livef12@frank-schwenk.de` port **12121** (spec 121212 is invalid TCP)
|
- Password / secrets in server `.env` only (`chmod 600`)
|
||||||
- Password in server `.env` only (`chmod 600`); ntfy sent to phone
|
|
||||||
- Data: `/home/frank/live.f12.rocks/data` (inbox not auto-cleaned)
|
|
||||||
- First rembg job downloads ~176 MB model into docker volume `rembg_cache`
|
- First rembg job downloads ~176 MB model into docker volume `rembg_cache`
|
||||||
|
|
||||||
|
## Syncthing cutover (2026-07-18)
|
||||||
|
|
||||||
|
- SFTPGo removed; drop path is Syncthing
|
||||||
|
- Data share: `/home/frank/sync.schwenk.online/data/livef12`
|
||||||
|
(`incoming/` + `jobs/` + `processed.json`)
|
||||||
|
- Old path `/home/frank/live.f12.rocks/data` no longer used by compose
|
||||||
|
(migration/cleanup manual)
|
||||||
|
- Host port 12121 can be closed after cutover
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
# live.f12.rocks
|
# live.f12.rocks
|
||||||
|
|
||||||
Event photo pep pipeline: guests upload a photo via SFTP, a worker runs it
|
Event photo pep pipeline: guests drop a photo into a Syncthing folder, a
|
||||||
through `rembg` (background removal) + `gmic` (random filter/blend
|
worker runs it through `rembg` (background removal) + `gmic` (random
|
||||||
compositing, ported from `make_random.py`), and a small web UI shows the
|
filter/blend compositing, ported from `make_random.py`), and a small web
|
||||||
results with a "remix" option to try different filters on the same photo.
|
UI shows the results with a "remix" option to try different filters on
|
||||||
|
the same photo. Finished jobs sync back via the same share.
|
||||||
|
|
||||||
No web auth. This is an event tool, not a DAM.
|
No web auth. This is an event tool, not a DAM.
|
||||||
|
|
||||||
@@ -11,25 +12,26 @@ No web auth. This is an event tool, not a DAM.
|
|||||||
|
|
||||||
| Service | What | Port |
|
| Service | What | Port |
|
||||||
|----------|--------------------------------------------------|--------------------------|
|
|----------|--------------------------------------------------|--------------------------|
|
||||||
| `sftpgo` | SFTP drop point, `drakkan/sftpgo` | `12121` (host) -> `2022` |
|
| `worker` | Watches `incoming/`, runs the compose pipeline | none published |
|
||||||
| `worker` | Watches inbox, runs the compose pipeline | none published |
|
|
||||||
| `web` | FastAPI + Jinja UI, browse jobs, remix | via Traefik only |
|
| `web` | FastAPI + Jinja UI, browse jobs, remix | via Traefik only |
|
||||||
|
|
||||||
All three share one host directory (`DATA_HOST_DIR`, default `./data`),
|
Both share one host directory (`DATA_HOST_DIR`) mounted at `/data`.
|
||||||
mounted at different paths — see the comment at the top of `compose.yml`.
|
Syncthing runs on the host — not in this compose.
|
||||||
|
|
||||||
## Data layout (`./data`)
|
## Data layout (Syncthing share)
|
||||||
|
|
||||||
|
Prod path: `/home/frank/sync.schwenk.online/data/livef12`
|
||||||
|
|
||||||
```
|
```
|
||||||
inbox/ # SFTP drop — worker only ever reads/copies from here
|
incoming/ # phone drop — worker only ever reads/copies from here
|
||||||
jobs/<job_id>/
|
jobs/<job_id>/
|
||||||
original.<ext> # private copy of the uploaded photo
|
original.<ext> # private copy of the uploaded photo
|
||||||
rembg.png # background removed (computed once, reused)
|
rembg.png # background removed (computed once, reused)
|
||||||
intermediates/ # every intermediate step, kept for inspection
|
intermediates/ # every intermediate step, kept for inspection
|
||||||
variants/ # final composed images
|
variants/ # final composed images ({stem}_v1.png, …)
|
||||||
manifest.json # filter names/commands/blends per variant
|
manifest.json # filter names/commands/blends per variant
|
||||||
status.json # pending | processing | done | error
|
status.json # pending | processing | done | error
|
||||||
processed.json # worker bookkeeping: which inbox files were handled
|
processed.json # worker bookkeeping: which incoming files were handled
|
||||||
```
|
```
|
||||||
|
|
||||||
`assets/` (filter lists + trimmed `filters.json`) lives in the repo and is
|
`assets/` (filter lists + trimmed `filters.json`) lives in the repo and is
|
||||||
@@ -39,27 +41,20 @@ bind-mounted read-only into `worker`/`web` at `/data/assets`.
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp .env.example .env
|
cp .env.example .env
|
||||||
# edit .env: set a real SFTP_PASSWORD, and DATA_HOST_DIR on the server
|
# local: set DATA_HOST_DIR=./data
|
||||||
|
# prod: leave unset (defaults to the Syncthing share path)
|
||||||
docker compose up -d --build
|
docker compose up -d --build
|
||||||
```
|
```
|
||||||
|
|
||||||
Upload a photo:
|
Drop a photo into `incoming/` (via Syncthing or locally). After a few
|
||||||
|
seconds (poll interval + processing time) it shows up on the web UI as a
|
||||||
```bash
|
new job; the full `jobs/<id>/` tree syncs back to the phone.
|
||||||
sftp -P 12121 livef12@<host>
|
|
||||||
put photo.jpg
|
|
||||||
```
|
|
||||||
|
|
||||||
After a few seconds (poll interval + processing time) it shows up on the
|
|
||||||
web UI as a new job.
|
|
||||||
|
|
||||||
## Configuration (`.env`, see `.env.example`)
|
## Configuration (`.env`, see `.env.example`)
|
||||||
|
|
||||||
| Var | Default | Notes |
|
| Var | Default | Notes |
|
||||||
|-----|---------|-------|
|
|-----|---------|-------|
|
||||||
| `SFTP_USER` / `SFTP_PASSWORD` | `livef12` / `changeme` | SFTP login, home dir is locked to the shared inbox |
|
| `DATA_HOST_DIR` | `/home/frank/sync.schwenk.online/data/livef12` | Syncthing share root; use `./data` locally |
|
||||||
| `DATA_HOST_DIR` | `./data` | On the server: `/home/frank/live.f12.rocks/data` |
|
|
||||||
| `SFTP_HOST_PORT` | `12121` | Host-side SFTP port (121212 is invalid TCP) |
|
|
||||||
| `OUTPUT_COUNT` | `3` | Variants generated per uploaded photo |
|
| `OUTPUT_COUNT` | `3` | Variants generated per uploaded photo |
|
||||||
| `BLEND_OPACITY` | `30%` | Default blend opacity for auto-generated variants |
|
| `BLEND_OPACITY` | `30%` | Default blend opacity for auto-generated variants |
|
||||||
| `FILTER_TIMEOUT` | `120` | Seconds before a single gmic call is killed |
|
| `FILTER_TIMEOUT` | `120` | Seconds before a single gmic call is killed |
|
||||||
@@ -68,26 +63,17 @@ web UI as a new job.
|
|||||||
|
|
||||||
## Deploy notes / caveats
|
## Deploy notes / caveats
|
||||||
|
|
||||||
- **Port note:** Spec originally said `121212`, which exceeds TCP max
|
- **Firewall:** Traefik must already route `live.f12.rocks` — this repo
|
||||||
(65535). Production default is **`12121`**.
|
only adds the router labels; it assumes the external `traefik` docker
|
||||||
- **Firewall:** open host port `12121` (SFTP) and make sure Traefik
|
network exists. No SFTP port needed.
|
||||||
already routes `live.f12.rocks` — this repo only adds the router labels,
|
|
||||||
it assumes the external `traefik` docker network exists.
|
|
||||||
- **SFTPGo first run:** the `sftpgo` service auto-creates the SFTP user
|
|
||||||
from `SFTP_USER`/`SFTP_PASSWORD` on every start via
|
|
||||||
`sftpgo/entrypoint.sh` (uses `jq`, bundled in the official image, to
|
|
||||||
build a `loaddata` JSON safely — no manual admin setup needed). Host SSH
|
|
||||||
keys persist in the `sftpgo_state` named volume, not in `./data`.
|
|
||||||
The SFTPGo web admin exists internally on port 8080 but is deliberately
|
|
||||||
**not** published or routed — there's no need for it here.
|
|
||||||
- **rembg model download:** first background-removal call downloads the
|
- **rembg model download:** first background-removal call downloads the
|
||||||
`u2net` ONNX model (~176 MB) from GitHub. This needs outbound internet
|
`u2net` ONNX model (~176 MB) from GitHub. This needs outbound internet
|
||||||
on first run and can take a minute or two depending on the link; the
|
on first run and can take a minute or two depending on the link; the
|
||||||
model is cached in the `rembg_cache` named volume afterwards, so
|
model is cached in the `rembg_cache` named volume afterwards, so
|
||||||
restarts don't re-download it.
|
restarts don't re-download it.
|
||||||
- **Inbox is append-only:** the worker only ever copies out of `inbox/`
|
- **Incoming is append-only:** the worker only ever copies out of
|
||||||
and never deletes or moves anything there — plan disk space
|
`incoming/` and never deletes or moves anything there — plan disk space
|
||||||
accordingly, or clean up `inbox/` manually between events.
|
accordingly, or clean up `incoming/` manually between events.
|
||||||
- **Sequential processing:** the worker handles one photo at a time
|
- **Sequential processing:** the worker handles one photo at a time
|
||||||
(`cpus: "1.0"`, no concurrency) — fine for an event pace, but a burst of
|
(`cpus: "1.0"`, no concurrency) — fine for an event pace, but a burst of
|
||||||
uploads will just queue up and get processed in order.
|
uploads will just queue up and get processed in order.
|
||||||
|
|||||||
@@ -6,23 +6,24 @@
|
|||||||
|
|
||||||
## One-Liner
|
## One-Liner
|
||||||
|
|
||||||
Event photo pep pipeline: SFTP -> gmic/rembg -> web. Upload a photo at the
|
Event photo pep pipeline: Syncthing drop -> gmic/rembg -> web. Upload a
|
||||||
event, get back weird/fun filtered variants seconds later.
|
photo at the event, get back weird/fun filtered variants seconds later.
|
||||||
|
|
||||||
## Vision
|
## Vision
|
||||||
|
|
||||||
At an event (f12 meetup/party), someone drops a photo into an SFTP inbox
|
At an event (f12 meetup/party), someone drops a photo into a Syncthing
|
||||||
from a laptop/camera rig. A worker rips the background off, runs it
|
folder from a phone/camera rig. A worker rips the background off, runs it
|
||||||
through a random gmic filter/blend chain, and a handful of variants show
|
through a random gmic filter/blend chain, and a handful of variants show
|
||||||
up on a shared web page almost immediately — no app install, no login, no
|
up on a shared web page almost immediately — no app install, no login, no
|
||||||
waiting for someone to "process the photos later". If a variant is fun,
|
waiting for someone to "process the photos later". If a variant is fun,
|
||||||
remix it with different filters right there.
|
remix it with different filters right there. Finished jobs sync back to
|
||||||
|
the phone via the same Syncthing share.
|
||||||
|
|
||||||
## Audience
|
## Audience
|
||||||
|
|
||||||
Event organizers and attendees at f12 events, on their phones, often on
|
Event organizers and attendees at f12 events, on their phones, often on
|
||||||
bad venue wifi. Nobody signs up, nobody logs in. Whoever has the link
|
bad venue wifi. Nobody signs up, nobody logs in. Whoever has the link
|
||||||
(and whoever's uploading from the SFTP side) is the whole audience.
|
(and whoever's syncing photos in) is the whole audience.
|
||||||
|
|
||||||
## Tone & Wording
|
## Tone & Wording
|
||||||
|
|
||||||
@@ -41,7 +42,7 @@ bad venue wifi. Nobody signs up, nobody logs in. Whoever has the link
|
|||||||
|---------|------|-----|
|
|---------|------|-----|
|
||||||
| Success | `Remix erzeugt.` | `Dein Kunstwerk ist fertig! 🎉` |
|
| Success | `Remix erzeugt.` | `Dein Kunstwerk ist fertig! 🎉` |
|
||||||
| Error | `Foreground-Filter fehlgeschlagen: Timeout nach 120s` | `Etwas ist schiefgelaufen.` |
|
| Error | `Foreground-Filter fehlgeschlagen: Timeout nach 120s` | `Etwas ist schiefgelaufen.` |
|
||||||
| Empty state | `Noch keine Jobs. Bild per SFTP hochladen, dann kurz warten.` | `Hier ist noch nichts los... lade doch was hoch! 😊` |
|
| Empty state | `Noch keine Jobs. Bild in den Sync-Ordner legen, dann kurz warten.` | `Hier ist noch nichts los... lade doch was hoch! 😊` |
|
||||||
|
|
||||||
## Design
|
## Design
|
||||||
|
|
||||||
@@ -55,33 +56,32 @@ bad venue wifi. Nobody signs up, nobody logs in. Whoever has the link
|
|||||||
|
|
||||||
- No web auth / accounts / login product — anyone with the link sees
|
- No web auth / accounts / login product — anyone with the link sees
|
||||||
everything, by design, for this event tool.
|
everything, by design, for this event tool.
|
||||||
- No camera-to-phone client — upload path is SFTP only.
|
- No camera-to-phone client in this repo — upload path is Syncthing
|
||||||
|
(phone/folder setup is out of band).
|
||||||
- No beamer/projector product built into this repo.
|
- No beamer/projector product built into this repo.
|
||||||
- Not a DAM (digital asset manager) — no albums, tagging, search,
|
- Not a DAM (digital asset manager) — no albums, tagging, search,
|
||||||
retention policies. `inbox/` and `jobs/` are the whole data model.
|
retention policies. `incoming/` and `jobs/` are the whole data model.
|
||||||
- No auto-deletion of inbox uploads. Ever. The worker only reads/copies.
|
- No auto-deletion of incoming uploads. Ever. The worker only reads/copies.
|
||||||
|
- No Syncthing container in this compose — Syncthing runs on the host.
|
||||||
|
|
||||||
## Infrastructure (project-local)
|
## Infrastructure (project-local)
|
||||||
|
|
||||||
- Workdir on server: `/home/frank/live.f12.rocks`
|
- Workdir on server: `/home/frank/live.f12.rocks`
|
||||||
- Data volume: `${DATA_HOST_DIR}` (default `./data` locally, absolute
|
- Data volume (Syncthing share): `${DATA_HOST_DIR}` — prod default
|
||||||
path `/home/frank/live.f12.rocks/data` on the server) — bind-mounted
|
`/home/frank/sync.schwenk.online/data/livef12`, local override `./data`.
|
||||||
into all three services (`sftpgo`, `worker`, `web`) at different
|
Bind-mounted into `worker` and `web` at `/data` (see `compose.yml`).
|
||||||
container paths, see `compose.yml` header comment.
|
- Layout under the share: `incoming/` (phone drop), `jobs/` (full tree
|
||||||
- SFTP: host port `12121` -> `sftpgo:2022` (spec `121212` is not a valid
|
syncs back), `processed.json`.
|
||||||
TCP port). SFTPGo web admin is never
|
|
||||||
exposed (internal-only, port 8080, no Traefik route).
|
|
||||||
- Domain: `live.f12.rocks`, routed via the shared external `traefik`
|
- Domain: `live.f12.rocks`, routed via the shared external `traefik`
|
||||||
network, TLS via `myresolver` (see `INFRASTRUCTURE.md`).
|
network, TLS via `myresolver` (see `INFRASTRUCTURE.md`).
|
||||||
- rembg model cache and SFTPGo host keys live in named Docker volumes
|
- rembg model cache lives in the named Docker volume `rembg_cache`, not
|
||||||
(`rembg_cache`, `sftpgo_state`), not under `./data` — they're
|
under the Syncthing share — container state, not event data.
|
||||||
container state, not event data.
|
|
||||||
|
|
||||||
## Project-Specific Rules
|
## Project-Specific Rules
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
- Never delete or move files under inbox/ from worker code — SFTP is the
|
- Never delete or move files under incoming/ from worker code — Syncthing
|
||||||
only writer there. (reason: uploads are the one copy of the original
|
is the writer there. (reason: uploads are the one copy of the original
|
||||||
that exists outside a phone's camera roll)
|
that exists outside a phone's camera roll)
|
||||||
- Keep every intermediate image in jobs/<id>/intermediates/ — don't clean
|
- Keep every intermediate image in jobs/<id>/intermediates/ — don't clean
|
||||||
them up automatically. (reason: useful for debugging bad filter picks,
|
them up automatically. (reason: useful for debugging bad filter picks,
|
||||||
|
|||||||
+1
-1
@@ -1 +1 @@
|
|||||||
"""live.f12.rocks — event photo pipeline (SFTP inbox -> gmic/rembg -> web)."""
|
"""live.f12.rocks — event photo pipeline (Syncthing drop -> gmic/rembg -> web)."""
|
||||||
|
|||||||
+1
-1
@@ -26,7 +26,7 @@ def _float_env(name: str, default: float) -> float:
|
|||||||
|
|
||||||
# --- Paths -------------------------------------------------------------
|
# --- Paths -------------------------------------------------------------
|
||||||
DATA_DIR = Path(os.environ.get("DATA_DIR", "/data"))
|
DATA_DIR = Path(os.environ.get("DATA_DIR", "/data"))
|
||||||
INBOX_DIR = DATA_DIR / "inbox"
|
INBOX_DIR = DATA_DIR / "incoming"
|
||||||
JOBS_DIR = DATA_DIR / "jobs"
|
JOBS_DIR = DATA_DIR / "jobs"
|
||||||
ASSETS_DIR = DATA_DIR / "assets"
|
ASSETS_DIR = DATA_DIR / "assets"
|
||||||
PROCESSED_FILE = DATA_DIR / "processed.json"
|
PROCESSED_FILE = DATA_DIR / "processed.json"
|
||||||
|
|||||||
+59
-9
@@ -210,7 +210,7 @@ def preprocess_original(paths: JobPaths) -> JobPaths:
|
|||||||
"""Downscale to MAX_EDGE_PX and normalize to sRGB JPEG as `original.jpg`.
|
"""Downscale to MAX_EDGE_PX and normalize to sRGB JPEG as `original.jpg`.
|
||||||
|
|
||||||
Replaces any non-jpg original in the job dir. Side-effect free for the
|
Replaces any non-jpg original in the job dir. Side-effect free for the
|
||||||
inbox copy — only mutates jobs/<id>/.
|
incoming copy — only mutates jobs/<id>/.
|
||||||
"""
|
"""
|
||||||
src = paths.original
|
src = paths.original
|
||||||
if not src.exists():
|
if not src.exists():
|
||||||
@@ -371,12 +371,37 @@ def next_variant_id(manifest: dict[str, Any]) -> str:
|
|||||||
return f"v{i}"
|
return f"v{i}"
|
||||||
|
|
||||||
|
|
||||||
|
_UNSAFE_STEM_RE = re.compile(r"[^\w.\-]+", re.UNICODE)
|
||||||
|
|
||||||
|
|
||||||
|
def sanitize_stem(name: str) -> str:
|
||||||
|
"""Safe basename stem for variant files (no path separators / junk)."""
|
||||||
|
stem = Path(name).stem if name else ""
|
||||||
|
stem = stem.replace("/", "_").replace("\\", "_").strip().strip(".")
|
||||||
|
stem = _UNSAFE_STEM_RE.sub("_", stem).strip("._")
|
||||||
|
return stem or "photo"
|
||||||
|
|
||||||
|
|
||||||
|
def variant_filename(stem: str, variant_id: str) -> str:
|
||||||
|
return f"{sanitize_stem(stem)}_{variant_id}.png"
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_source_stem(manifest: dict[str, Any] | None, paths: JobPaths) -> str:
|
||||||
|
"""Prefer manifest source_stem; fall back to original path stem."""
|
||||||
|
if manifest:
|
||||||
|
stored = manifest.get("source_stem")
|
||||||
|
if isinstance(stored, str) and stored.strip():
|
||||||
|
return sanitize_stem(stored)
|
||||||
|
return sanitize_stem(paths.original.stem)
|
||||||
|
|
||||||
|
|
||||||
def compose_variant(
|
def compose_variant(
|
||||||
paths: JobPaths,
|
paths: JobPaths,
|
||||||
assets: FilterAssets,
|
assets: FilterAssets,
|
||||||
*,
|
*,
|
||||||
variant_id: str,
|
variant_id: str,
|
||||||
source: str,
|
source: str,
|
||||||
|
variant_stem: str,
|
||||||
bg_name: str | None = None,
|
bg_name: str | None = None,
|
||||||
bg_mode: str | None = None,
|
bg_mode: str | None = None,
|
||||||
fg_name: str | None = None,
|
fg_name: str | None = None,
|
||||||
@@ -389,10 +414,13 @@ def compose_variant(
|
|||||||
If bg_name/fg_name/bg_mode/fg_mode are given (remix path) they are used
|
If bg_name/fg_name/bg_mode/fg_mode are given (remix path) they are used
|
||||||
directly. Otherwise a random working filter is picked with retries,
|
directly. Otherwise a random working filter is picked with retries,
|
||||||
exactly like make_random.py's compose_one().
|
exactly like make_random.py's compose_one().
|
||||||
|
|
||||||
|
Final file is `variants/{stem}_{variant_id}.png`; manifest id stays `vN`.
|
||||||
"""
|
"""
|
||||||
rng = rng or random.Random()
|
rng = rng or random.Random()
|
||||||
tmp_dir = paths.intermediates
|
tmp_dir = paths.intermediates
|
||||||
tmp_dir.mkdir(parents=True, exist_ok=True)
|
tmp_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
stem = sanitize_stem(variant_stem)
|
||||||
|
|
||||||
bg_mode = bg_mode or rng.choice(assets.blend_modes)
|
bg_mode = bg_mode or rng.choice(assets.blend_modes)
|
||||||
fg_mode = fg_mode or rng.choice(assets.blend_modes)
|
fg_mode = fg_mode or rng.choice(assets.blend_modes)
|
||||||
@@ -421,7 +449,7 @@ def compose_variant(
|
|||||||
"step2": tmp_dir / f"{variant_id}_rembg_alpha.png",
|
"step2": tmp_dir / f"{variant_id}_rembg_alpha.png",
|
||||||
"fg_filtered": tmp_dir / f"{variant_id}_fg_filtered.png",
|
"fg_filtered": tmp_dir / f"{variant_id}_fg_filtered.png",
|
||||||
"composed": tmp_dir / f"{variant_id}_composed.png",
|
"composed": tmp_dir / f"{variant_id}_composed.png",
|
||||||
"final": paths.variants / f"{variant_id}.png",
|
"final": paths.variants / variant_filename(stem, variant_id),
|
||||||
}
|
}
|
||||||
paths.variants.mkdir(parents=True, exist_ok=True)
|
paths.variants.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
@@ -467,13 +495,19 @@ def compose_variant(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def process_job(job_id: str, source_path: Path, original_suffix: str) -> None:
|
def process_job(
|
||||||
"""Full pipeline for a freshly ingested inbox file: preprocess, rembg
|
job_id: str,
|
||||||
|
source_path: Path,
|
||||||
|
original_suffix: str,
|
||||||
|
*,
|
||||||
|
source_stem: str | None = None,
|
||||||
|
) -> None:
|
||||||
|
"""Full pipeline for a freshly ingested incoming file: preprocess, rembg
|
||||||
once, generate OUTPUT_COUNT variants, write manifest + status.
|
once, generate OUTPUT_COUNT variants, write manifest + status.
|
||||||
|
|
||||||
`source_path` must already be a private copy (jobs/<id>/original.*) —
|
`source_path` must already be a private copy (jobs/<id>/original.*) —
|
||||||
callers (worker.py) are responsible for copying out of inbox first, so
|
callers (worker.py) are responsible for copying out of incoming first, so
|
||||||
the inbox file itself is never touched here.
|
the incoming file itself is never touched here.
|
||||||
|
|
||||||
Variants that hit FILTER_TIMEOUT during the normal pass are retried
|
Variants that hit FILTER_TIMEOUT during the normal pass are retried
|
||||||
once at the end with FILTER_TIMEOUT_LONG. Filter probes in
|
once at the end with FILTER_TIMEOUT_LONG. Filter probes in
|
||||||
@@ -483,8 +517,9 @@ def process_job(job_id: str, source_path: Path, original_suffix: str) -> None:
|
|||||||
paths.root.mkdir(parents=True, exist_ok=True)
|
paths.root.mkdir(parents=True, exist_ok=True)
|
||||||
paths.intermediates.mkdir(parents=True, exist_ok=True)
|
paths.intermediates.mkdir(parents=True, exist_ok=True)
|
||||||
paths.variants.mkdir(parents=True, exist_ok=True)
|
paths.variants.mkdir(parents=True, exist_ok=True)
|
||||||
|
stem = sanitize_stem(source_stem or source_path.stem)
|
||||||
|
|
||||||
write_status(paths, "processing", source_file=str(source_path.name))
|
write_status(paths, "processing", source_file=str(source_path.name), source_stem=stem)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
paths = preprocess_original(paths)
|
paths = preprocess_original(paths)
|
||||||
@@ -493,6 +528,7 @@ def process_job(job_id: str, source_path: Path, original_suffix: str) -> None:
|
|||||||
"job_id": job_id,
|
"job_id": job_id,
|
||||||
"original_file": paths.original.name,
|
"original_file": paths.original.name,
|
||||||
"rembg_file": paths.rembg.name,
|
"rembg_file": paths.rembg.name,
|
||||||
|
"source_stem": stem,
|
||||||
"created_at": now_iso(),
|
"created_at": now_iso(),
|
||||||
"variants": [],
|
"variants": [],
|
||||||
}
|
}
|
||||||
@@ -509,7 +545,14 @@ def process_job(job_id: str, source_path: Path, original_suffix: str) -> None:
|
|||||||
for i in range(1, config.OUTPUT_COUNT + 1):
|
for i in range(1, config.OUTPUT_COUNT + 1):
|
||||||
variant_id = f"v{i}"
|
variant_id = f"v{i}"
|
||||||
try:
|
try:
|
||||||
entry = compose_variant(paths, assets, variant_id=variant_id, source="auto", rng=rng)
|
entry = compose_variant(
|
||||||
|
paths,
|
||||||
|
assets,
|
||||||
|
variant_id=variant_id,
|
||||||
|
source="auto",
|
||||||
|
variant_stem=stem,
|
||||||
|
rng=rng,
|
||||||
|
)
|
||||||
manifest["variants"].append(entry)
|
manifest["variants"].append(entry)
|
||||||
write_manifest(paths, manifest)
|
write_manifest(paths, manifest)
|
||||||
except FilterTimeoutError as exc:
|
except FilterTimeoutError as exc:
|
||||||
@@ -530,7 +573,14 @@ def process_job(job_id: str, source_path: Path, original_suffix: str) -> None:
|
|||||||
with filter_timeout(config.FILTER_TIMEOUT_LONG):
|
with filter_timeout(config.FILTER_TIMEOUT_LONG):
|
||||||
for variant_id in timeout_retries:
|
for variant_id in timeout_retries:
|
||||||
try:
|
try:
|
||||||
entry = compose_variant(paths, assets, variant_id=variant_id, source="auto", rng=rng)
|
entry = compose_variant(
|
||||||
|
paths,
|
||||||
|
assets,
|
||||||
|
variant_id=variant_id,
|
||||||
|
source="auto",
|
||||||
|
variant_stem=stem,
|
||||||
|
rng=rng,
|
||||||
|
)
|
||||||
manifest["variants"].append(entry)
|
manifest["variants"].append(entry)
|
||||||
write_manifest(paths, manifest)
|
write_manifest(paths, manifest)
|
||||||
except PipelineError as exc:
|
except PipelineError as exc:
|
||||||
|
|||||||
@@ -49,6 +49,8 @@ def create_remix_variant(job_id: str, original_suffix: str, choice: RemixChoice)
|
|||||||
"created_at": pipeline.now_iso(),
|
"created_at": pipeline.now_iso(),
|
||||||
"variants": [],
|
"variants": [],
|
||||||
}
|
}
|
||||||
|
variant_stem = pipeline.resolve_source_stem(manifest, paths)
|
||||||
|
manifest.setdefault("source_stem", variant_stem)
|
||||||
variant_id = pipeline.next_variant_id(manifest)
|
variant_id = pipeline.next_variant_id(manifest)
|
||||||
|
|
||||||
entry = pipeline.compose_variant(
|
entry = pipeline.compose_variant(
|
||||||
@@ -56,6 +58,7 @@ def create_remix_variant(job_id: str, original_suffix: str, choice: RemixChoice)
|
|||||||
assets,
|
assets,
|
||||||
variant_id=variant_id,
|
variant_id=variant_id,
|
||||||
source="remix",
|
source="remix",
|
||||||
|
variant_stem=variant_stem,
|
||||||
bg_name=choice.bg_filter,
|
bg_name=choice.bg_filter,
|
||||||
bg_mode=choice.bg_blend,
|
bg_mode=choice.bg_blend,
|
||||||
fg_name=choice.fg_filter,
|
fg_name=choice.fg_filter,
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
{% block title %}{{ site_title }}{% endblock %}
|
{% block title %}{{ site_title }}{% endblock %}
|
||||||
{% block content %}
|
{% block content %}
|
||||||
{% if not jobs %}
|
{% if not jobs %}
|
||||||
<p class="empty-state">Noch keine Jobs. Bild per SFTP hochladen, dann kurz warten.</p>
|
<p class="empty-state">Noch keine Jobs. Bild in den Sync-Ordner legen, dann kurz warten.</p>
|
||||||
{% else %}
|
{% else %}
|
||||||
<ul class="job-grid">
|
<ul class="job-grid">
|
||||||
{% for job in jobs %}
|
{% for job in jobs %}
|
||||||
|
|||||||
+21
-8
@@ -1,8 +1,8 @@
|
|||||||
"""Inbox watcher + sequential job runner.
|
"""Incoming watcher + sequential job runner.
|
||||||
|
|
||||||
Polls `DATA_DIR/inbox` for new, size-stable image files, copies each one
|
Polls `DATA_DIR/incoming` for new, size-stable image files, copies each one
|
||||||
into its own `jobs/<job_id>/original.*` and runs the compose pipeline on
|
into its own `jobs/<job_id>/original.*` and runs the compose pipeline on
|
||||||
it. Files are NEVER deleted or moved from the inbox — `processed.json`
|
it. Files are NEVER deleted or moved from incoming — `processed.json`
|
||||||
tracks what has already been handled (by path + size + mtime) so restarts
|
tracks what has already been handled (by path + size + mtime) so restarts
|
||||||
don't reprocess everything.
|
don't reprocess everything.
|
||||||
|
|
||||||
@@ -61,9 +61,18 @@ def _is_already_processed(processed: dict[str, Any], path: Path, stat: Any) -> b
|
|||||||
return entry.get("size") == stat.st_size and entry.get("mtime") == stat.st_mtime
|
return entry.get("size") == stat.st_size and entry.get("mtime") == stat.st_mtime
|
||||||
|
|
||||||
|
|
||||||
|
def _is_ignored_name(name: str) -> bool:
|
||||||
|
"""Skip Syncthing metadata/temp files and other dotfiles."""
|
||||||
|
if name.startswith("."):
|
||||||
|
return True
|
||||||
|
if name.startswith(".syncthing.") or ".syncthing." in name:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
def _is_stable(path: Path) -> bool:
|
def _is_stable(path: Path) -> bool:
|
||||||
"""A file is "stable" if its size doesn't change across a short wait —
|
"""A file is "stable" if its size doesn't change across a short wait —
|
||||||
cheap way to avoid picking up a half-uploaded SFTP transfer."""
|
cheap way to avoid picking up a half-written Syncthing transfer."""
|
||||||
try:
|
try:
|
||||||
size_before = path.stat().st_size
|
size_before = path.stat().st_size
|
||||||
except OSError:
|
except OSError:
|
||||||
@@ -77,12 +86,15 @@ def _is_stable(path: Path) -> bool:
|
|||||||
|
|
||||||
|
|
||||||
def find_new_files(processed: dict[str, Any]) -> list[Path]:
|
def find_new_files(processed: dict[str, Any]) -> list[Path]:
|
||||||
|
"""Top-level images under incoming/ only — never recurse into jobs/."""
|
||||||
if not config.INBOX_DIR.exists():
|
if not config.INBOX_DIR.exists():
|
||||||
return []
|
return []
|
||||||
candidates: list[Path] = []
|
candidates: list[Path] = []
|
||||||
for path in sorted(config.INBOX_DIR.rglob("*")):
|
for path in sorted(config.INBOX_DIR.iterdir()):
|
||||||
if not path.is_file():
|
if not path.is_file():
|
||||||
continue
|
continue
|
||||||
|
if _is_ignored_name(path.name):
|
||||||
|
continue
|
||||||
if path.suffix.lower() not in config.SUPPORTED_EXTENSIONS:
|
if path.suffix.lower() not in config.SUPPORTED_EXTENSIONS:
|
||||||
continue
|
continue
|
||||||
try:
|
try:
|
||||||
@@ -105,13 +117,14 @@ def handle_file(path: Path, processed: dict[str, Any]) -> None:
|
|||||||
suffix = path.suffix.lower()
|
suffix = path.suffix.lower()
|
||||||
paths = pipeline.job_paths(job_id, suffix)
|
paths = pipeline.job_paths(job_id, suffix)
|
||||||
paths.root.mkdir(parents=True, exist_ok=True)
|
paths.root.mkdir(parents=True, exist_ok=True)
|
||||||
|
source_stem = pipeline.sanitize_stem(path.stem)
|
||||||
|
|
||||||
logger.info("new inbox file %s -> job %s", path.name, job_id)
|
logger.info("new incoming file %s -> job %s (stem=%s)", path.name, job_id, source_stem)
|
||||||
# Copy (never move) so the inbox stays untouched.
|
# Copy (never move) so incoming stays untouched.
|
||||||
shutil.copy2(path, paths.original)
|
shutil.copy2(path, paths.original)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
pipeline.process_job(job_id, paths.original, suffix)
|
pipeline.process_job(job_id, paths.original, suffix, source_stem=source_stem)
|
||||||
finally:
|
finally:
|
||||||
# Mark as processed regardless of pipeline outcome so a permanently
|
# Mark as processed regardless of pipeline outcome so a permanently
|
||||||
# broken image doesn't get retried forever; failures are visible in
|
# broken image doesn't get retried forever; failures are visible in
|
||||||
|
|||||||
+9
-43
@@ -1,51 +1,18 @@
|
|||||||
# live.f12.rocks — event photo pipeline: SFTP inbox -> gmic/rembg -> web.
|
# live.f12.rocks — event photo pipeline: Syncthing drop -> gmic/rembg -> web.
|
||||||
#
|
#
|
||||||
# All three services share one host directory (default ./data, see
|
# worker + web share one host directory (Syncthing share root):
|
||||||
# .env.example DATA_HOST_DIR) at different mount points:
|
# ${DATA_HOST_DIR:-/home/frank/sync.schwenk.online/data/livef12} -> /data
|
||||||
# sftpgo -> /srv/sftpgo/data (SFTP user home = /srv/sftpgo/data/inbox)
|
# worker watches /data/incoming, writes /data/jobs
|
||||||
# worker -> /data (watches /data/inbox, writes /data/jobs)
|
# web reads jobs, writes remix variants
|
||||||
# web -> /data (reads jobs, writes remix variants)
|
|
||||||
#
|
#
|
||||||
# Copy .env.example to .env and fill in real SFTP credentials before
|
# Syncthing itself runs on the host (not in this compose). Copy .env.example
|
||||||
# running this. Never commit .env.
|
# to .env for local overrides (e.g. DATA_HOST_DIR=./data). Never commit .env.
|
||||||
|
|
||||||
services:
|
services:
|
||||||
sftpgo:
|
|
||||||
image: drakkan/sftpgo:v2
|
|
||||||
restart: unless-stopped
|
|
||||||
# The image defaults to a fixed uid 1000 user. worker/web run as root
|
|
||||||
# (see Dockerfile), and Docker itself creates first-run bind-mount
|
|
||||||
# directories (./data/inbox, ./data/jobs, ...) as root:root — running
|
|
||||||
# sftpgo as root too avoids a uid mismatch on the shared ./data tree.
|
|
||||||
user: root
|
|
||||||
entrypoint: ["/bin/sh", "/entrypoint.sh"]
|
|
||||||
environment:
|
|
||||||
SFTP_USER: ${SFTP_USER:-livef12}
|
|
||||||
SFTP_PASSWORD: ${SFTP_PASSWORD:-changeme}
|
|
||||||
# Minutes. Default SFTPGo is 15 — too aggressive for Android clients
|
|
||||||
# that idle after a successful upload (shows up as EOF in the logs).
|
|
||||||
SFTPGO_COMMON__IDLE_TIMEOUT: ${SFTPGO_IDLE_TIMEOUT:-120}
|
|
||||||
volumes:
|
|
||||||
- ${DATA_HOST_DIR:-./data}:/srv/sftpgo/data
|
|
||||||
- sftpgo_state:/var/lib/sftpgo
|
|
||||||
- ./sftpgo/entrypoint.sh:/entrypoint.sh:ro
|
|
||||||
ports:
|
|
||||||
# SFTP only. The SFTPGo web admin (container port 8080) is
|
|
||||||
# intentionally NOT published here and NOT attached to the
|
|
||||||
# `traefik` network — it stays unreachable from outside docker.
|
|
||||||
#
|
|
||||||
# Host SFTP port. Spec asked for 121212 which exceeds TCP max
|
|
||||||
# (65535); production default is 12121.
|
|
||||||
- "${SFTP_HOST_PORT:-12121}:2022"
|
|
||||||
networks:
|
|
||||||
- internal
|
|
||||||
|
|
||||||
worker:
|
worker:
|
||||||
build: .
|
build: .
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
command: ["python3", "-m", "app.worker"]
|
command: ["python3", "-m", "app.worker"]
|
||||||
depends_on:
|
|
||||||
- sftpgo
|
|
||||||
environment:
|
environment:
|
||||||
DATA_DIR: /data
|
DATA_DIR: /data
|
||||||
OUTPUT_COUNT: ${OUTPUT_COUNT:-3}
|
OUTPUT_COUNT: ${OUTPUT_COUNT:-3}
|
||||||
@@ -60,7 +27,7 @@ services:
|
|||||||
REMBG_ALPHA: ${REMBG_ALPHA:-1}
|
REMBG_ALPHA: ${REMBG_ALPHA:-1}
|
||||||
NICE_LEVEL: ${NICE_LEVEL:-18}
|
NICE_LEVEL: ${NICE_LEVEL:-18}
|
||||||
volumes:
|
volumes:
|
||||||
- ${DATA_HOST_DIR:-./data}:/data
|
- ${DATA_HOST_DIR:-/home/frank/sync.schwenk.online/data/livef12}:/data
|
||||||
- ./assets:/data/assets:ro
|
- ./assets:/data/assets:ro
|
||||||
- rembg_cache:/app/.home
|
- rembg_cache:/app/.home
|
||||||
# Sequential-only by design (no threads/async in worker.py); these
|
# Sequential-only by design (no threads/async in worker.py); these
|
||||||
@@ -92,7 +59,7 @@ services:
|
|||||||
NICE_LEVEL: ${NICE_LEVEL:-18}
|
NICE_LEVEL: ${NICE_LEVEL:-18}
|
||||||
SITE_TITLE: live.f12.rocks
|
SITE_TITLE: live.f12.rocks
|
||||||
volumes:
|
volumes:
|
||||||
- ${DATA_HOST_DIR:-./data}:/data
|
- ${DATA_HOST_DIR:-/home/frank/sync.schwenk.online/data/livef12}:/data
|
||||||
- ./assets:/data/assets:ro
|
- ./assets:/data/assets:ro
|
||||||
labels:
|
labels:
|
||||||
- "traefik.enable=true"
|
- "traefik.enable=true"
|
||||||
@@ -110,5 +77,4 @@ networks:
|
|||||||
external: true
|
external: true
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
sftpgo_state:
|
|
||||||
rembg_cache:
|
rembg_cache:
|
||||||
|
|||||||
@@ -1,44 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
# Builds a SFTPGo "loaddata" JSON (the format the dumpdata/loaddata REST
|
|
||||||
# API uses) with exactly one user, then starts sftpgo with
|
|
||||||
# --loaddata-from that file. Runs on every container start — mode 0
|
|
||||||
# (the default) adds new objects and updates existing ones, so re-running
|
|
||||||
# this is safe and picks up a changed SFTP_PASSWORD on restart.
|
|
||||||
#
|
|
||||||
# The user's home_dir is /srv/sftpgo/data/inbox, which — via the
|
|
||||||
# ./data:/srv/sftpgo/data bind mount in compose.yml — is the same
|
|
||||||
# directory as the worker's DATA_DIR/inbox. Whatever lands here over SFTP
|
|
||||||
# is exactly what the worker watches.
|
|
||||||
#
|
|
||||||
# Uses `jq` to build the JSON so SFTP_USER/SFTP_PASSWORD are always
|
|
||||||
# correctly escaped (arbitrary passwords, including quotes/backslashes,
|
|
||||||
# are safe). jq ships in the official drakkan/sftpgo image; if a future
|
|
||||||
# image drops it, this script needs an alternative (e.g. python/perl,
|
|
||||||
# both otherwise present at the time of writing).
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
OUTPUT="/var/lib/sftpgo/loaddata.json"
|
|
||||||
|
|
||||||
: "${SFTP_USER:?SFTP_USER must be set}"
|
|
||||||
: "${SFTP_PASSWORD:?SFTP_PASSWORD must be set}"
|
|
||||||
|
|
||||||
jq -n \
|
|
||||||
--arg user "$SFTP_USER" \
|
|
||||||
--arg pass "$SFTP_PASSWORD" \
|
|
||||||
'{
|
|
||||||
users: [
|
|
||||||
{
|
|
||||||
status: 1,
|
|
||||||
username: $user,
|
|
||||||
password: $pass,
|
|
||||||
home_dir: "/srv/sftpgo/data/inbox",
|
|
||||||
permissions: {
|
|
||||||
"/": ["list", "download", "upload", "overwrite", "create_dirs"]
|
|
||||||
},
|
|
||||||
filesystem: { provider: 0 }
|
|
||||||
}
|
|
||||||
],
|
|
||||||
version: 15
|
|
||||||
}' > "$OUTPUT"
|
|
||||||
|
|
||||||
exec sftpgo serve --loaddata-from "$OUTPUT"
|
|
||||||
Reference in New Issue
Block a user