feat: replace SFTP drop with Syncthing share path

Remove SFTPGo; mount event data from the Syncthing folder, watch
incoming/, and name variants after the source stem.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Frank Schwenk
2026-07-18 14:06:57 +02:00
parent 83d4468b69
commit d5b44b221e
13 changed files with 167 additions and 189 deletions
+6 -16
View File
@@ -1,21 +1,11 @@
# Copy to `.env` and fill in real values. Never commit `.env`. # Copy to `.env` and fill in real values. Never commit `.env`.
# --- SFTPGo credentials ----------------------------------------------- # --- Host data path (Syncthing share root) ------------------------------
# Whoever runs the event camera/laptop uploads photos here. # Prod default (compose.yml): /home/frank/sync.schwenk.online/data/livef12
SFTP_USER=livef12 # Layout under that path: incoming/ jobs/ processed.json
SFTP_PASSWORD=changeme # Local override when the absolute path does not exist:
# DATA_HOST_DIR=./data
# --- Host paths / ports -------------------------------------------------- # DATA_HOST_DIR=/home/frank/sync.schwenk.online/data/livef12
# On the server this should be an absolute path, e.g.
# /home/frank/live.f12.rocks/data
DATA_HOST_DIR=./data
# Spec asked for 121212 which exceeds TCP max (65535). Use 12121.
SFTP_HOST_PORT=12121
# Minutes until SFTPGo closes an idle connection. Android/SSHJ clients
# often sit idle after upload; the default 15m shows up as a spurious EOF.
SFTPGO_IDLE_TIMEOUT=120
# --- Pipeline tuning (see make_random.py for background) ----------------- # --- Pipeline tuning (see make_random.py for background) -----------------
OUTPUT_COUNT=3 OUTPUT_COUNT=3
+8 -1
View File
@@ -1,7 +1,7 @@
# Secrets # Secrets
.env .env
# Runtime data (bind-mounted, never belongs in git) # Runtime data (bind-mounted / Syncthing share — never in git)
/data/ /data/
# Local rembg/withoutbg comparison scratch # Local rembg/withoutbg comparison scratch
@@ -14,8 +14,15 @@
# Python # Python
__pycache__/ __pycache__/
*.pyc *.pyc
*.pyo
.venv/ .venv/
.pytest_cache/
.mypy_cache/
.ruff_cache/
# OS / editor noise # OS / editor noise
.DS_Store .DS_Store
*.swp *.swp
*~
.idea/
.vscode/
+10 -3
View File
@@ -4,7 +4,14 @@
- Live on boka: `/home/frank/live.f12.rocks` - Live on boka: `/home/frank/live.f12.rocks`
- Web: https://live.f12.rocks - Web: https://live.f12.rocks
- SFTP: `livef12@frank-schwenk.de` port **12121** (spec 121212 is invalid TCP) - Password / secrets in server `.env` only (`chmod 600`)
- Password in server `.env` only (`chmod 600`); ntfy sent to phone
- Data: `/home/frank/live.f12.rocks/data` (inbox not auto-cleaned)
- First rembg job downloads ~176 MB model into docker volume `rembg_cache` - First rembg job downloads ~176 MB model into docker volume `rembg_cache`
## Syncthing cutover (2026-07-18)
- SFTPGo removed; drop path is Syncthing
- Data share: `/home/frank/sync.schwenk.online/data/livef12`
(`incoming/` + `jobs/` + `processed.json`)
- Old path `/home/frank/live.f12.rocks/data` no longer used by compose
(migration/cleanup manual)
- Host port 12121 can be closed after cutover
+26 -40
View File
@@ -1,9 +1,10 @@
# live.f12.rocks # live.f12.rocks
Event photo pep pipeline: guests upload a photo via SFTP, a worker runs it Event photo pep pipeline: guests drop a photo into a Syncthing folder, a
through `rembg` (background removal) + `gmic` (random filter/blend worker runs it through `rembg` (background removal) + `gmic` (random
compositing, ported from `make_random.py`), and a small web UI shows the filter/blend compositing, ported from `make_random.py`), and a small web
results with a "remix" option to try different filters on the same photo. UI shows the results with a "remix" option to try different filters on
the same photo. Finished jobs sync back via the same share.
No web auth. This is an event tool, not a DAM. No web auth. This is an event tool, not a DAM.
@@ -11,25 +12,26 @@ No web auth. This is an event tool, not a DAM.
| Service | What | Port | | Service | What | Port |
|----------|--------------------------------------------------|--------------------------| |----------|--------------------------------------------------|--------------------------|
| `sftpgo` | SFTP drop point, `drakkan/sftpgo` | `12121` (host) -> `2022` | | `worker` | Watches `incoming/`, runs the compose pipeline | none published |
| `worker` | Watches inbox, runs the compose pipeline | none published |
| `web` | FastAPI + Jinja UI, browse jobs, remix | via Traefik only | | `web` | FastAPI + Jinja UI, browse jobs, remix | via Traefik only |
All three share one host directory (`DATA_HOST_DIR`, default `./data`), Both share one host directory (`DATA_HOST_DIR`) mounted at `/data`.
mounted at different paths — see the comment at the top of `compose.yml`. Syncthing runs on the host — not in this compose.
## Data layout (`./data`) ## Data layout (Syncthing share)
Prod path: `/home/frank/sync.schwenk.online/data/livef12`
``` ```
inbox/ # SFTP drop — worker only ever reads/copies from here incoming/ # phone drop — worker only ever reads/copies from here
jobs/<job_id>/ jobs/<job_id>/
original.<ext> # private copy of the uploaded photo original.<ext> # private copy of the uploaded photo
rembg.png # background removed (computed once, reused) rembg.png # background removed (computed once, reused)
intermediates/ # every intermediate step, kept for inspection intermediates/ # every intermediate step, kept for inspection
variants/ # final composed images variants/ # final composed images ({stem}_v1.png, …)
manifest.json # filter names/commands/blends per variant manifest.json # filter names/commands/blends per variant
status.json # pending | processing | done | error status.json # pending | processing | done | error
processed.json # worker bookkeeping: which inbox files were handled processed.json # worker bookkeeping: which incoming files were handled
``` ```
`assets/` (filter lists + trimmed `filters.json`) lives in the repo and is `assets/` (filter lists + trimmed `filters.json`) lives in the repo and is
@@ -39,27 +41,20 @@ bind-mounted read-only into `worker`/`web` at `/data/assets`.
```bash ```bash
cp .env.example .env cp .env.example .env
# edit .env: set a real SFTP_PASSWORD, and DATA_HOST_DIR on the server # local: set DATA_HOST_DIR=./data
# prod: leave unset (defaults to the Syncthing share path)
docker compose up -d --build docker compose up -d --build
``` ```
Upload a photo: Drop a photo into `incoming/` (via Syncthing or locally). After a few
seconds (poll interval + processing time) it shows up on the web UI as a
```bash new job; the full `jobs/<id>/` tree syncs back to the phone.
sftp -P 12121 livef12@<host>
put photo.jpg
```
After a few seconds (poll interval + processing time) it shows up on the
web UI as a new job.
## Configuration (`.env`, see `.env.example`) ## Configuration (`.env`, see `.env.example`)
| Var | Default | Notes | | Var | Default | Notes |
|-----|---------|-------| |-----|---------|-------|
| `SFTP_USER` / `SFTP_PASSWORD` | `livef12` / `changeme` | SFTP login, home dir is locked to the shared inbox | | `DATA_HOST_DIR` | `/home/frank/sync.schwenk.online/data/livef12` | Syncthing share root; use `./data` locally |
| `DATA_HOST_DIR` | `./data` | On the server: `/home/frank/live.f12.rocks/data` |
| `SFTP_HOST_PORT` | `12121` | Host-side SFTP port (121212 is invalid TCP) |
| `OUTPUT_COUNT` | `3` | Variants generated per uploaded photo | | `OUTPUT_COUNT` | `3` | Variants generated per uploaded photo |
| `BLEND_OPACITY` | `30%` | Default blend opacity for auto-generated variants | | `BLEND_OPACITY` | `30%` | Default blend opacity for auto-generated variants |
| `FILTER_TIMEOUT` | `120` | Seconds before a single gmic call is killed | | `FILTER_TIMEOUT` | `120` | Seconds before a single gmic call is killed |
@@ -68,26 +63,17 @@ web UI as a new job.
## Deploy notes / caveats ## Deploy notes / caveats
- **Port note:** Spec originally said `121212`, which exceeds TCP max - **Firewall:** Traefik must already route `live.f12.rocks` — this repo
(65535). Production default is **`12121`**. only adds the router labels; it assumes the external `traefik` docker
- **Firewall:** open host port `12121` (SFTP) and make sure Traefik network exists. No SFTP port needed.
already routes `live.f12.rocks` — this repo only adds the router labels,
it assumes the external `traefik` docker network exists.
- **SFTPGo first run:** the `sftpgo` service auto-creates the SFTP user
from `SFTP_USER`/`SFTP_PASSWORD` on every start via
`sftpgo/entrypoint.sh` (uses `jq`, bundled in the official image, to
build a `loaddata` JSON safely — no manual admin setup needed). Host SSH
keys persist in the `sftpgo_state` named volume, not in `./data`.
The SFTPGo web admin exists internally on port 8080 but is deliberately
**not** published or routed — there's no need for it here.
- **rembg model download:** first background-removal call downloads the - **rembg model download:** first background-removal call downloads the
`u2net` ONNX model (~176 MB) from GitHub. This needs outbound internet `u2net` ONNX model (~176 MB) from GitHub. This needs outbound internet
on first run and can take a minute or two depending on the link; the on first run and can take a minute or two depending on the link; the
model is cached in the `rembg_cache` named volume afterwards, so model is cached in the `rembg_cache` named volume afterwards, so
restarts don't re-download it. restarts don't re-download it.
- **Inbox is append-only:** the worker only ever copies out of `inbox/` - **Incoming is append-only:** the worker only ever copies out of
and never deletes or moves anything there — plan disk space `incoming/` and never deletes or moves anything there — plan disk space
accordingly, or clean up `inbox/` manually between events. accordingly, or clean up `incoming/` manually between events.
- **Sequential processing:** the worker handles one photo at a time - **Sequential processing:** the worker handles one photo at a time
(`cpus: "1.0"`, no concurrency) — fine for an event pace, but a burst of (`cpus: "1.0"`, no concurrency) — fine for an event pace, but a burst of
uploads will just queue up and get processed in order. uploads will just queue up and get processed in order.
+22 -22
View File
@@ -6,23 +6,24 @@
## One-Liner ## One-Liner
Event photo pep pipeline: SFTP -> gmic/rembg -> web. Upload a photo at the Event photo pep pipeline: Syncthing drop -> gmic/rembg -> web. Upload a
event, get back weird/fun filtered variants seconds later. photo at the event, get back weird/fun filtered variants seconds later.
## Vision ## Vision
At an event (f12 meetup/party), someone drops a photo into an SFTP inbox At an event (f12 meetup/party), someone drops a photo into a Syncthing
from a laptop/camera rig. A worker rips the background off, runs it folder from a phone/camera rig. A worker rips the background off, runs it
through a random gmic filter/blend chain, and a handful of variants show through a random gmic filter/blend chain, and a handful of variants show
up on a shared web page almost immediately — no app install, no login, no up on a shared web page almost immediately — no app install, no login, no
waiting for someone to "process the photos later". If a variant is fun, waiting for someone to "process the photos later". If a variant is fun,
remix it with different filters right there. remix it with different filters right there. Finished jobs sync back to
the phone via the same Syncthing share.
## Audience ## Audience
Event organizers and attendees at f12 events, on their phones, often on Event organizers and attendees at f12 events, on their phones, often on
bad venue wifi. Nobody signs up, nobody logs in. Whoever has the link bad venue wifi. Nobody signs up, nobody logs in. Whoever has the link
(and whoever's uploading from the SFTP side) is the whole audience. (and whoever's syncing photos in) is the whole audience.
## Tone & Wording ## Tone & Wording
@@ -41,7 +42,7 @@ bad venue wifi. Nobody signs up, nobody logs in. Whoever has the link
|---------|------|-----| |---------|------|-----|
| Success | `Remix erzeugt.` | `Dein Kunstwerk ist fertig! 🎉` | | Success | `Remix erzeugt.` | `Dein Kunstwerk ist fertig! 🎉` |
| Error | `Foreground-Filter fehlgeschlagen: Timeout nach 120s` | `Etwas ist schiefgelaufen.` | | Error | `Foreground-Filter fehlgeschlagen: Timeout nach 120s` | `Etwas ist schiefgelaufen.` |
| Empty state | `Noch keine Jobs. Bild per SFTP hochladen, dann kurz warten.` | `Hier ist noch nichts los... lade doch was hoch! 😊` | | Empty state | `Noch keine Jobs. Bild in den Sync-Ordner legen, dann kurz warten.` | `Hier ist noch nichts los... lade doch was hoch! 😊` |
## Design ## Design
@@ -55,33 +56,32 @@ bad venue wifi. Nobody signs up, nobody logs in. Whoever has the link
- No web auth / accounts / login product — anyone with the link sees - No web auth / accounts / login product — anyone with the link sees
everything, by design, for this event tool. everything, by design, for this event tool.
- No camera-to-phone client — upload path is SFTP only. - No camera-to-phone client in this repo — upload path is Syncthing
(phone/folder setup is out of band).
- No beamer/projector product built into this repo. - No beamer/projector product built into this repo.
- Not a DAM (digital asset manager) — no albums, tagging, search, - Not a DAM (digital asset manager) — no albums, tagging, search,
retention policies. `inbox/` and `jobs/` are the whole data model. retention policies. `incoming/` and `jobs/` are the whole data model.
- No auto-deletion of inbox uploads. Ever. The worker only reads/copies. - No auto-deletion of incoming uploads. Ever. The worker only reads/copies.
- No Syncthing container in this compose — Syncthing runs on the host.
## Infrastructure (project-local) ## Infrastructure (project-local)
- Workdir on server: `/home/frank/live.f12.rocks` - Workdir on server: `/home/frank/live.f12.rocks`
- Data volume: `${DATA_HOST_DIR}` (default `./data` locally, absolute - Data volume (Syncthing share): `${DATA_HOST_DIR}` — prod default
path `/home/frank/live.f12.rocks/data` on the server) — bind-mounted `/home/frank/sync.schwenk.online/data/livef12`, local override `./data`.
into all three services (`sftpgo`, `worker`, `web`) at different Bind-mounted into `worker` and `web` at `/data` (see `compose.yml`).
container paths, see `compose.yml` header comment. - Layout under the share: `incoming/` (phone drop), `jobs/` (full tree
- SFTP: host port `12121` -> `sftpgo:2022` (spec `121212` is not a valid syncs back), `processed.json`.
TCP port). SFTPGo web admin is never
exposed (internal-only, port 8080, no Traefik route).
- Domain: `live.f12.rocks`, routed via the shared external `traefik` - Domain: `live.f12.rocks`, routed via the shared external `traefik`
network, TLS via `myresolver` (see `INFRASTRUCTURE.md`). network, TLS via `myresolver` (see `INFRASTRUCTURE.md`).
- rembg model cache and SFTPGo host keys live in named Docker volumes - rembg model cache lives in the named Docker volume `rembg_cache`, not
(`rembg_cache`, `sftpgo_state`), not under `./data` — they're under the Syncthing share — container state, not event data.
container state, not event data.
## Project-Specific Rules ## Project-Specific Rules
```markdown ```markdown
- Never delete or move files under inbox/ from worker code — SFTP is the - Never delete or move files under incoming/ from worker code — Syncthing
only writer there. (reason: uploads are the one copy of the original is the writer there. (reason: uploads are the one copy of the original
that exists outside a phone's camera roll) that exists outside a phone's camera roll)
- Keep every intermediate image in jobs/<id>/intermediates/ — don't clean - Keep every intermediate image in jobs/<id>/intermediates/ — don't clean
them up automatically. (reason: useful for debugging bad filter picks, them up automatically. (reason: useful for debugging bad filter picks,
+1 -1
View File
@@ -1 +1 @@
"""live.f12.rocks — event photo pipeline (SFTP inbox -> gmic/rembg -> web).""" """live.f12.rocks — event photo pipeline (Syncthing drop -> gmic/rembg -> web)."""
+1 -1
View File
@@ -26,7 +26,7 @@ def _float_env(name: str, default: float) -> float:
# --- Paths ------------------------------------------------------------- # --- Paths -------------------------------------------------------------
DATA_DIR = Path(os.environ.get("DATA_DIR", "/data")) DATA_DIR = Path(os.environ.get("DATA_DIR", "/data"))
INBOX_DIR = DATA_DIR / "inbox" INBOX_DIR = DATA_DIR / "incoming"
JOBS_DIR = DATA_DIR / "jobs" JOBS_DIR = DATA_DIR / "jobs"
ASSETS_DIR = DATA_DIR / "assets" ASSETS_DIR = DATA_DIR / "assets"
PROCESSED_FILE = DATA_DIR / "processed.json" PROCESSED_FILE = DATA_DIR / "processed.json"
+59 -9
View File
@@ -210,7 +210,7 @@ def preprocess_original(paths: JobPaths) -> JobPaths:
"""Downscale to MAX_EDGE_PX and normalize to sRGB JPEG as `original.jpg`. """Downscale to MAX_EDGE_PX and normalize to sRGB JPEG as `original.jpg`.
Replaces any non-jpg original in the job dir. Side-effect free for the Replaces any non-jpg original in the job dir. Side-effect free for the
inbox copy — only mutates jobs/<id>/. incoming copy — only mutates jobs/<id>/.
""" """
src = paths.original src = paths.original
if not src.exists(): if not src.exists():
@@ -371,12 +371,37 @@ def next_variant_id(manifest: dict[str, Any]) -> str:
return f"v{i}" return f"v{i}"
_UNSAFE_STEM_RE = re.compile(r"[^\w.\-]+", re.UNICODE)
def sanitize_stem(name: str) -> str:
"""Safe basename stem for variant files (no path separators / junk)."""
stem = Path(name).stem if name else ""
stem = stem.replace("/", "_").replace("\\", "_").strip().strip(".")
stem = _UNSAFE_STEM_RE.sub("_", stem).strip("._")
return stem or "photo"
def variant_filename(stem: str, variant_id: str) -> str:
return f"{sanitize_stem(stem)}_{variant_id}.png"
def resolve_source_stem(manifest: dict[str, Any] | None, paths: JobPaths) -> str:
"""Prefer manifest source_stem; fall back to original path stem."""
if manifest:
stored = manifest.get("source_stem")
if isinstance(stored, str) and stored.strip():
return sanitize_stem(stored)
return sanitize_stem(paths.original.stem)
def compose_variant( def compose_variant(
paths: JobPaths, paths: JobPaths,
assets: FilterAssets, assets: FilterAssets,
*, *,
variant_id: str, variant_id: str,
source: str, source: str,
variant_stem: str,
bg_name: str | None = None, bg_name: str | None = None,
bg_mode: str | None = None, bg_mode: str | None = None,
fg_name: str | None = None, fg_name: str | None = None,
@@ -389,10 +414,13 @@ def compose_variant(
If bg_name/fg_name/bg_mode/fg_mode are given (remix path) they are used If bg_name/fg_name/bg_mode/fg_mode are given (remix path) they are used
directly. Otherwise a random working filter is picked with retries, directly. Otherwise a random working filter is picked with retries,
exactly like make_random.py's compose_one(). exactly like make_random.py's compose_one().
Final file is `variants/{stem}_{variant_id}.png`; manifest id stays `vN`.
""" """
rng = rng or random.Random() rng = rng or random.Random()
tmp_dir = paths.intermediates tmp_dir = paths.intermediates
tmp_dir.mkdir(parents=True, exist_ok=True) tmp_dir.mkdir(parents=True, exist_ok=True)
stem = sanitize_stem(variant_stem)
bg_mode = bg_mode or rng.choice(assets.blend_modes) bg_mode = bg_mode or rng.choice(assets.blend_modes)
fg_mode = fg_mode or rng.choice(assets.blend_modes) fg_mode = fg_mode or rng.choice(assets.blend_modes)
@@ -421,7 +449,7 @@ def compose_variant(
"step2": tmp_dir / f"{variant_id}_rembg_alpha.png", "step2": tmp_dir / f"{variant_id}_rembg_alpha.png",
"fg_filtered": tmp_dir / f"{variant_id}_fg_filtered.png", "fg_filtered": tmp_dir / f"{variant_id}_fg_filtered.png",
"composed": tmp_dir / f"{variant_id}_composed.png", "composed": tmp_dir / f"{variant_id}_composed.png",
"final": paths.variants / f"{variant_id}.png", "final": paths.variants / variant_filename(stem, variant_id),
} }
paths.variants.mkdir(parents=True, exist_ok=True) paths.variants.mkdir(parents=True, exist_ok=True)
@@ -467,13 +495,19 @@ def compose_variant(
} }
def process_job(job_id: str, source_path: Path, original_suffix: str) -> None: def process_job(
"""Full pipeline for a freshly ingested inbox file: preprocess, rembg job_id: str,
source_path: Path,
original_suffix: str,
*,
source_stem: str | None = None,
) -> None:
"""Full pipeline for a freshly ingested incoming file: preprocess, rembg
once, generate OUTPUT_COUNT variants, write manifest + status. once, generate OUTPUT_COUNT variants, write manifest + status.
`source_path` must already be a private copy (jobs/<id>/original.*) — `source_path` must already be a private copy (jobs/<id>/original.*) —
callers (worker.py) are responsible for copying out of inbox first, so callers (worker.py) are responsible for copying out of incoming first, so
the inbox file itself is never touched here. the incoming file itself is never touched here.
Variants that hit FILTER_TIMEOUT during the normal pass are retried Variants that hit FILTER_TIMEOUT during the normal pass are retried
once at the end with FILTER_TIMEOUT_LONG. Filter probes in once at the end with FILTER_TIMEOUT_LONG. Filter probes in
@@ -483,8 +517,9 @@ def process_job(job_id: str, source_path: Path, original_suffix: str) -> None:
paths.root.mkdir(parents=True, exist_ok=True) paths.root.mkdir(parents=True, exist_ok=True)
paths.intermediates.mkdir(parents=True, exist_ok=True) paths.intermediates.mkdir(parents=True, exist_ok=True)
paths.variants.mkdir(parents=True, exist_ok=True) paths.variants.mkdir(parents=True, exist_ok=True)
stem = sanitize_stem(source_stem or source_path.stem)
write_status(paths, "processing", source_file=str(source_path.name)) write_status(paths, "processing", source_file=str(source_path.name), source_stem=stem)
try: try:
paths = preprocess_original(paths) paths = preprocess_original(paths)
@@ -493,6 +528,7 @@ def process_job(job_id: str, source_path: Path, original_suffix: str) -> None:
"job_id": job_id, "job_id": job_id,
"original_file": paths.original.name, "original_file": paths.original.name,
"rembg_file": paths.rembg.name, "rembg_file": paths.rembg.name,
"source_stem": stem,
"created_at": now_iso(), "created_at": now_iso(),
"variants": [], "variants": [],
} }
@@ -509,7 +545,14 @@ def process_job(job_id: str, source_path: Path, original_suffix: str) -> None:
for i in range(1, config.OUTPUT_COUNT + 1): for i in range(1, config.OUTPUT_COUNT + 1):
variant_id = f"v{i}" variant_id = f"v{i}"
try: try:
entry = compose_variant(paths, assets, variant_id=variant_id, source="auto", rng=rng) entry = compose_variant(
paths,
assets,
variant_id=variant_id,
source="auto",
variant_stem=stem,
rng=rng,
)
manifest["variants"].append(entry) manifest["variants"].append(entry)
write_manifest(paths, manifest) write_manifest(paths, manifest)
except FilterTimeoutError as exc: except FilterTimeoutError as exc:
@@ -530,7 +573,14 @@ def process_job(job_id: str, source_path: Path, original_suffix: str) -> None:
with filter_timeout(config.FILTER_TIMEOUT_LONG): with filter_timeout(config.FILTER_TIMEOUT_LONG):
for variant_id in timeout_retries: for variant_id in timeout_retries:
try: try:
entry = compose_variant(paths, assets, variant_id=variant_id, source="auto", rng=rng) entry = compose_variant(
paths,
assets,
variant_id=variant_id,
source="auto",
variant_stem=stem,
rng=rng,
)
manifest["variants"].append(entry) manifest["variants"].append(entry)
write_manifest(paths, manifest) write_manifest(paths, manifest)
except PipelineError as exc: except PipelineError as exc:
+3
View File
@@ -49,6 +49,8 @@ def create_remix_variant(job_id: str, original_suffix: str, choice: RemixChoice)
"created_at": pipeline.now_iso(), "created_at": pipeline.now_iso(),
"variants": [], "variants": [],
} }
variant_stem = pipeline.resolve_source_stem(manifest, paths)
manifest.setdefault("source_stem", variant_stem)
variant_id = pipeline.next_variant_id(manifest) variant_id = pipeline.next_variant_id(manifest)
entry = pipeline.compose_variant( entry = pipeline.compose_variant(
@@ -56,6 +58,7 @@ def create_remix_variant(job_id: str, original_suffix: str, choice: RemixChoice)
assets, assets,
variant_id=variant_id, variant_id=variant_id,
source="remix", source="remix",
variant_stem=variant_stem,
bg_name=choice.bg_filter, bg_name=choice.bg_filter,
bg_mode=choice.bg_blend, bg_mode=choice.bg_blend,
fg_name=choice.fg_filter, fg_name=choice.fg_filter,
+1 -1
View File
@@ -2,7 +2,7 @@
{% block title %}{{ site_title }}{% endblock %} {% block title %}{{ site_title }}{% endblock %}
{% block content %} {% block content %}
{% if not jobs %} {% if not jobs %}
<p class="empty-state">Noch keine Jobs. Bild per SFTP hochladen, dann kurz warten.</p> <p class="empty-state">Noch keine Jobs. Bild in den Sync-Ordner legen, dann kurz warten.</p>
{% else %} {% else %}
<ul class="job-grid"> <ul class="job-grid">
{% for job in jobs %} {% for job in jobs %}
+21 -8
View File
@@ -1,8 +1,8 @@
"""Inbox watcher + sequential job runner. """Incoming watcher + sequential job runner.
Polls `DATA_DIR/inbox` for new, size-stable image files, copies each one Polls `DATA_DIR/incoming` for new, size-stable image files, copies each one
into its own `jobs/<job_id>/original.*` and runs the compose pipeline on into its own `jobs/<job_id>/original.*` and runs the compose pipeline on
it. Files are NEVER deleted or moved from the inbox — `processed.json` it. Files are NEVER deleted or moved from incoming — `processed.json`
tracks what has already been handled (by path + size + mtime) so restarts tracks what has already been handled (by path + size + mtime) so restarts
don't reprocess everything. don't reprocess everything.
@@ -61,9 +61,18 @@ def _is_already_processed(processed: dict[str, Any], path: Path, stat: Any) -> b
return entry.get("size") == stat.st_size and entry.get("mtime") == stat.st_mtime return entry.get("size") == stat.st_size and entry.get("mtime") == stat.st_mtime
def _is_ignored_name(name: str) -> bool:
"""Skip Syncthing metadata/temp files and other dotfiles."""
if name.startswith("."):
return True
if name.startswith(".syncthing.") or ".syncthing." in name:
return True
return False
def _is_stable(path: Path) -> bool: def _is_stable(path: Path) -> bool:
"""A file is "stable" if its size doesn't change across a short wait — """A file is "stable" if its size doesn't change across a short wait —
cheap way to avoid picking up a half-uploaded SFTP transfer.""" cheap way to avoid picking up a half-written Syncthing transfer."""
try: try:
size_before = path.stat().st_size size_before = path.stat().st_size
except OSError: except OSError:
@@ -77,12 +86,15 @@ def _is_stable(path: Path) -> bool:
def find_new_files(processed: dict[str, Any]) -> list[Path]: def find_new_files(processed: dict[str, Any]) -> list[Path]:
"""Top-level images under incoming/ only — never recurse into jobs/."""
if not config.INBOX_DIR.exists(): if not config.INBOX_DIR.exists():
return [] return []
candidates: list[Path] = [] candidates: list[Path] = []
for path in sorted(config.INBOX_DIR.rglob("*")): for path in sorted(config.INBOX_DIR.iterdir()):
if not path.is_file(): if not path.is_file():
continue continue
if _is_ignored_name(path.name):
continue
if path.suffix.lower() not in config.SUPPORTED_EXTENSIONS: if path.suffix.lower() not in config.SUPPORTED_EXTENSIONS:
continue continue
try: try:
@@ -105,13 +117,14 @@ def handle_file(path: Path, processed: dict[str, Any]) -> None:
suffix = path.suffix.lower() suffix = path.suffix.lower()
paths = pipeline.job_paths(job_id, suffix) paths = pipeline.job_paths(job_id, suffix)
paths.root.mkdir(parents=True, exist_ok=True) paths.root.mkdir(parents=True, exist_ok=True)
source_stem = pipeline.sanitize_stem(path.stem)
logger.info("new inbox file %s -> job %s", path.name, job_id) logger.info("new incoming file %s -> job %s (stem=%s)", path.name, job_id, source_stem)
# Copy (never move) so the inbox stays untouched. # Copy (never move) so incoming stays untouched.
shutil.copy2(path, paths.original) shutil.copy2(path, paths.original)
try: try:
pipeline.process_job(job_id, paths.original, suffix) pipeline.process_job(job_id, paths.original, suffix, source_stem=source_stem)
finally: finally:
# Mark as processed regardless of pipeline outcome so a permanently # Mark as processed regardless of pipeline outcome so a permanently
# broken image doesn't get retried forever; failures are visible in # broken image doesn't get retried forever; failures are visible in
+9 -43
View File
@@ -1,51 +1,18 @@
# live.f12.rocks — event photo pipeline: SFTP inbox -> gmic/rembg -> web. # live.f12.rocks — event photo pipeline: Syncthing drop -> gmic/rembg -> web.
# #
# All three services share one host directory (default ./data, see # worker + web share one host directory (Syncthing share root):
# .env.example DATA_HOST_DIR) at different mount points: # ${DATA_HOST_DIR:-/home/frank/sync.schwenk.online/data/livef12} -> /data
# sftpgo -> /srv/sftpgo/data (SFTP user home = /srv/sftpgo/data/inbox) # worker watches /data/incoming, writes /data/jobs
# worker -> /data (watches /data/inbox, writes /data/jobs) # web reads jobs, writes remix variants
# web -> /data (reads jobs, writes remix variants)
# #
# Copy .env.example to .env and fill in real SFTP credentials before # Syncthing itself runs on the host (not in this compose). Copy .env.example
# running this. Never commit .env. # to .env for local overrides (e.g. DATA_HOST_DIR=./data). Never commit .env.
services: services:
sftpgo:
image: drakkan/sftpgo:v2
restart: unless-stopped
# The image defaults to a fixed uid 1000 user. worker/web run as root
# (see Dockerfile), and Docker itself creates first-run bind-mount
# directories (./data/inbox, ./data/jobs, ...) as root:root — running
# sftpgo as root too avoids a uid mismatch on the shared ./data tree.
user: root
entrypoint: ["/bin/sh", "/entrypoint.sh"]
environment:
SFTP_USER: ${SFTP_USER:-livef12}
SFTP_PASSWORD: ${SFTP_PASSWORD:-changeme}
# Minutes. Default SFTPGo is 15 — too aggressive for Android clients
# that idle after a successful upload (shows up as EOF in the logs).
SFTPGO_COMMON__IDLE_TIMEOUT: ${SFTPGO_IDLE_TIMEOUT:-120}
volumes:
- ${DATA_HOST_DIR:-./data}:/srv/sftpgo/data
- sftpgo_state:/var/lib/sftpgo
- ./sftpgo/entrypoint.sh:/entrypoint.sh:ro
ports:
# SFTP only. The SFTPGo web admin (container port 8080) is
# intentionally NOT published here and NOT attached to the
# `traefik` network — it stays unreachable from outside docker.
#
# Host SFTP port. Spec asked for 121212 which exceeds TCP max
# (65535); production default is 12121.
- "${SFTP_HOST_PORT:-12121}:2022"
networks:
- internal
worker: worker:
build: . build: .
restart: unless-stopped restart: unless-stopped
command: ["python3", "-m", "app.worker"] command: ["python3", "-m", "app.worker"]
depends_on:
- sftpgo
environment: environment:
DATA_DIR: /data DATA_DIR: /data
OUTPUT_COUNT: ${OUTPUT_COUNT:-3} OUTPUT_COUNT: ${OUTPUT_COUNT:-3}
@@ -60,7 +27,7 @@ services:
REMBG_ALPHA: ${REMBG_ALPHA:-1} REMBG_ALPHA: ${REMBG_ALPHA:-1}
NICE_LEVEL: ${NICE_LEVEL:-18} NICE_LEVEL: ${NICE_LEVEL:-18}
volumes: volumes:
- ${DATA_HOST_DIR:-./data}:/data - ${DATA_HOST_DIR:-/home/frank/sync.schwenk.online/data/livef12}:/data
- ./assets:/data/assets:ro - ./assets:/data/assets:ro
- rembg_cache:/app/.home - rembg_cache:/app/.home
# Sequential-only by design (no threads/async in worker.py); these # Sequential-only by design (no threads/async in worker.py); these
@@ -92,7 +59,7 @@ services:
NICE_LEVEL: ${NICE_LEVEL:-18} NICE_LEVEL: ${NICE_LEVEL:-18}
SITE_TITLE: live.f12.rocks SITE_TITLE: live.f12.rocks
volumes: volumes:
- ${DATA_HOST_DIR:-./data}:/data - ${DATA_HOST_DIR:-/home/frank/sync.schwenk.online/data/livef12}:/data
- ./assets:/data/assets:ro - ./assets:/data/assets:ro
labels: labels:
- "traefik.enable=true" - "traefik.enable=true"
@@ -110,5 +77,4 @@ networks:
external: true external: true
volumes: volumes:
sftpgo_state:
rembg_cache: rembg_cache:
-44
View File
@@ -1,44 +0,0 @@
#!/bin/sh
# Builds a SFTPGo "loaddata" JSON (the format the dumpdata/loaddata REST
# API uses) with exactly one user, then starts sftpgo with
# --loaddata-from that file. Runs on every container start — mode 0
# (the default) adds new objects and updates existing ones, so re-running
# this is safe and picks up a changed SFTP_PASSWORD on restart.
#
# The user's home_dir is /srv/sftpgo/data/inbox, which — via the
# ./data:/srv/sftpgo/data bind mount in compose.yml — is the same
# directory as the worker's DATA_DIR/inbox. Whatever lands here over SFTP
# is exactly what the worker watches.
#
# Uses `jq` to build the JSON so SFTP_USER/SFTP_PASSWORD are always
# correctly escaped (arbitrary passwords, including quotes/backslashes,
# are safe). jq ships in the official drakkan/sftpgo image; if a future
# image drops it, this script needs an alternative (e.g. python/perl,
# both otherwise present at the time of writing).
set -eu
OUTPUT="/var/lib/sftpgo/loaddata.json"
: "${SFTP_USER:?SFTP_USER must be set}"
: "${SFTP_PASSWORD:?SFTP_PASSWORD must be set}"
jq -n \
--arg user "$SFTP_USER" \
--arg pass "$SFTP_PASSWORD" \
'{
users: [
{
status: 1,
username: $user,
password: $pass,
home_dir: "/srv/sftpgo/data/inbox",
permissions: {
"/": ["list", "download", "upload", "overwrite", "create_dirs"]
},
filesystem: { provider: 0 }
}
],
version: 15
}' > "$OUTPUT"
exec sftpgo serve --loaddata-from "$OUTPUT"