Frank Schwenk
ffc28914e1
Document resume, dependencies, external tools, dev commands, and MIT license. Co-authored-by: Cursor <cursoragent@cursor.com>
60 lines
2.4 KiB
Markdown
60 lines
2.4 KiB
Markdown
# Architecture
|
|
|
|
Imagepipeline is a small Python framework for defining **batch image pipelines** as DAGs. Each step is a registered module; each run writes numbered folders under a timestamped output root.
|
|
|
|
## Layout
|
|
|
|
```
|
|
imagepipeline/
|
|
├── core/ # Pipeline, runner, resume, manifest, params, logging
|
|
├── modules/ # Processing steps (@register)
|
|
├── utils/ # files, subprocess, gmic, gimp helpers
|
|
├── ai/ # Optional torch models (HDRNet, Zero-DCE)
|
|
└── cli.py # `imagepipeline list-modules`
|
|
|
|
pipelines/ # Runnable scripts (machine-local INPUT paths)
|
|
tests/ # pytest suite
|
|
docs/ # Developer docs
|
|
workflows/comfy/ # ComfyUI workflow JSON (optional AI path)
|
|
```
|
|
|
|
## Execution flow
|
|
|
|
1. **Define** — `Pipeline(name=..., input_dir=...)` collects `step()` calls (`StepDefinition` DAG).
|
|
2. **Run** — `PipelineRunner` topologically sorts steps, matches inputs by filename stem, builds `ModuleContext`.
|
|
3. **Resume** — `continue_from` / `existing_outputs` reuse prior run folders; modules declare `expected_output_filenames` when extensions change.
|
|
4. **Manifest** — `pipeline_manifest.json` records steps, params, and paths.
|
|
|
|
## Module contract
|
|
|
|
Every module subclasses `BaseModule` or `SubprocessModule`:
|
|
|
|
- `name`, `description`, `parameters()` schema
|
|
- `run(ctx: ModuleContext)` writes into `ctx.output_dir`
|
|
- Optional `expected_output_filenames()` for resume when output names differ from inputs
|
|
- `check_dependencies()` for external CLI tools
|
|
|
|
Registration happens via `@register` and eager import in `imagepipeline/modules/__init__.py`.
|
|
|
|
## External tools
|
|
|
|
Many modules shell out to CLIs (not Python packages):
|
|
|
|
| Tool | Modules |
|
|
|------|---------|
|
|
| ImageMagick (`magick`/`convert`) | `imagemagick_*`, `composite`, `color_to_alpha`, `crop_square` |
|
|
| G'MIC | `gmic`, `gmic_grayscale` |
|
|
| rembg | `rembg` |
|
|
| darktable-cli | `darktable_style` |
|
|
| GIMP | `xcf_stack` |
|
|
|
|
AI modules need `pip install -e ".[ai]"` (torch, numpy) and optionally API keys in `.env`.
|
|
|
|
## Design choices
|
|
|
|
- **Plain Python pipelines** — no YAML DSL; full control and easy resume constants in script.
|
|
- **Stem matching** — multi-input steps align files by basename across step folders.
|
|
- **Symlink input** — default run copies/symlinks source images into `input/` for reproducibility.
|
|
|
|
See [MODULE_DEVELOPMENT.md](MODULE_DEVELOPMENT.md) for adding modules.
|