# Architecture

Imagepipeline is a small Python framework for defining **batch image pipelines** as DAGs. Each step is a registered module; each run writes numbered folders under a timestamped output root.

## Layout

```
imagepipeline/
├── core/           # Pipeline, runner, resume, manifest, params, logging
├── modules/        # Processing steps (@register)
├── utils/          # files, subprocess, gmic, gimp helpers
├── ai/             # Optional torch models (HDRNet, Zero-DCE)
└── cli.py          # `imagepipeline list-modules`

pipelines/          # Runnable scripts (machine-local INPUT paths)
tests/              # pytest suite
docs/               # Developer docs
workflows/comfy/    # ComfyUI workflow JSON (optional AI path)
```

## Execution flow

1. **Define** — `Pipeline(name=..., input_dir=...)` collects `step()` calls (`StepDefinition` DAG).
2. **Run** — `PipelineRunner` topologically sorts steps, matches inputs by filename stem, builds `ModuleContext`.
3. **Resume** — `continue_from` / `existing_outputs` reuse prior run folders; modules declare `expected_output_filenames` when extensions change.
4. **Manifest** — `pipeline_manifest.json` records steps, params, and paths.

## Module contract

Every module subclasses `BaseModule` or `SubprocessModule`:

- `name`, `description`, `parameters()` schema
- `run(ctx: ModuleContext)` writes into `ctx.output_dir`
- Optional `expected_output_filenames()` for resume when output names differ from inputs
- `check_dependencies()` for external CLI tools

Registration happens via `@register` and eager import in `imagepipeline/modules/__init__.py`.

## External tools

Many modules shell out to CLIs (not Python packages):

| Tool | Modules |
|------|---------|
| ImageMagick (`magick`/`convert`) | `imagemagick_*`, `composite`, `color_to_alpha`, `crop_square` |
| G'MIC | `gmic`, `gmic_grayscale` |
| rembg | `rembg` |
| darktable-cli | `darktable_style` |
| GIMP | `xcf_stack` |

AI modules need `pip install -e ".[ai]"` (torch, numpy) and optionally API keys in `.env`.

## Design choices

- **Plain Python pipelines** — no YAML DSL; full control and easy resume constants in script.
- **Stem matching** — multi-input steps align files by basename across step folders.
- **Symlink input** — default run copies/symlinks source images into `input/` for reproducibility.

See [MODULE_DEVELOPMENT.md](MODULE_DEVELOPMENT.md) for adding modules.