imagepipeline/docs/ARCHITECTURE.md

Architecture

Imagepipeline is a small Python framework for defining batch image pipelines as DAGs. Each step is a registered module; each run writes numbered folders under a timestamped output root.

Layout

imagepipeline/
├── core/           # Pipeline, runner, resume, manifest, params, logging
├── modules/        # Processing steps (@register)
├── utils/          # files, subprocess, gmic, gimp helpers
├── ai/             # Optional torch models (HDRNet, Zero-DCE)
└── cli.py          # `imagepipeline list-modules`

pipelines/          # Runnable scripts (machine-local INPUT paths)
tests/              # pytest suite
docs/               # Developer docs
workflows/comfy/    # ComfyUI workflow JSON (optional AI path)

Execution flow

1. Define — Pipeline(name=..., input_dir=...) collects step() calls (StepDefinition DAG).
2. Run — PipelineRunner topologically sorts steps, matches inputs by filename stem, builds ModuleContext.
3. Resume — continue_from / existing_outputs reuse prior run folders; modules declare expected_output_filenames when extensions change.
4. Manifest — pipeline_manifest.json records steps, params, and paths.

Module contract

Every module subclasses BaseModule or SubprocessModule:

• name, description, parameters() schema
• run(ctx: ModuleContext) writes into ctx.output_dir
• Optional expected_output_filenames() for resume when output names differ from inputs
• check_dependencies() for external CLI tools

Registration happens via @register and eager import in imagepipeline/modules/__init__.py.

External tools

Many modules shell out to CLIs (not Python packages):

Tool	Modules
ImageMagick (magick/convert)	imagemagick_*, composite, color_to_alpha, crop_square
G'MIC	gmic, gmic_grayscale
rembg	rembg
darktable-cli	darktable_style
GIMP	xcf_stack

AI modules need pip install -e ".[ai]" (torch, numpy) and optionally API keys in .env.

Design choices

• Plain Python pipelines — no YAML DSL; full control and easy resume constants in script.
• Stem matching — multi-input steps align files by basename across step folders.
• Symlink input — default run copies/symlinks source images into input/ for reproducibility.

See MODULE_DEVELOPMENT.md for adding modules.