imagepipeline/README.md

Image Pipeline

Modular Python framework for chaining image processing steps after Darktable export.

Each pipeline is a Python script that defines a DAG of processing steps. Every step writes its output to a numbered subfolder inside a timestamped run directory.

Requirements

• Python 3.11+
• ImageMagick (magick or convert on PATH) — most pipelines
• Optional CLIs per module: G'MIC, rembg, darktable-cli, GIMP (gimp-console)

Installation

cd /path/to/imagepipeline
pip install -e ".[dev]"      # core + tests
pip install -e ".[dev,ai]"   # + torch/numpy for AI modules

Copy .env.example to .env and set OPENROUTER_API_KEY when using openrouter_edit or Comfy-related workflows.

Quick Start

Edit the input path in pipelines/example_grayscale.py, then run:

python pipelines/example_grayscale.py

Or from Python:

from pathlib import Path
from imagepipeline import Pipeline

with Pipeline(name="my_run", input_dir=Path("/path/to/export")) as p:
    gray = p.step("imagemagick_grayscale", inputs="input")
    p.run()

List registered modules: imagepipeline list-modules

Output Structure

Each run creates a folder like my_run_20260527143022/ under ~/pipeline_output/ (or output_base):

my_run_20260527143022/
├── pipeline_manifest.json
├── input/                      # symlinks to source images
├── imagemagick_grayscale_01/
│   └── photo.jpg
└── ...

Step folders are named {module_name}_{nn} by default (two-digit counter per module name). Pass optional step_id="input_bokeh" to p.step() for a custom folder name and step reference (see docs/MODULE_DEVELOPMENT.md).

Resume

Pipelines support resuming interrupted runs:

CONTINUE_FROM = Path("~/pipeline_output/my_run_260718120000")
EXISTING_OUTPUTS = {"rembg_01": CONTINUE_FROM / "rembg_01"}

with Pipeline(
    name="my_run",
    input_dir=INPUT,
    continue_from=CONTINUE_FROM,
    existing_outputs=EXISTING_OUTPUTS,
) as p:
    ...

Modules that change file extensions must implement expected_output_filenames so skip logic works (e.g. rembg → .png).

Writing Pipelines

Pipelines are plain Python scripts. Reference previous steps via StepRef objects returned by p.step():

with Pipeline(name="colorsplash", input_dir=INPUT) as p:
    rembg_out = p.step("rembg", inputs="input")
    bw = p.step("imagemagick_grayscale", inputs="input")
    combined = p.step("composite", inputs=[bw, rembg_out], mode="foreground_over")
    p.step("darktable_style", inputs=combined, style="vintage.dtstyle")
    p.run()

• "input" refers to the original input directory
• Parameters are passed as kwargs and validated against each module's schema
• Multiple uses of the same module get separate numbered folders

Declarative building blocks for agents and humans: RECIPES.md.

Adding Modules

See docs/MODULE_DEVELOPMENT.md. Architecture overview: docs/ARCHITECTURE.md.

Development

ruff check .
ruff format .
pytest                      # full suite (uses local CLIs when present)
pytest -m "not integration" # fast subset (CI default)

See CONTRIBUTING.md.

Tests

pytest

Optional AI tests require pip install -e ".[ai]".