Frank Schwenk
ffc28914e1
Document resume, dependencies, external tools, dev commands, and MIT license. Co-authored-by: Cursor <cursoragent@cursor.com>
119 lines
3.5 KiB
Markdown
119 lines
3.5 KiB
Markdown
# Image Pipeline
|
|
|
|
Modular Python framework for chaining image processing steps after Darktable export.
|
|
|
|
Each pipeline is a Python script that defines a DAG of processing steps. Every step writes its output to a numbered subfolder inside a timestamped run directory.
|
|
|
|
## Requirements
|
|
|
|
- Python 3.11+
|
|
- [ImageMagick](https://imagemagick.org/) (`magick` or `convert` on PATH) — most pipelines
|
|
- Optional CLIs per module: [G'MIC](https://gmic.eu/), [rembg](https://github.com/danielgatis/rembg), [darktable-cli](https://www.darktable.org/), GIMP (`gimp-console`)
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
cd /path/to/imagepipeline
|
|
pip install -e ".[dev]" # core + tests
|
|
pip install -e ".[dev,ai]" # + torch/numpy for AI modules
|
|
```
|
|
|
|
Copy `.env.example` to `.env` and set `OPENROUTER_API_KEY` when using `openrouter_edit` or Comfy-related workflows.
|
|
|
|
## Quick Start
|
|
|
|
Edit the input path in `pipelines/example_grayscale.py`, then run:
|
|
|
|
```bash
|
|
python pipelines/example_grayscale.py
|
|
```
|
|
|
|
Or from Python:
|
|
|
|
```python
|
|
from pathlib import Path
|
|
from imagepipeline import Pipeline
|
|
|
|
with Pipeline(name="my_run", input_dir=Path("/path/to/export")) as p:
|
|
gray = p.step("imagemagick_grayscale", inputs="input")
|
|
p.run()
|
|
```
|
|
|
|
List registered modules: `imagepipeline list-modules`
|
|
|
|
## Output Structure
|
|
|
|
Each run creates a folder like `my_run_20260527143022/` under `~/pipeline_output/` (or `output_base`):
|
|
|
|
```
|
|
my_run_20260527143022/
|
|
├── pipeline_manifest.json
|
|
├── input/ # symlinks to source images
|
|
├── imagemagick_grayscale_01/
|
|
│ └── photo.jpg
|
|
└── ...
|
|
```
|
|
|
|
Step folders are named `{module_name}_{nn}` by default (two-digit counter per module name). Pass optional `step_id="input_bokeh"` to `p.step()` for a custom folder name and step reference (see [docs/MODULE_DEVELOPMENT.md](docs/MODULE_DEVELOPMENT.md#step-folder-naming)).
|
|
|
|
## Resume
|
|
|
|
Pipelines support resuming interrupted runs:
|
|
|
|
```python
|
|
CONTINUE_FROM = Path("~/pipeline_output/my_run_260718120000")
|
|
EXISTING_OUTPUTS = {"rembg_01": CONTINUE_FROM / "rembg_01"}
|
|
|
|
with Pipeline(
|
|
name="my_run",
|
|
input_dir=INPUT,
|
|
continue_from=CONTINUE_FROM,
|
|
existing_outputs=EXISTING_OUTPUTS,
|
|
) as p:
|
|
...
|
|
```
|
|
|
|
Modules that change file extensions must implement `expected_output_filenames` so skip logic works (e.g. `rembg` → `.png`).
|
|
|
|
## Writing Pipelines
|
|
|
|
Pipelines are plain Python scripts. Reference previous steps via `StepRef` objects returned by `p.step()`:
|
|
|
|
```python
|
|
with Pipeline(name="colorsplash", input_dir=INPUT) as p:
|
|
rembg_out = p.step("rembg", inputs="input")
|
|
bw = p.step("imagemagick_grayscale", inputs="input")
|
|
combined = p.step("composite", inputs=[bw, rembg_out], mode="foreground_over")
|
|
p.step("darktable_style", inputs=combined, style="vintage.dtstyle")
|
|
p.run()
|
|
```
|
|
|
|
- `"input"` refers to the original input directory
|
|
- Parameters are passed as kwargs and validated against each module's schema
|
|
- Multiple uses of the same module get separate numbered folders
|
|
|
|
Declarative building blocks for agents and humans: [RECIPES.md](RECIPES.md).
|
|
|
|
## Adding Modules
|
|
|
|
See [docs/MODULE_DEVELOPMENT.md](docs/MODULE_DEVELOPMENT.md). Architecture overview: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
|
|
|
|
## Development
|
|
|
|
```bash
|
|
ruff check .
|
|
ruff format .
|
|
pytest # full suite (uses local CLIs when present)
|
|
pytest -m "not integration" # fast subset (CI default)
|
|
```
|
|
|
|
See [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
pytest
|
|
```
|
|
|
|
Optional AI tests require `pip install -e ".[ai]"`.
|