docs: expand README and add architecture/contributing notes

Document resume, dependencies, external tools, dev commands, and MIT license.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Frank Schwenk
2026-07-18 17:34:19 +02:00
parent 3443c8826d
commit ffc28914e1
4 changed files with 160 additions and 4 deletions
+36
View File
@@ -0,0 +1,36 @@
# Contributing
## Setup
```bash
pip install -e ".[dev,ai]"
```
Use `.[dev]` only for core tests; AI module tests need `.[ai]` (torch). OpenRouter/Comfy steps need `OPENROUTER_API_KEY` in `.env` (see `.env.example`).
## Code style
- **Ruff** for lint and format (`ruff check .`, `ruff format .`)
- Python **3.11+**, type hints on public APIs
- CLI messages: direct English, no marketing tone (see `SOUL.md`)
## Tests
```bash
pytest # full suite (may call local CLIs)
pytest -m "not integration" # fast subset for CI
```
Mark new external-tool tests with `@pytest.mark.integration`.
## Adding a module
See [MODULE_DEVELOPMENT.md](MODULE_DEVELOPMENT.md). Register in `imagepipeline/modules/__init__.py` and add param/behavior tests.
## Pipelines
Scripts in `pipelines/` are examples and shoot-specific recipes. Machine-local `INPUT` paths are intentional. Reset `CONTINUE_FROM` before sharing a script.
## Commits
Conventional Commits (`feat:`, `fix:`, `chore:`, …). One logical change per commit.
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Frank
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+44 -4
View File
@@ -7,15 +7,19 @@ Each pipeline is a Python script that defines a DAG of processing steps. Every s
## Requirements ## Requirements
- Python 3.11+ - Python 3.11+
- [ImageMagick](https://imagemagick.org/) (`magick` or `convert` on PATH) - [ImageMagick](https://imagemagick.org/) (`magick` or `convert` on PATH) — most pipelines
- Optional CLIs per module: [G'MIC](https://gmic.eu/), [rembg](https://github.com/danielgatis/rembg), [darktable-cli](https://www.darktable.org/), GIMP (`gimp-console`)
## Installation ## Installation
```bash ```bash
cd /path/to/imagepipeline cd /path/to/imagepipeline
pip install -e ".[dev]" pip install -e ".[dev]" # core + tests
pip install -e ".[dev,ai]" # + torch/numpy for AI modules
``` ```
Copy `.env.example` to `.env` and set `OPENROUTER_API_KEY` when using `openrouter_edit` or Comfy-related workflows.
## Quick Start ## Quick Start
Edit the input path in `pipelines/example_grayscale.py`, then run: Edit the input path in `pipelines/example_grayscale.py`, then run:
@@ -35,9 +39,11 @@ with Pipeline(name="my_run", input_dir=Path("/path/to/export")) as p:
p.run() p.run()
``` ```
List registered modules: `imagepipeline list-modules`
## Output Structure ## Output Structure
Each run creates a folder like `my_run_20260527143022/`: Each run creates a folder like `my_run_20260527143022/` under `~/pipeline_output/` (or `output_base`):
``` ```
my_run_20260527143022/ my_run_20260527143022/
@@ -50,6 +56,25 @@ my_run_20260527143022/
Step folders are named `{module_name}_{nn}` by default (two-digit counter per module name). Pass optional `step_id="input_bokeh"` to `p.step()` for a custom folder name and step reference (see [docs/MODULE_DEVELOPMENT.md](docs/MODULE_DEVELOPMENT.md#step-folder-naming)). Step folders are named `{module_name}_{nn}` by default (two-digit counter per module name). Pass optional `step_id="input_bokeh"` to `p.step()` for a custom folder name and step reference (see [docs/MODULE_DEVELOPMENT.md](docs/MODULE_DEVELOPMENT.md#step-folder-naming)).
## Resume
Pipelines support resuming interrupted runs:
```python
CONTINUE_FROM = Path("~/pipeline_output/my_run_260718120000")
EXISTING_OUTPUTS = {"rembg_01": CONTINUE_FROM / "rembg_01"}
with Pipeline(
name="my_run",
input_dir=INPUT,
continue_from=CONTINUE_FROM,
existing_outputs=EXISTING_OUTPUTS,
) as p:
...
```
Modules that change file extensions must implement `expected_output_filenames` so skip logic works (e.g. `rembg``.png`).
## Writing Pipelines ## Writing Pipelines
Pipelines are plain Python scripts. Reference previous steps via `StepRef` objects returned by `p.step()`: Pipelines are plain Python scripts. Reference previous steps via `StepRef` objects returned by `p.step()`:
@@ -67,12 +92,27 @@ with Pipeline(name="colorsplash", input_dir=INPUT) as p:
- Parameters are passed as kwargs and validated against each module's schema - Parameters are passed as kwargs and validated against each module's schema
- Multiple uses of the same module get separate numbered folders - Multiple uses of the same module get separate numbered folders
Declarative building blocks for agents and humans: [RECIPES.md](RECIPES.md).
## Adding Modules ## Adding Modules
See [docs/MODULE_DEVELOPMENT.md](docs/MODULE_DEVELOPMENT.md). See [docs/MODULE_DEVELOPMENT.md](docs/MODULE_DEVELOPMENT.md). Architecture overview: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
## Development
```bash
ruff check .
ruff format .
pytest # full suite (uses local CLIs when present)
pytest -m "not integration" # fast subset (CI default)
```
See [CONTRIBUTING.md](CONTRIBUTING.md).
## Tests ## Tests
```bash ```bash
pytest pytest
``` ```
Optional AI tests require `pip install -e ".[ai]"`.
+59
View File
@@ -0,0 +1,59 @@
# Architecture
Imagepipeline is a small Python framework for defining **batch image pipelines** as DAGs. Each step is a registered module; each run writes numbered folders under a timestamped output root.
## Layout
```
imagepipeline/
├── core/ # Pipeline, runner, resume, manifest, params, logging
├── modules/ # Processing steps (@register)
├── utils/ # files, subprocess, gmic, gimp helpers
├── ai/ # Optional torch models (HDRNet, Zero-DCE)
└── cli.py # `imagepipeline list-modules`
pipelines/ # Runnable scripts (machine-local INPUT paths)
tests/ # pytest suite
docs/ # Developer docs
workflows/comfy/ # ComfyUI workflow JSON (optional AI path)
```
## Execution flow
1. **Define**`Pipeline(name=..., input_dir=...)` collects `step()` calls (`StepDefinition` DAG).
2. **Run**`PipelineRunner` topologically sorts steps, matches inputs by filename stem, builds `ModuleContext`.
3. **Resume**`continue_from` / `existing_outputs` reuse prior run folders; modules declare `expected_output_filenames` when extensions change.
4. **Manifest**`pipeline_manifest.json` records steps, params, and paths.
## Module contract
Every module subclasses `BaseModule` or `SubprocessModule`:
- `name`, `description`, `parameters()` schema
- `run(ctx: ModuleContext)` writes into `ctx.output_dir`
- Optional `expected_output_filenames()` for resume when output names differ from inputs
- `check_dependencies()` for external CLI tools
Registration happens via `@register` and eager import in `imagepipeline/modules/__init__.py`.
## External tools
Many modules shell out to CLIs (not Python packages):
| Tool | Modules |
|------|---------|
| ImageMagick (`magick`/`convert`) | `imagemagick_*`, `composite`, `color_to_alpha`, `crop_square` |
| G'MIC | `gmic`, `gmic_grayscale` |
| rembg | `rembg` |
| darktable-cli | `darktable_style` |
| GIMP | `xcf_stack` |
AI modules need `pip install -e ".[ai]"` (torch, numpy) and optionally API keys in `.env`.
## Design choices
- **Plain Python pipelines** — no YAML DSL; full control and easy resume constants in script.
- **Stem matching** — multi-input steps align files by basename across step folders.
- **Symlink input** — default run copies/symlinks source images into `input/` for reproducibility.
See [MODULE_DEVELOPMENT.md](MODULE_DEVELOPMENT.md) for adding modules.