diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..1fc5e98 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,36 @@ +# Contributing + +## Setup + +```bash +pip install -e ".[dev,ai]" +``` + +Use `.[dev]` only for core tests; AI module tests need `.[ai]` (torch). OpenRouter/Comfy steps need `OPENROUTER_API_KEY` in `.env` (see `.env.example`). + +## Code style + +- **Ruff** for lint and format (`ruff check .`, `ruff format .`) +- Python **3.11+**, type hints on public APIs +- CLI messages: direct English, no marketing tone (see `SOUL.md`) + +## Tests + +```bash +pytest # full suite (may call local CLIs) +pytest -m "not integration" # fast subset for CI +``` + +Mark new external-tool tests with `@pytest.mark.integration`. + +## Adding a module + +See [MODULE_DEVELOPMENT.md](MODULE_DEVELOPMENT.md). Register in `imagepipeline/modules/__init__.py` and add param/behavior tests. + +## Pipelines + +Scripts in `pipelines/` are examples and shoot-specific recipes. Machine-local `INPUT` paths are intentional. Reset `CONTINUE_FROM` before sharing a script. + +## Commits + +Conventional Commits (`feat:`, `fix:`, `chore:`, …). One logical change per commit. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..6a19352 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Frank + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 05410de..6268944 100644 --- a/README.md +++ b/README.md @@ -7,15 +7,19 @@ Each pipeline is a Python script that defines a DAG of processing steps. Every s ## Requirements - Python 3.11+ -- [ImageMagick](https://imagemagick.org/) (`magick` or `convert` on PATH) +- [ImageMagick](https://imagemagick.org/) (`magick` or `convert` on PATH) — most pipelines +- Optional CLIs per module: [G'MIC](https://gmic.eu/), [rembg](https://github.com/danielgatis/rembg), [darktable-cli](https://www.darktable.org/), GIMP (`gimp-console`) ## Installation ```bash cd /path/to/imagepipeline -pip install -e ".[dev]" +pip install -e ".[dev]" # core + tests +pip install -e ".[dev,ai]" # + torch/numpy for AI modules ``` +Copy `.env.example` to `.env` and set `OPENROUTER_API_KEY` when using `openrouter_edit` or Comfy-related workflows. + ## Quick Start Edit the input path in `pipelines/example_grayscale.py`, then run: @@ -35,9 +39,11 @@ with Pipeline(name="my_run", input_dir=Path("/path/to/export")) as p: p.run() ``` +List registered modules: `imagepipeline list-modules` + ## Output Structure -Each run creates a folder like `my_run_20260527143022/`: +Each run creates a folder like `my_run_20260527143022/` under `~/pipeline_output/` (or `output_base`): ``` my_run_20260527143022/ @@ -50,6 +56,25 @@ my_run_20260527143022/ Step folders are named `{module_name}_{nn}` by default (two-digit counter per module name). Pass optional `step_id="input_bokeh"` to `p.step()` for a custom folder name and step reference (see [docs/MODULE_DEVELOPMENT.md](docs/MODULE_DEVELOPMENT.md#step-folder-naming)). +## Resume + +Pipelines support resuming interrupted runs: + +```python +CONTINUE_FROM = Path("~/pipeline_output/my_run_260718120000") +EXISTING_OUTPUTS = {"rembg_01": CONTINUE_FROM / "rembg_01"} + +with Pipeline( + name="my_run", + input_dir=INPUT, + continue_from=CONTINUE_FROM, + existing_outputs=EXISTING_OUTPUTS, +) as p: + ... +``` + +Modules that change file extensions must implement `expected_output_filenames` so skip logic works (e.g. `rembg` → `.png`). + ## Writing Pipelines Pipelines are plain Python scripts. Reference previous steps via `StepRef` objects returned by `p.step()`: @@ -67,12 +92,27 @@ with Pipeline(name="colorsplash", input_dir=INPUT) as p: - Parameters are passed as kwargs and validated against each module's schema - Multiple uses of the same module get separate numbered folders +Declarative building blocks for agents and humans: [RECIPES.md](RECIPES.md). + ## Adding Modules -See [docs/MODULE_DEVELOPMENT.md](docs/MODULE_DEVELOPMENT.md). +See [docs/MODULE_DEVELOPMENT.md](docs/MODULE_DEVELOPMENT.md). Architecture overview: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). + +## Development + +```bash +ruff check . +ruff format . +pytest # full suite (uses local CLIs when present) +pytest -m "not integration" # fast subset (CI default) +``` + +See [CONTRIBUTING.md](CONTRIBUTING.md). ## Tests ```bash pytest ``` + +Optional AI tests require `pip install -e ".[ai]"`. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..c8b41c1 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,59 @@ +# Architecture + +Imagepipeline is a small Python framework for defining **batch image pipelines** as DAGs. Each step is a registered module; each run writes numbered folders under a timestamped output root. + +## Layout + +``` +imagepipeline/ +├── core/ # Pipeline, runner, resume, manifest, params, logging +├── modules/ # Processing steps (@register) +├── utils/ # files, subprocess, gmic, gimp helpers +├── ai/ # Optional torch models (HDRNet, Zero-DCE) +└── cli.py # `imagepipeline list-modules` + +pipelines/ # Runnable scripts (machine-local INPUT paths) +tests/ # pytest suite +docs/ # Developer docs +workflows/comfy/ # ComfyUI workflow JSON (optional AI path) +``` + +## Execution flow + +1. **Define** — `Pipeline(name=..., input_dir=...)` collects `step()` calls (`StepDefinition` DAG). +2. **Run** — `PipelineRunner` topologically sorts steps, matches inputs by filename stem, builds `ModuleContext`. +3. **Resume** — `continue_from` / `existing_outputs` reuse prior run folders; modules declare `expected_output_filenames` when extensions change. +4. **Manifest** — `pipeline_manifest.json` records steps, params, and paths. + +## Module contract + +Every module subclasses `BaseModule` or `SubprocessModule`: + +- `name`, `description`, `parameters()` schema +- `run(ctx: ModuleContext)` writes into `ctx.output_dir` +- Optional `expected_output_filenames()` for resume when output names differ from inputs +- `check_dependencies()` for external CLI tools + +Registration happens via `@register` and eager import in `imagepipeline/modules/__init__.py`. + +## External tools + +Many modules shell out to CLIs (not Python packages): + +| Tool | Modules | +|------|---------| +| ImageMagick (`magick`/`convert`) | `imagemagick_*`, `composite`, `color_to_alpha`, `crop_square` | +| G'MIC | `gmic`, `gmic_grayscale` | +| rembg | `rembg` | +| darktable-cli | `darktable_style` | +| GIMP | `xcf_stack` | + +AI modules need `pip install -e ".[ai]"` (torch, numpy) and optionally API keys in `.env`. + +## Design choices + +- **Plain Python pipelines** — no YAML DSL; full control and easy resume constants in script. +- **Stem matching** — multi-input steps align files by basename across step folders. +- **Symlink input** — default run copies/symlinks source images into `input/` for reproducibility. + +See [MODULE_DEVELOPMENT.md](MODULE_DEVELOPMENT.md) for adding modules.