Cleanup/quality pass #1

Merged
froxxxy merged 14 commits from cleanup/quality-pass into main 2026-07-18 17:53:57 +02:00
7 changed files with 177 additions and 55 deletions
Showing only changes of commit 0daf3e2315 - Show all commits
+28 -18
View File
@@ -24,7 +24,7 @@ Not executable code. Named building blocks for creating or editing pipelines in
```
6. **Shared expensive steps** — `rembg` runs once per pipeline and is reused. Recipe lines say `rembg`, but the agent must not re-run it for every composite.
7. **`xcf_stack` (GIMP export)** — stacks **all prior step outputs** (plus `input/` by default) into one `.xcf` per image with layers named after step ids. Usually the **last step** after all composites. Requires `gimp` on PATH. Not a layer inside a `combine` block — a separate finishing recipe.
7. **`xcf_stack` (GIMP export)** — stacks **listed step outputs** into one `.xcf` per image. Pass step refs in `inputs=[...]` (bottom layer → top). The runner waits until all listed steps finish. Requires `gimp` on PATH. Not a layer inside a `combine` block — a separate recipe.
8. **Placeholders** — `COLOR1`, `COLOR2`, `COLOR`, `STYLE`, `MAX_EDGE`, `PROMPT`, `MODEL`, `TEMPLATE_IMAGE`, etc. are filled in at pipeline creation time (from your prompt or constants at the top of the script).
9. **Colors** — hex for backgrounds (`#RRGGBB` or `#RRGGBBAA`). G'MIC filters need `R,G,B` tuples (see [GMIC color derivation](#6-gmic-color-derivation) below).
10. **Single-step vs composite vs pipeline** — one declarative line = one module step. Multiple indented lines in a **composite** recipe = layers (bottom → top). Multiple indented lines in a **pipeline** recipe = sequential steps (first step first, output feeds the next).
@@ -67,7 +67,7 @@ Not executable code. Named building blocks for creating or editing pipelines in
- **Recipe inclusion:** if an indented line is a known recipe id (e.g. `colorsplash` inside `colorsplash-watermark`), look up that recipe, expand it, and use its output as the input for the next step. Expand recursively. Do not duplicate shared steps — one `rembg` (etc.) per pipeline when multiple included recipes need it.
- **Pipeline vs composite:** multiple indented lines that are **sequential steps or recipe refs** = pipeline recipe (chain). Multiple indented **layer** lines (original, rembg, backgrounds, gmic-on-rembg, …) = composite recipe (combine). When unsure: if any line is another recipe id, treat the parent as a pipeline recipe.
- **Ignore notes:** skip any line that is only a `#` comment (after indent trim). Strip inline `# …` suffixes on recipe-id lines. Do not copy note text into Python comments unless Fränky asks.
- **`xcf_stack`:** when a pipeline includes recipe `xcf-stack` (or the user asks for GIMP layers), append as the final step: `p.step("xcf_stack", inputs="input")`. The module auto-collects `prior_steps` from the runner — do not list every composite as explicit inputs. See `imagepipeline/modules/xcf_stack.py`.
- **`xcf_stack`:** list every layer source in `inputs=[...]` (bottom → top). Include `"input"` for the original. The runner schedules `xcf_stack` after all listed steps complete. See `imagepipeline/modules/xcf_stack.py` and `pipelines/pipeline_rezepttest.py`.
- **Shared steps:** define `rembg` (and other expensive steps) **once** per pipeline and reference the step in composites — mirror `pipelines/pipeline_baxxter.py`.
- Follow the **Rules cheat sheet** above — especially layer order and chained composites.
- **3+ layers:** chain `composite` steps (see baxxter, crusaders, orange).
@@ -87,14 +87,14 @@ Not executable code. Named building blocks for creating or editing pipelines in
| `gradient background COLOR1 COLOR2 radial` | `imagemagick_fill` radial |
| `COLOR background` | `imagemagick_fill` solid `color1=COLOR` |
| `rembg with gmic: FILTER` | `gmic` on rembg output; command = `-FILTER` |
| `original with gmic: FILTER` | `gmic` on input; command = `-FILTER` |
| `layer opacity N%` | `composite` `foreground_opacity=N/100` on that layer |
| `make layer 5% bigger then crop to original size` | `imagemagick_scale_crop` `scale=1.05` |
| `COLOR to alpha` | `color_to_alpha` `color=COLOR` |
| `resize max edge MAX_EDGE` | `imagemagick_resize` `max_edge=MAX_EDGE` |
| `crop square` | `crop_square` |
| `darktable style STYLE` | `darktable_style` `style=STYLE` |
| `xcf stack all steps` | `xcf_stack` — GIMP layer export of all prior steps |
| `xcf stack without input` | `xcf_stack` `include_input=false` |
| `xcf stack layers: LAYER1, LAYER2, …` | `xcf_stack` with `inputs=[...]` in that order |
| `xcf stack skip missing` | `xcf_stack` `skip_missing=true` |
| `openrouter edit PROMPT MODEL` | `openrouter_edit` |
| `openrouter gallery match TEMPLATE_IMAGE` | `openrouter_edit` with `template_image` |
@@ -230,24 +230,15 @@ crop-square
```text
xcf-stack # GIMP layer export — one .xcf per input image
xcf stack all steps
xcf stack layers: input, rembg, composite_01, composite_02, …
```
- **Module:** `xcf_stack` — `p.step("xcf_stack", inputs="input")` as **last step**
- **Layer order in XCF:** `input/` (bottom, default) then prior steps in pipeline definition order
- **Module:** `xcf_stack` — `p.step("xcf_stack", inputs=["input", step_a, step_b, …])`
- **Layer order in XCF:** same as `inputs` list (bottom → top); GIMP layer names = step ids (`input` for originals)
- **Matching:** layers matched by filename stem (extension may differ, e.g. rembg `.png` on `.jpg` input)
- **Output:** `{stem}.xcf` per input image
- **Requires:** `gimp` on PATH
- **No existing pipeline uses this yet** — append to any multi-output pipeline (e.g. baxxter variants + `xcf-stack`)
### xcf-stack-no-input
```text
xcf-stack-no-input
xcf stack without input
```
- **Module:** `xcf_stack` — `include_input=False`
- **Source:** `pipelines/pipeline_rezepttest.py`
### xcf-stack-skip-missing
@@ -461,6 +452,19 @@ original-jpr-smooth-rembg
- **Middle layer:** jpr smooth on rembg, then scale 5% + center crop
- **Source:** baxxter, crusaders
### bokeh-oktagon
```text
bokeh-oktagon
original
original with gmic: fx_bokeh 3,5,0,30,8,4,0.3,0.2,R1,G1,B1,ALPHA1,0.7,30,20,20,1,2,R2,G2,B2,ALPHA2,0.15
rembg
```
- **3 layers:** composite(original, input_bokeh) → composite(result, rembg)
- **Colors:** `R1,G1,B1` from **COLOR1**; `R2,G2,B2` from **COLOR2**; `ALPHA1` / `ALPHA2` are 0255 (defaults 160 / 110)
- **G'MIC:** octagonal bokeh discs — first color pair `R1,G1,B1,ALPHA1`, second `R2,G2,B2,ALPHA2`
### original-jpr-smooth-grey-alpha-rembg
```text
@@ -556,6 +560,7 @@ G'MIC commands in parameterized recipes use **comma-separated RGB integers**, no
| `original-bwrecolor-rembg` | highlight `R2,G2,B2`, accent `R1,G1,B1` in `fx_bwrecolorize` |
| `rembg-custom-gradient-a` | starts with `R1,G1,B1`, transitions via `R2,G2,B2` |
| `rembg-custom-gradient-b` | starts with `R2,G2,B2`, transitions via `R1,G1,B1` |
| `bokeh-oktagon` | bokeh tints `R1,G1,B1,ALPHA1` and `R2,G2,B2,ALPHA2` in `fx_bokeh` |
| `imagemagick_fill` backgrounds | use full hex including alpha if needed (`#d7fd00ff`) |
### Agent workflow
@@ -596,7 +601,12 @@ colorsplash-watermark
Create pipeline "foo" with recipes colorsplash, rembg-radial-2colors, and xcf-stack
```
Agent: all composite outputs first, then `p.step("xcf_stack", inputs="input")` as the final step.
Agent: build all composite steps, then `p.step("xcf_stack", inputs=["input", rembg_out, …, composite_final])` listing every layer bottom → top.
```text
xcf-stack
xcf stack layers: input, rembg, composite_colorsplash, composite_radial, …
```
### Full combine list (baxxter style)
+2 -1
View File
@@ -18,7 +18,8 @@ class ModuleContext:
pipeline_output_root: Path
step_id: str
matched_groups: list[list[Path]] = field(default_factory=list)
prior_steps: list[tuple[str, Path]] = field(default_factory=list)
input_refs: list[str] = field(default_factory=list)
input_layer_dirs: list[tuple[str, Path]] = field(default_factory=list)
logger: PipelineLogger | None = None
@property
+17 -5
View File
@@ -151,9 +151,23 @@ class PipelineRunner:
step.module.check_dependencies()
validated = step.module.validate_module_params(step.params)
if step.module_name == "xcf_stack":
if not step.input_refs:
raise ValidationError("xcf_stack requires at least one input layer")
source_lists = [self._resolve_source_paths(ref) for ref in step.input_refs]
input_paths = source_lists[0]
matched_groups = [[path] for path in input_paths]
input_layer_dirs: list[tuple[str, Path]] = []
for ref in step.input_refs:
if ref == INPUT_SOURCE:
input_layer_dirs.append(("input", self._input_link_dir))
else:
input_layer_dirs.append((ref, self._results[ref].output_dir))
else:
source_lists = [self._resolve_source_paths(ref) for ref in step.input_refs]
matched_groups = match_by_stem(source_lists)
input_paths = flatten_matched(matched_groups)
input_layer_dirs = []
output_dir = self.output_root / step.output_dir_name
output_dir.mkdir(parents=True, exist_ok=True)
@@ -220,7 +234,8 @@ class PipelineRunner:
pipeline_output_root=self.output_root,
step_id=step.step_id,
matched_groups=matched_groups,
prior_steps=[(sid, res.output_dir) for sid, res in self._results.items()],
input_refs=list(step.input_refs),
input_layer_dirs=input_layer_dirs,
logger=self.logger,
)
@@ -280,7 +295,4 @@ class PipelineRunner:
if len(ordered_ids) != len(self.steps):
raise CycleError("Pipeline contains a cycle in step dependencies")
ordered = [step_by_id[step_id] for step_id in ordered_ids]
deferred = [step for step in ordered if step.module.runs_last]
regular = [step for step in ordered if not step.module.runs_last]
return regular + deferred
return [step_by_id[step_id] for step_id in ordered_ids]
-1
View File
@@ -14,7 +14,6 @@ class BaseModule(ABC):
name: ClassVar[str]
description: ClassVar[str] = ""
runs_last: ClassVar[bool] = False
supported_input_formats: ClassVar[tuple[str, ...]] = (
".jpg",
".jpeg",
+13 -20
View File
@@ -4,18 +4,22 @@ from pathlib import Path
from imagepipeline.core.context import ModuleContext
from imagepipeline.core.params import Param
from imagepipeline.core.step import INPUT_SOURCE
from imagepipeline.modules.base import SubprocessModule
from imagepipeline.modules.registry import register
from imagepipeline.utils.files import find_image_by_stem
from imagepipeline.utils.gimp import stack_images_to_xcf
def _layer_name(ref: str) -> str:
return "input" if ref == INPUT_SOURCE else ref
@register
class XcfStackModule(SubprocessModule):
name = "xcf_stack"
runs_last = True
description = (
"Stack all prior pipeline step outputs as GIMP layers into one XCF per image"
"Stack listed pipeline step outputs as GIMP layers into one XCF per image"
)
command_candidates = ("gimp-console", "gimp")
@@ -32,11 +36,6 @@ class XcfStackModule(SubprocessModule):
@classmethod
def parameters(cls) -> dict[str, Param]:
return {
"include_input": Param(
"bool",
default=True,
help="Include pipeline input/ directory as bottom layer",
),
"skip_missing": Param(
"bool",
default=False,
@@ -50,19 +49,13 @@ class XcfStackModule(SubprocessModule):
}
def run(self, ctx: ModuleContext) -> None:
include_input = ctx.params["include_input"]
skip_missing = ctx.params["skip_missing"]
timeout = ctx.params["timeout"]
layer_sources: list[tuple[str, Path]] = []
if include_input:
layer_sources.append(("input", ctx.pipeline_output_root / "input"))
layer_sources.extend(ctx.prior_steps)
if not layer_sources:
if not ctx.input_layer_dirs:
raise ValueError(
"xcf_stack has no layer sources "
"(include_input=False and no prior_steps)"
"xcf_stack requires at least one input layer "
"(pass step refs in inputs=[...], bottom to top)"
)
ctx.output_dir.mkdir(parents=True, exist_ok=True)
@@ -73,19 +66,19 @@ class XcfStackModule(SubprocessModule):
stem = input_path.stem
layers: list[tuple[str, Path]] = []
for step_id, directory in layer_sources:
for ref, directory in ctx.input_layer_dirs:
image_path = find_image_by_stem(directory, stem)
if image_path is None:
if not skip_missing:
raise ValueError(
f"xcf_stack: no image with stem '{stem}' in step "
f"'{step_id}' ({directory})"
f"'{_layer_name(ref)}' ({directory})"
)
continue
layers.append((step_id, image_path))
layers.append((_layer_name(ref), image_path))
if not layers:
checked = ", ".join(step_id for step_id, _ in layer_sources)
checked = ", ".join(name for name, _ in ctx.input_layer_dirs)
raise ValueError(
f"xcf_stack: no layers found for stem '{stem}' "
f"(checked: {checked})"
+101
View File
@@ -0,0 +1,101 @@
#!/usr/bin/env python3
"""Rezepttest pipeline — RECIPES.md smoke test (Indians Aichelberg logo colors)."""
from pathlib import Path
from imagepipeline import Pipeline
# Change to your Darktable export folder.
INPUT = Path("/home/frank/tmp/bar")
OUTPUT_BASE = Path.home() / "pipeline_output"
EXISTING_OUTPUTS: dict[str, Path] = {}
CONTINUE_FROM: Path | None = None
# Colors from Indians Aichelberg logo (orange text + red stitching).
COLOR1 = "#f0b020" # 240, 176, 32
COLOR2 = "#f01000" # 240, 16, 0
ALPHA1 = 160
ALPHA2 = 110
GMIC_DROP_SHADOW = "-fx_drop_shadow3d 0,0,0,10,1,1,2,0.5,240,16,0,200,0"
GMIC_JPR_SMOOTH = "-jpr_gradient_smooth 0,1.5"
GMIC_BOKEH = (
f"-fx_bokeh 3,5,0,30,8,4,0.3,0.2,240,176,32,{ALPHA1},0.7,30,20,20,1,2,"
f"240,16,0,{ALPHA2},0.15"
)
def main() -> None:
with Pipeline(
name="rezepttest",
input_dir=INPUT,
output_base=OUTPUT_BASE,
existing_outputs=EXISTING_OUTPUTS or None,
continue_from=CONTINUE_FROM,
) as p:
rembg_out = p.step("rembg", inputs="input")
grayscale = p.step("gmic_grayscale", inputs="input")
gradient_radial_bg = p.step(
"imagemagick_fill",
inputs="input",
color1=COLOR1,
color2=COLOR2,
gradient=True,
radial=True,
)
rembg_shadow = p.step("gmic", inputs=rembg_out, command=GMIC_DROP_SHADOW)
rembg_jpr_smooth = p.step("gmic", inputs=rembg_out, command=GMIC_JPR_SMOOTH)
rembg_jpr_smooth_sized = p.step(
"imagemagick_scale_crop",
inputs=rembg_jpr_smooth,
scale=1.05,
)
input_bokeh = p.step("gmic", inputs="input", command=GMIC_BOKEH)
# recipe: colorsplash
composite_colorsplash = p.step("composite", inputs=[grayscale, rembg_out])
# recipe: rembg-radial-2colors (gradient-radial)
composite_radial = p.step("composite", inputs=[gradient_radial_bg, rembg_out])
# recipe: original-drop-shadow-rembg
shadow_mid = p.step("composite", inputs=["input", rembg_shadow])
composite_shadow = p.step("composite", inputs=[shadow_mid, rembg_out])
# recipe: original-jpr-smooth-rembg
smooth_mid = p.step("composite", inputs=["input", rembg_jpr_smooth_sized])
composite_smooth = p.step("composite", inputs=[smooth_mid, rembg_out])
# recipe: bokeh-oktagon
bokeh_mid = p.step("composite", inputs=["input", input_bokeh])
composite_bokeh = p.step("composite", inputs=[bokeh_mid, rembg_out])
# recipe: xcf-stack — explicit layer list, bottom to top
p.step(
"xcf_stack",
inputs=[
"input",
rembg_out,
grayscale,
gradient_radial_bg,
rembg_shadow,
rembg_jpr_smooth_sized,
input_bokeh,
composite_colorsplash,
composite_radial,
composite_shadow,
composite_smooth,
composite_bokeh,
],
)
output_root = p.run()
print(f"Pipeline finished. Output: {output_root}")
if __name__ == "__main__":
main()
+13 -7
View File
@@ -41,7 +41,7 @@ class TestModuleRegistration:
def test_get_module_returns_xcf_stack_class(self) -> None:
assert get_module("xcf_stack") is XcfStackModule
def test_xcf_stack_runs_last(self) -> None:
def test_xcf_stack_waits_for_layer_inputs(self) -> None:
from imagepipeline.core.runner import PipelineRunner
from imagepipeline.core.step import StepDefinition
from imagepipeline.modules.imagemagick_grayscale import ImageMagickGrayscale
@@ -51,7 +51,7 @@ class TestModuleRegistration:
step_id="xcf_stack_01",
module_name="xcf_stack",
module=XcfStackModule,
input_refs=["input"],
input_refs=["input", "imagemagick_grayscale_01"],
params={},
output_dir_name="xcf_stack_01",
),
@@ -71,7 +71,10 @@ class TestModuleRegistration:
steps=steps,
)
ordered = runner._topological_sort()
assert ordered[-1].module_name == "xcf_stack"
assert [step.step_id for step in ordered] == [
"imagemagick_grayscale_01",
"xcf_stack_01",
]
class TestExpectedOutputFilenames:
@@ -116,7 +119,7 @@ def _make_stack_fixture(tmp_path: Path) -> dict[str, Path]:
class TestXcfStackRun:
@patch("imagepipeline.modules.xcf_stack.stack_images_to_xcf")
def test_collects_layers_from_input_and_prior_steps(
def test_collects_layers_from_explicit_inputs(
self, mock_stack: object, tmp_path: Path
) -> None:
paths = _make_stack_fixture(tmp_path)
@@ -130,7 +133,8 @@ class TestXcfStackRun:
params=XcfStackModule.validate_module_params({}),
pipeline_output_root=paths["root"],
step_id="xcf_stack_01",
prior_steps=[
input_layer_dirs=[
("input", paths["input_dir"]),
("step_a", paths["step_a"]),
("step_b", paths["step_b"]),
],
@@ -161,7 +165,8 @@ class TestXcfStackRun:
params=XcfStackModule.validate_module_params({"skip_missing": True}),
pipeline_output_root=paths["root"],
step_id="xcf_stack_01",
prior_steps=[
input_layer_dirs=[
("input", paths["input_dir"]),
("step_a", paths["step_a"]),
("step_b", paths["step_b"]),
],
@@ -187,7 +192,8 @@ class TestXcfStackRun:
params=XcfStackModule.validate_module_params({"skip_missing": False}),
pipeline_output_root=paths["root"],
step_id="xcf_stack_01",
prior_steps=[
input_layer_dirs=[
("input", paths["input_dir"]),
("step_a", paths["step_a"]),
("step_b", paths["step_b"]),
],