diff --git a/README.md b/README.md index 04bea47..1737a20 100644 --- a/README.md +++ b/README.md @@ -1,62 +1,157 @@ # Repixelizer -Repixelizer is a standalone Python CLI that turns "fake pixel art" into true grid-aligned pixel art for icons and sprites. +Repixelizer is a standalone Python CLI for turning "fake pixel art" into true grid-aligned pixel art. -It does not rely on a single resize pass. Instead it: +The core idea is simple: generated images often look like pixel art without actually obeying a single clean pixel grid. Repixelizer treats that as an optimization problem instead of a resize problem. It estimates the implied lattice, builds a real target grid, optimizes how output pixels sample the source image, then cleans the result up on the discrete grid. -1. infers the source image's implied pixel lattice, -2. initializes a target grid at that resolution, -3. runs a continuous UV optimization stage in premultiplied RGBA space, -4. projects the result onto a real pixel grid with discrete cleanup, -5. optionally constrains the output to a palette, and -6. writes diagnostics and baseline comparisons. +This project is meant to stand on its own. It should be understandable without any StreamPixels-specific context. -## Installation +## What it does + +Repixelizer currently focuses on single-image inputs such as: +- icons +- emblems +- logos +- simple sprites + +The current pipeline is: + +1. infer the source image's implied fake-pixel lattice +2. choose a target output resolution, unless the user overrides it +3. initialize a UV/sample field over the target grid +4. optimize that field in premultiplied RGBA space with PyTorch +5. project onto a true pixel grid +6. run discrete cleanup to reduce noise and isolated-pixel garbage +7. optionally quantize into a palette-constrained result +8. write diagnostics and baseline comparisons + +## Project docs + +- Product and technical spec: [docs/spec.md]() +- Living implementation plan: [docs/implementation-plan.md]() + +If you are moving this project into a fresh workspace, those two docs are the main handoff. + +## Quickstart ```powershell python -m venv .venv .venv\Scripts\python -m pip install -e .[dev] ``` -## Usage +Run the optimizer: ```powershell repixelize input.png --out output.png repixelize input.png --out output.png --target-size 120 --diagnostics-dir diagnostics +``` + +Run the optimizer plus baselines: + +```powershell repixelize compare input.png --out output.png --diagnostics-dir diagnostics ``` +## CLI contract + +Primary command shape: + +```powershell +repixelize input.png --out output.png +repixelize compare input.png --out output.png +``` + +Important flags: + +- `--target-size `: override inferred target max dimension +- `--palette `: palette file in `.gpl`, `.txt`, or `.json` +- `--palette-mode off|fit|strict`: palette behavior +- `--diagnostics-dir `: write visual and JSON diagnostics +- `--seed `: deterministic optimizer seed +- `--steps `: number of optimization steps +- `--device cpu|cuda`: PyTorch device + ### Palette modes -- `off`: keep the full-color RGBA output. -- `fit`: derive or adapt to a useful palette and write palette diagnostics. -- `strict`: require a supplied palette file and stay inside it exactly. +- `off`: keep the output unconstrained +- `fit`: adapt the result to a derived or supplied palette +- `strict`: stay strictly inside the supplied palette -Palette files can be `.gpl`, `.txt`, or `.json`. +Palette constraints are optional by design. Many fake-pixel-art inputs are not born from a coherent palette, so palette enforcement is useful but not assumed. -## Diagnostics +## Outputs -When `--diagnostics-dir` is supplied, Repixelizer writes: +Main output: +- final RGBA PNG -- `run.json`: chosen size, phase, score breakdown, timings, and settings -- `lattice-overlay.png`: inferred lattice drawn over the source +Optional diagnostics: +- `run.json`: chosen size, phase, settings, timings, and score breakdown +- `lattice-overlay.png`: inferred lattice preview over the source image - `comparison.png`: source, optimized output, and nearest-neighbor preview -- `alpha-preview.png`: source/result alpha -- `noise-heatmap.png`: local noise and isolated-pixel hotspots +- `alpha-preview.png`: alpha behavior before and after +- `noise-heatmap.png`: isolated-pixel and local-noise hotspots - `cluster-preview.png`: coarse material/color clustering -- compare mode extras: - - `compare.json` - - `compare.csv` - - `compare-sheet.png` -## Architecture +Additional compare-mode outputs: +- `compare.json` +- `compare.csv` +- `compare-sheet.png` -- `repixelizer.inference`: lattice and target-size inference -- `repixelizer.continuous`: PyTorch UV-field optimization -- `repixelizer.discrete`: local-search cleanup on the final grid -- `repixelizer.baselines`: naive and error-diffusion baselines -- `repixelizer.diagnostics`: machine-readable and visual debug outputs +## Repo layout -## Current scope +- `src/repixelizer`: core package +- `tests`: synthetic and pipeline-focused tests +- `examples`: reserved for future real example inputs +- `artifacts`: local smoke-test outputs, not core source -Version `0.1.0` is single-image only. It is aimed at icons, emblems, and simple sprites first, with automatic resolution inference and no manual masks. +Core modules: +- `inference.py`: target-size and phase inference +- `analysis.py`: edge, alpha, and coarse clustering analysis +- `continuous.py`: UV-field optimization +- `discrete.py`: local cleanup after projection +- `palette.py`: palette loading and optional quantization +- `compare.py`: baseline comparisons and contact-sheet output +- `synthetic.py`: benchmark generators for tests + +## Current status + +The project is bootstrapped and runnable: +- editable install works +- PyTorch-based optimization is wired in +- synthetic tests are in place +- compare mode writes diagnostics and baseline metrics + +What exists today is a serious experimental baseline, not a finished research result. It is strong enough to iterate on algorithmically, but it is not yet proven across a broad real-world image corpus. + +## Current limitations + +- single-image only +- no multi-frame sprite-sheet consistency yet +- no GUI yet +- no user-painted masks or region hints yet +- lattice inference is heuristic, not learned +- continuous optimization is real, but still early and intentionally conservative + +## Validation + +The current test suite checks: +- resolution inference on synthetic emblem inputs +- palette loading and strict palette enforcement +- pipeline output and diagnostics writing +- compare-mode artifact generation +- a basic default CLI execution path + +Run it with: + +```powershell +.venv\Scripts\python -m pytest -q +``` + +## Near-term direction + +The next useful steps are: +- improve inference on real-world generated inputs +- make the optimizer outperform resize-and-dither baselines more consistently +- expand the curated benchmark corpus +- add smarter discrete cleanup rules for outlines, highlights, and material bands +- optionally add manual guidance in a later version, once the automatic baseline is stable diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md new file mode 100644 index 0000000..d521412 --- /dev/null +++ b/docs/implementation-plan.md @@ -0,0 +1,117 @@ +# Repixelizer Implementation Plan + +## Current state + +The project is bootstrapped and runnable as a standalone Python repo. + +Implemented: +- package metadata and editable install +- CLI entrypoint with `run` and `compare` modes +- lattice/phase inference with coherence scoring plus source-period prior +- source-image structural analysis +- PyTorch-based continuous UV-field optimization +- discrete cleanup on the output grid +- optional palette loading and quantization modes +- compare mode with naive and diffusion baselines +- diagnostics writing +- synthetic test fixtures and automated tests + +Verified: +- local editable install in a dedicated venv +- full test suite currently passing +- compare-mode smoke run against a real emblem image + +## What is still provisional + +The current implementation should be treated as a strong experimental baseline, not a final algorithm. + +The most provisional areas are: +- lattice inference on real, messy generated imagery +- the exact loss weighting in the continuous stage +- the strength and shape of discrete cleanup heuristics +- compare-mode metrics as a proxy for actual visual quality + +## Near-term priorities + +### 1. Build a real benchmark corpus + +Add curated real-world cases, not just synthetic ones: +- emblems and logos with transparency +- icons with strong outlines +- simple sprite-like figures +- examples that are known to fail naive resize-and-dither + +Each case should include: +- source image +- expected target size or plausible size range +- short notes on known failure modes + +### 2. Improve lattice inference + +The current inference is workable but still heuristic. + +Next improvements should focus on: +- better periodicity detection for noisy or partially aligned images +- stronger penalties against oversized candidate grids +- confidence calibration that reflects ambiguity more honestly + +### 3. Strengthen the discrete cleanup stage + +Current cleanup is local and generic. + +Next version should add: +- stronger outline continuity rules +- better preservation of highlight bands +- better handling of metallic or gem-like materials +- smarter alpha-edge cleanup for transparent icons + +### 4. Make compare mode more trustworthy + +The current metrics are useful but still internal. + +Next work should: +- expand score breakdowns +- validate whether the metrics correlate with actual human preference +- make it easier to compare runs across parameter sweeps + +## Suggested next milestones + +### Milestone 1: Real-image evaluation harness + +- add curated real-image fixtures +- add golden metadata for expected target-size ranges +- add a repeatable batch runner +- emit one summary table per corpus run + +### Milestone 2: Solver tuning pass + +- tune continuous-stage loss weights against the benchmark corpus +- test more robust anti-speckle and cluster-preservation behavior +- reduce cases where the solver underperforms the naive baseline + +### Milestone 3: Better cleanup heuristics + +- add contour-aware cleanup +- add material-sensitive smoothing for highlights and trim +- reduce the number of visually good-but-metric-weak outputs + +### Milestone 4: Human-in-the-loop v2 planning + +Only after the automatic baseline is strong: +- explore optional region hints +- evaluate a minimal GUI or notebook layer +- consider multi-frame support + +## Risks + +- The solver can still produce outputs that score well but feel over-smoothed. +- Real generated images may contain multiple inconsistent local lattice cues. +- Continuous optimization is flexible, but easy to overfit to the reconstruction term. +- Metrics that reward coherence can accidentally reward blandness if not balanced carefully. + +## Assumptions + +- Single-image quality matters more than batch throughput in the current phase. +- Palette constraints are optional and should stay optional. +- The project is better served by improving automatic behavior first than by rushing into manual tooling. +- Python plus PyTorch remains the right stack for the current research/prototyping phase. diff --git a/docs/spec.md b/docs/spec.md new file mode 100644 index 0000000..baa72dc --- /dev/null +++ b/docs/spec.md @@ -0,0 +1,203 @@ +# Repixelizer Spec + +## Summary + +Repixelizer is a standalone Python CLI for converting "fake pixel art" into true grid-aligned pixel art. The target use cases are single-image icons, emblems, logos, and simple sprites, especially images that already resemble pixel art but fail to obey a consistent pixel lattice. + +The project exists because generated imagery often creates appealing pixel-like local patterns while violating the actual constraints of pixel art: +- inconsistent subpixel alignment +- locally plausible but globally incoherent clusters +- noisy highlights and outlines +- fake micro-detail that collapses when downsampled + +Repixelizer treats this as an optimization and reconstruction problem rather than a resizing problem. + +## Goals + +- Recover an implied target pixel grid from a fake-pixel-art source image. +- Produce a true discrete-grid output image that looks authored, not merely resized. +- Preserve alpha, silhouette clarity, and strong interior structure. +- Support palette-free output by default, with optional palette-constrained modes. +- Provide diagnostics and baseline comparisons so results are inspectable and measurable. + +## Non-goals for v1 + +- full sprite-sheet or animation consistency +- manual painting or mask authoring as a required workflow +- a desktop GUI +- generalized photo-to-pixel-art stylization +- a learned model trained on a large dataset + +## Primary users + +- artists or developers trying to salvage generated fake pixel art +- tool builders experimenting with constrained repixelization +- game/UI developers creating icons, emblems, badges, and simple sprites + +## Input and output contract + +### Input + +- one RGBA image file +- optional explicit target size override +- optional palette file +- optional diagnostics directory +- optional solver/device settings + +### Output + +- one final RGBA PNG +- optional palette-constrained output behavior +- optional diagnostics bundle containing JSON and visual artifacts + +## Product behavior + +### Default mode + +The default workflow should be fully automatic: + +1. load the source image +2. infer lattice size and phase +3. analyze source structure +4. optimize a continuous UV/sample field +5. project that result to a real pixel grid +6. clean up obvious discrete-grid artifacts +7. optionally quantize to a palette +8. write output and diagnostics + +No manual masks or user-authored region hints are assumed in v1. + +### CLI shape + +Primary commands: + +```powershell +repixelize input.png --out output.png +repixelize compare input.png --out output.png +``` + +Supported flags: +- `--target-size` +- `--palette` +- `--palette-mode off|fit|strict` +- `--diagnostics-dir` +- `--seed` +- `--steps` +- `--device cpu|cuda` + +### Palette behavior + +- `off`: produce unconstrained RGBA output +- `fit`: adapt to a useful palette derived from or compatible with the result +- `strict`: stay strictly inside a supplied palette + +Palette constraints are optional because many generated fake-pixel-art images do not originate from a coherent palette. + +## Technical design + +### 1. Lattice inference + +The tool must estimate the resolution of the fake lattice being mimicked. + +Requirements: +- search over candidate target sizes +- search over subpixel phase offsets for each candidate +- score candidates using coherence-oriented metrics rather than reconstruction alone +- expose the top-ranked alternatives in diagnostics +- if `--target-size` is provided, skip size search but still estimate phase + +Scoring goals: +- low isolated-pixel rate +- high cluster continuity +- strong alpha-edge crispness +- straighter outlines +- lower local color chatter +- reasonable agreement with source-derived periodicity priors + +### 2. Source analysis + +The solver should derive automatic guidance from the image itself: +- alpha map +- edge map +- coarse material/color clustering + +This guidance is used to preserve important boundaries while allowing smoother regions to collapse into cleaner pixel clusters. + +### 3. Continuous optimization + +The continuous stage represents each output pixel by a source UV/sample coordinate. + +Requirements: +- operate in premultiplied RGBA space +- use PyTorch autograd +- optimize a UV field initialized from the inferred lattice +- preserve local adjacency instead of letting the sample field scramble neighborhoods + +Loss components should include: +- reconstruction loss against the source image +- low-distortion regularization against the initialized UV grid +- edge-aware smoothing so hard boundaries stay crisp +- anti-speckle pressure to reduce isolated outlier pixels +- alpha regularization to avoid muddy semi-transparent edges + +### 4. Discrete projection and cleanup + +After the continuous stage, the result must be treated as a true pixel grid. + +The discrete stage should: +- evaluate local neighborhoods +- remove orphan pixels where possible +- merge obvious micro-clusters +- reduce accidental highlight static +- avoid damaging strong guided boundaries + +This stage is intentionally heuristic and local in v1. + +### 5. Baselines and comparison mode + +The project must ship comparison baselines: +- naive resize +- resize plus error diffusion + +`compare` mode should run: +- the optimized solver +- both baselines +- metric collection +- a visual contact sheet + +The project should treat these baselines as the minimum bar for usefulness. + +## Diagnostics + +Diagnostics should be machine-readable and human-readable. + +Required artifacts: +- `run.json` +- `lattice-overlay.png` +- `comparison.png` +- `alpha-preview.png` +- `noise-heatmap.png` +- `cluster-preview.png` + +Compare mode should additionally write: +- `compare.json` +- `compare.csv` +- `compare-sheet.png` + +## Acceptance criteria + +The tool is successful for v1 if it can: +- infer sensible target sizes on curated synthetic cases +- preserve transparency and silhouette quality +- produce outputs that are visibly more grid-coherent than simple resize baselines +- avoid obvious isolated-pixel explosions +- generate diagnostics that explain how and why a run behaved the way it did + +## Future expansion + +The design should leave room for: +- manual region hints or lock zones +- multi-frame consistency for sprite sheets +- GUI tooling +- more advanced warping or lattice-field models +- learned priors or ranking models for candidate outputs