Document the repixelizer project context
This commit is contained in:
Meta Vens
parent
aceefe46b2
commit
634ca7f002
3 changed files with 447 additions and 32 deletions
117
docs/implementation-plan.md
Normal file
117
docs/implementation-plan.md
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
# Repixelizer Implementation Plan
|
||||
|
||||
## Current state
|
||||
|
||||
The project is bootstrapped and runnable as a standalone Python repo.
|
||||
|
||||
Implemented:
|
||||
- package metadata and editable install
|
||||
- CLI entrypoint with `run` and `compare` modes
|
||||
- lattice/phase inference with coherence scoring plus source-period prior
|
||||
- source-image structural analysis
|
||||
- PyTorch-based continuous UV-field optimization
|
||||
- discrete cleanup on the output grid
|
||||
- optional palette loading and quantization modes
|
||||
- compare mode with naive and diffusion baselines
|
||||
- diagnostics writing
|
||||
- synthetic test fixtures and automated tests
|
||||
|
||||
Verified:
|
||||
- local editable install in a dedicated venv
|
||||
- full test suite currently passing
|
||||
- compare-mode smoke run against a real emblem image
|
||||
|
||||
## What is still provisional
|
||||
|
||||
The current implementation should be treated as a strong experimental baseline, not a final algorithm.
|
||||
|
||||
The most provisional areas are:
|
||||
- lattice inference on real, messy generated imagery
|
||||
- the exact loss weighting in the continuous stage
|
||||
- the strength and shape of discrete cleanup heuristics
|
||||
- compare-mode metrics as a proxy for actual visual quality
|
||||
|
||||
## Near-term priorities
|
||||
|
||||
### 1. Build a real benchmark corpus
|
||||
|
||||
Add curated real-world cases, not just synthetic ones:
|
||||
- emblems and logos with transparency
|
||||
- icons with strong outlines
|
||||
- simple sprite-like figures
|
||||
- examples that are known to fail naive resize-and-dither
|
||||
|
||||
Each case should include:
|
||||
- source image
|
||||
- expected target size or plausible size range
|
||||
- short notes on known failure modes
|
||||
|
||||
### 2. Improve lattice inference
|
||||
|
||||
The current inference is workable but still heuristic.
|
||||
|
||||
Next improvements should focus on:
|
||||
- better periodicity detection for noisy or partially aligned images
|
||||
- stronger penalties against oversized candidate grids
|
||||
- confidence calibration that reflects ambiguity more honestly
|
||||
|
||||
### 3. Strengthen the discrete cleanup stage
|
||||
|
||||
Current cleanup is local and generic.
|
||||
|
||||
Next version should add:
|
||||
- stronger outline continuity rules
|
||||
- better preservation of highlight bands
|
||||
- better handling of metallic or gem-like materials
|
||||
- smarter alpha-edge cleanup for transparent icons
|
||||
|
||||
### 4. Make compare mode more trustworthy
|
||||
|
||||
The current metrics are useful but still internal.
|
||||
|
||||
Next work should:
|
||||
- expand score breakdowns
|
||||
- validate whether the metrics correlate with actual human preference
|
||||
- make it easier to compare runs across parameter sweeps
|
||||
|
||||
## Suggested next milestones
|
||||
|
||||
### Milestone 1: Real-image evaluation harness
|
||||
|
||||
- add curated real-image fixtures
|
||||
- add golden metadata for expected target-size ranges
|
||||
- add a repeatable batch runner
|
||||
- emit one summary table per corpus run
|
||||
|
||||
### Milestone 2: Solver tuning pass
|
||||
|
||||
- tune continuous-stage loss weights against the benchmark corpus
|
||||
- test more robust anti-speckle and cluster-preservation behavior
|
||||
- reduce cases where the solver underperforms the naive baseline
|
||||
|
||||
### Milestone 3: Better cleanup heuristics
|
||||
|
||||
- add contour-aware cleanup
|
||||
- add material-sensitive smoothing for highlights and trim
|
||||
- reduce the number of visually good-but-metric-weak outputs
|
||||
|
||||
### Milestone 4: Human-in-the-loop v2 planning
|
||||
|
||||
Only after the automatic baseline is strong:
|
||||
- explore optional region hints
|
||||
- evaluate a minimal GUI or notebook layer
|
||||
- consider multi-frame support
|
||||
|
||||
## Risks
|
||||
|
||||
- The solver can still produce outputs that score well but feel over-smoothed.
|
||||
- Real generated images may contain multiple inconsistent local lattice cues.
|
||||
- Continuous optimization is flexible, but easy to overfit to the reconstruction term.
|
||||
- Metrics that reward coherence can accidentally reward blandness if not balanced carefully.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- Single-image quality matters more than batch throughput in the current phase.
|
||||
- Palette constraints are optional and should stay optional.
|
||||
- The project is better served by improving automatic behavior first than by rushing into manual tooling.
|
||||
- Python plus PyTorch remains the right stack for the current research/prototyping phase.
|
||||
203
docs/spec.md
Normal file
203
docs/spec.md
Normal file
|
|
@ -0,0 +1,203 @@
|
|||
# Repixelizer Spec
|
||||
|
||||
## Summary
|
||||
|
||||
Repixelizer is a standalone Python CLI for converting "fake pixel art" into true grid-aligned pixel art. The target use cases are single-image icons, emblems, logos, and simple sprites, especially images that already resemble pixel art but fail to obey a consistent pixel lattice.
|
||||
|
||||
The project exists because generated imagery often creates appealing pixel-like local patterns while violating the actual constraints of pixel art:
|
||||
- inconsistent subpixel alignment
|
||||
- locally plausible but globally incoherent clusters
|
||||
- noisy highlights and outlines
|
||||
- fake micro-detail that collapses when downsampled
|
||||
|
||||
Repixelizer treats this as an optimization and reconstruction problem rather than a resizing problem.
|
||||
|
||||
## Goals
|
||||
|
||||
- Recover an implied target pixel grid from a fake-pixel-art source image.
|
||||
- Produce a true discrete-grid output image that looks authored, not merely resized.
|
||||
- Preserve alpha, silhouette clarity, and strong interior structure.
|
||||
- Support palette-free output by default, with optional palette-constrained modes.
|
||||
- Provide diagnostics and baseline comparisons so results are inspectable and measurable.
|
||||
|
||||
## Non-goals for v1
|
||||
|
||||
- full sprite-sheet or animation consistency
|
||||
- manual painting or mask authoring as a required workflow
|
||||
- a desktop GUI
|
||||
- generalized photo-to-pixel-art stylization
|
||||
- a learned model trained on a large dataset
|
||||
|
||||
## Primary users
|
||||
|
||||
- artists or developers trying to salvage generated fake pixel art
|
||||
- tool builders experimenting with constrained repixelization
|
||||
- game/UI developers creating icons, emblems, badges, and simple sprites
|
||||
|
||||
## Input and output contract
|
||||
|
||||
### Input
|
||||
|
||||
- one RGBA image file
|
||||
- optional explicit target size override
|
||||
- optional palette file
|
||||
- optional diagnostics directory
|
||||
- optional solver/device settings
|
||||
|
||||
### Output
|
||||
|
||||
- one final RGBA PNG
|
||||
- optional palette-constrained output behavior
|
||||
- optional diagnostics bundle containing JSON and visual artifacts
|
||||
|
||||
## Product behavior
|
||||
|
||||
### Default mode
|
||||
|
||||
The default workflow should be fully automatic:
|
||||
|
||||
1. load the source image
|
||||
2. infer lattice size and phase
|
||||
3. analyze source structure
|
||||
4. optimize a continuous UV/sample field
|
||||
5. project that result to a real pixel grid
|
||||
6. clean up obvious discrete-grid artifacts
|
||||
7. optionally quantize to a palette
|
||||
8. write output and diagnostics
|
||||
|
||||
No manual masks or user-authored region hints are assumed in v1.
|
||||
|
||||
### CLI shape
|
||||
|
||||
Primary commands:
|
||||
|
||||
```powershell
|
||||
repixelize input.png --out output.png
|
||||
repixelize compare input.png --out output.png
|
||||
```
|
||||
|
||||
Supported flags:
|
||||
- `--target-size`
|
||||
- `--palette`
|
||||
- `--palette-mode off|fit|strict`
|
||||
- `--diagnostics-dir`
|
||||
- `--seed`
|
||||
- `--steps`
|
||||
- `--device cpu|cuda`
|
||||
|
||||
### Palette behavior
|
||||
|
||||
- `off`: produce unconstrained RGBA output
|
||||
- `fit`: adapt to a useful palette derived from or compatible with the result
|
||||
- `strict`: stay strictly inside a supplied palette
|
||||
|
||||
Palette constraints are optional because many generated fake-pixel-art images do not originate from a coherent palette.
|
||||
|
||||
## Technical design
|
||||
|
||||
### 1. Lattice inference
|
||||
|
||||
The tool must estimate the resolution of the fake lattice being mimicked.
|
||||
|
||||
Requirements:
|
||||
- search over candidate target sizes
|
||||
- search over subpixel phase offsets for each candidate
|
||||
- score candidates using coherence-oriented metrics rather than reconstruction alone
|
||||
- expose the top-ranked alternatives in diagnostics
|
||||
- if `--target-size` is provided, skip size search but still estimate phase
|
||||
|
||||
Scoring goals:
|
||||
- low isolated-pixel rate
|
||||
- high cluster continuity
|
||||
- strong alpha-edge crispness
|
||||
- straighter outlines
|
||||
- lower local color chatter
|
||||
- reasonable agreement with source-derived periodicity priors
|
||||
|
||||
### 2. Source analysis
|
||||
|
||||
The solver should derive automatic guidance from the image itself:
|
||||
- alpha map
|
||||
- edge map
|
||||
- coarse material/color clustering
|
||||
|
||||
This guidance is used to preserve important boundaries while allowing smoother regions to collapse into cleaner pixel clusters.
|
||||
|
||||
### 3. Continuous optimization
|
||||
|
||||
The continuous stage represents each output pixel by a source UV/sample coordinate.
|
||||
|
||||
Requirements:
|
||||
- operate in premultiplied RGBA space
|
||||
- use PyTorch autograd
|
||||
- optimize a UV field initialized from the inferred lattice
|
||||
- preserve local adjacency instead of letting the sample field scramble neighborhoods
|
||||
|
||||
Loss components should include:
|
||||
- reconstruction loss against the source image
|
||||
- low-distortion regularization against the initialized UV grid
|
||||
- edge-aware smoothing so hard boundaries stay crisp
|
||||
- anti-speckle pressure to reduce isolated outlier pixels
|
||||
- alpha regularization to avoid muddy semi-transparent edges
|
||||
|
||||
### 4. Discrete projection and cleanup
|
||||
|
||||
After the continuous stage, the result must be treated as a true pixel grid.
|
||||
|
||||
The discrete stage should:
|
||||
- evaluate local neighborhoods
|
||||
- remove orphan pixels where possible
|
||||
- merge obvious micro-clusters
|
||||
- reduce accidental highlight static
|
||||
- avoid damaging strong guided boundaries
|
||||
|
||||
This stage is intentionally heuristic and local in v1.
|
||||
|
||||
### 5. Baselines and comparison mode
|
||||
|
||||
The project must ship comparison baselines:
|
||||
- naive resize
|
||||
- resize plus error diffusion
|
||||
|
||||
`compare` mode should run:
|
||||
- the optimized solver
|
||||
- both baselines
|
||||
- metric collection
|
||||
- a visual contact sheet
|
||||
|
||||
The project should treat these baselines as the minimum bar for usefulness.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
Diagnostics should be machine-readable and human-readable.
|
||||
|
||||
Required artifacts:
|
||||
- `run.json`
|
||||
- `lattice-overlay.png`
|
||||
- `comparison.png`
|
||||
- `alpha-preview.png`
|
||||
- `noise-heatmap.png`
|
||||
- `cluster-preview.png`
|
||||
|
||||
Compare mode should additionally write:
|
||||
- `compare.json`
|
||||
- `compare.csv`
|
||||
- `compare-sheet.png`
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
The tool is successful for v1 if it can:
|
||||
- infer sensible target sizes on curated synthetic cases
|
||||
- preserve transparency and silhouette quality
|
||||
- produce outputs that are visibly more grid-coherent than simple resize baselines
|
||||
- avoid obvious isolated-pixel explosions
|
||||
- generate diagnostics that explain how and why a run behaved the way it did
|
||||
|
||||
## Future expansion
|
||||
|
||||
The design should leave room for:
|
||||
- manual region hints or lock zones
|
||||
- multi-frame consistency for sprite sheets
|
||||
- GUI tooling
|
||||
- more advanced warping or lattice-field models
|
||||
- learned priors or ranking models for candidate outputs
|
||||
Loading…
Reference in a new issue