repixelizer/docs/tile-graph-algorithm-map.md

# Tile-Graph Algorithm Map

This is the end-to-end map of the current `tile-graph` path as it exists in the repo today.

It is written specifically to answer one question:

Why can a pinned lattice like `126x126` / phase `(0.0, -0.2)` produce a completely garbled result instead of merely a slightly worse one?

Short answer:

- the fixed-lattice pipeline wrapper is not corrupting the image
- the corruption is already present in the `tile-graph` initial assignment
- the current `tile-graph` implementation is much more lattice-conditioned than the intended "extract source-owned one-cell tiles, then place them" design

On the current broken badge run:

- source image: `1254x1254`
- fixed lattice: `126x126`
- phase: `(0.0, -0.2)`
- `tile_graph_initial_source_fidelity = 0.500884`
- `tile_graph_final_source_fidelity = 0.499863`
- every output cell that contains opaque sampled source pixels now does receive at least one extracted region candidate under the corrected full-size lattice mapping

That tiny change means the parity-update loop is not the main failure. The model build and initial assignment are.

## Bird's-Eye Flow

```mermaid
flowchart TD
    A["source_rgba (H x W x 4)"] --> B["run_pipeline(...)"]
    B --> C["infer_lattice(...) or infer_fixed_lattice(...)"]
    B --> D["analyze_source(...)"]
    C --> E["InferenceResult"]
    D --> F["SourceAnalysis"]
    E --> G["phase rerank probes (optional)"]
    F --> G
    G --> H["optimize_tile_graph(...)"]
    H --> I["build_source_lattice_reference(...)"]
    H --> J["_segment_atomic_source_regions(...)"]
    J --> K["_extract_source_region_tiles(...)"]
    I --> L["per-output-cell references"]
    K --> M["region buckets per output coord"]
    L --> N["build_tile_graph_model(...)"]
    M --> N
    N --> O["initial unary argmin assignment"]
    O --> P["checkerboard pairwise refinement"]
    P --> Q["keep refined or revert to initial"]
    Q --> R["cleanup_pixels(...)"]
    R --> S["diagnostics/run.json"]
```

## Core Data Objects

### `source_rgba`

- Type: `np.ndarray`
- Shape: `(H, W, 4)`
- Value range: float32-like normalized RGBA in `[0, 1]`
- Meaning: the loaded source facsimile, optionally background-stripped before anything else

### `InferenceResult`

- File: `src/repixelizer/types.py`
- Fields:
  - `target_width`, `target_height`: chosen output lattice size
  - `phase_x`, `phase_y`: sub-cell phase offsets in lattice units
  - `confidence`: gap between top two inference candidates
  - `top_candidates`: rerankable list of `InferenceCandidate`

### `SourceAnalysis`

- File: `src/repixelizer/types.py`
- Fields:
  - `edge_map`: per-source-pixel edge strength
  - `cluster_map`: k-means color labels
  - `cluster_centers`: color centroids
  - `alpha_map`: source alpha
  - `cluster_preview`: diagnostic preview

Important note:

- `cluster_map` is diagnostic/coarse analysis data
- it is not the current tile-graph region ownership model

### `SourceLatticeReference`

- File: `src/repixelizer/types.py`
- Meaning: lattice-conditioned summary of the source under one exact `(target_width, target_height, phase_x, phase_y)`
- Fields:
  - `mean_rgba`: average premultiplied color per inferred cell, then unpremultiplied
  - `sharp_rgba`: one chosen exemplar pixel per inferred cell
  - `dispersion`: average per-pixel deviation from cell mean
  - `lattice_indices`: source-pixel to inferred-cell assignment map
  - `cell_dispersion`, `cell_counts`, `cell_support`, `cell_alpha_max`
  - `sharp_x`, `sharp_y`
  - `edge_peak_x`, `edge_peak_y`
  - `edge_strength`, `edge_grad_x`, `edge_grad_y`
  - `delta_x`, `delta_y`, `delta_diag`, `delta_anti`

### `TileGraphModel`

- File: `src/repixelizer/tile_graph.py`
- Meaning: the fully built discrete candidate model for one source + one fixed lattice
- Fields:
  - `candidate_rgba`: all candidate pixel colors
  - `candidate_coords`: output coord `(y, x)` each candidate belongs to
  - `candidate_area_ratio`: how much of a cell-sized source window the source region covered
  - `candidate_coverage`: clipped version of area coverage
  - `candidate_deltas`: expected RGBA deltas to right/down/left/up neighbors sampled one cell away in source space
  - `cell_candidate_offsets`, `cell_candidate_indices`: CSR-like mapping from output cells to candidate rows
  - `reference_mean_rgba`, `reference_sharp_rgba`, `reference_edge_rgba`
  - `edge_strength`
  - `component_count`, `edge_density`, `average_choices`
  - `geometry_reference_rgba`, `geometry_strength`: optional hybrid-mode priors

## Stage 1: Pipeline Entry

File: `src/repixelizer/pipeline.py`

Main entry:

- `run_pipeline(...)`

Main variables:

- `source`: loaded RGBA source image
- `fixed_dims`: either `None` or exact `(target_width, target_height)` resolved from CLI
- `inference`: one `InferenceResult`
- `analysis`: one `SourceAnalysis`
- `cached_reconstruction`: optional reused probe result from phase rerank
- `solver`: `SolverArtifacts` from continuous, tile-graph, or hybrid
- `cleanup`: `CleanupArtifacts`
- `output_rgba`: final saved output

Control flow:

1. Load source.
2. Optionally preprocess background.
3. Resolve fixed-vs-searched lattice mode.
4. Run source analysis.
5. Optionally rerank top lattice candidates by actually reconstructing them.
6. Run the selected reconstruction mode.
7. Run cleanup and optional palette quantization.
8. Save output and diagnostics.

Important finding:

- The fixed-lattice pipeline path is behaving correctly.
- Running the same pinned `126x126` tile-graph case through:
  - direct `_run_reconstruction(...)`
  - `run_pipeline(...)`
  - the phase-rerank probe path
  produces the exact same output bytes.

So the pipeline wrapper is not introducing the corruption.

## Stage 2: Lattice Inference

File: `src/repixelizer/inference.py`

Two entry points:

- `infer_lattice(...)`
- `infer_fixed_lattice(...)`

### Searched path

`infer_lattice(...)` does:

1. Estimate spacing priors from source luminance/alpha changes.
2. Build candidate output sizes with `_candidate_dims(...)`.
3. For each size, score a `5x5` phase grid with `_score_phase_group(...)`.
4. Keep one best phase per size.
5. Rerank sizes with `source_lattice_evidence_breakdown(...)`.

Main variables:

- `spacing_x`, `spacing_y`: measured source spacing hints
- `hinted_sizes`: size hints derived from spacing
- `prior_cell_x`, `prior_cell_y`, `prior_reliability`: size prior for scoring
- `phase_values`: `[-0.4, -0.2, 0.0, 0.2, 0.4]`
- `candidates`: all scored `(size, phase)` candidates
- `size_candidates`: top one phase per size
- `reranked_candidates`: size candidates reordered by source-lattice evidence

### Fixed path

`infer_fixed_lattice(...)` does:

1. Keep the target size fixed.
2. If phase is pinned, score exactly that one phase.
3. If phase is not pinned, search only the phase grid for that fixed size.

Important consequence:

- fixing the lattice size does not merely "skip search"
- it locks in the exact cell size and phase that every downstream tile-graph stage uses

## Stage 3: Source Analysis

File: `src/repixelizer/analysis.py`

Entry point:

- `analyze_source(rgba, seed, cluster_count=6, device=None)`

Outputs:

- `edge_map`: edge strength from luminance plus alpha differences
- `cluster_map`: k-means labels on opaque pixels
- `cluster_centers`: k-means centroids
- `alpha_map`
- `cluster_preview`

Important reality check:

- tile-graph no longer uses `cluster_map` as its actual tile ownership map
- the tile-graph region ownership is built later by connected components over source pixels using color/alpha thresholds

## Stage 4: Optional Phase Rerank

File: `src/repixelizer/pipeline.py`

Helper:

- `_select_phase_candidate_with_reconstruction(...)`

What it does:

1. Only runs when inference confidence is low and there are multiple top candidates.
2. Reconstructs several top candidates with the active mode.
3. Scores each candidate by:
  - source support (`source_lattice_consistency_breakdown`)
  - edge position error
  - stroke wobble
  - edge concentration
  - size delta penalty
  - inference score penalty
4. Picks the best reranked candidate if it clears the configured margin.

Important note for tile-graph:

- this rerank is pipeline-level, not optimizer-specific
- today `optimize_tile_graph(...)` ignores `steps`, so tile-graph rerank probes are basically full tile-graph runs

That is a performance problem, but not the cause of the fixed-126 corruption.

## Stage 5: Source Lattice Reference

File: `src/repixelizer/source_reference.py`

Entry point:

- `build_source_lattice_reference(...)`

This is one of the most important stages, because it ties the source image to one specific inferred lattice.

### Inputs

- `source_rgba`
- `target_width`, `target_height`
- `phase_x`, `phase_y`
- `alpha_threshold`
- optional `edge_hint`

### What it computes

1. `lattice_indices(...)`
   - assigns every source pixel to one inferred lattice cell
   - this is a hard rectangular partition based on current size and phase
2. `mean_rgba`
   - per-cell premultiplied mean, then unpremultiplied
3. `sharp_rgba`
   - one exemplar per cell
   - chosen by minimizing `exemplar_cost = pixel_diff + tiny alpha underfill penalty`
4. `edge_peak`
   - strongest edge-supporting pixel per cell
5. `edge_strength`
   - strongest edge value seen in that cell
6. `cell_dispersion`
   - how mixed the cell is
7. `delta_x`, `delta_y`, `delta_diag`, `delta_anti`
   - deltas between neighboring `sharp` cells

### Critical variables

- `indices` / `indices_t`: source-pixel to cell assignment
- `flat_idx`
- `mean_premul_flat`
- `pixel_diff`: per-source-pixel deviation from the current cell mean
- `exemplar_cost`: what decides `sharp_rgba`
- `edge_score`
- `sharp_x_flat`, `sharp_y_flat`
- `edge_peak_x_flat`, `edge_peak_y_flat`

### Important consequence

The current tile-graph path is already strongly lattice-conditioned here.

If the pinned lattice is wrong for the source:

- `lattice_indices` groups the wrong source pixels together
- `mean_rgba` becomes the wrong per-cell average
- `sharp_rgba` becomes the wrong per-cell exemplar
- `edge_peak` becomes the wrong per-cell anchor

That poisoning then propagates into tile-graph candidate selection.

## Stage 6: Atomic Source Region Segmentation

File: `src/repixelizer/tile_graph.py`

Entry points:

- `_segment_atomic_source_regions_cpu(...)`
- `_segment_atomic_source_regions(...)`

This stage does not use inferred cells yet. It runs on the source image or a strided sample of it.

### Inputs

- `source_rgba` or `sampled_rgba`
- `edge_map` or `sampled_edge`
- `alpha_floor`
- `color_threshold`
- `alpha_threshold`
- `device`

### What it does

1. Mark opaque pixels.
2. Build `join_right` and `join_down` booleans for neighboring source pixels whose premultiplied color and alpha are close enough.
3. Run connected-component labeling over those joins.
4. For each component, compute:
  - `member_linear`
  - `edge_peak_linear`
  - `centroid_linear`
  - `centroid_x`, `centroid_y`
  - `axis_x`, `axis_y`
  - `linearity`
  - `major_span`, `minor_span`
  - `size`

### Important reality check

This is not "single-color clusters" in the strict literal sense the original idea implied.

It is:

- thresholded connected components in premultiplied RGBA space
- optionally on a strided sample of the source

So even before tile extraction, the current implementation is already an approximation.

## Stage 7: Cutting Source Regions Into Tile Proposals

File: `src/repixelizer/tile_graph.py`

Entry point:

- `_extract_source_region_tiles(...)`

This is the real source-side tile cutter.

### Inputs

- `components`
- `flat_rgba`, `flat_edge`, `flat_x`, `flat_y`
- `cell_w`, `cell_h`
- `sample_area`
- `target_width`, `target_height`
- `phase_x`, `phase_y`
- `min_region_area_ratio`
- `min_window_coverage`
- stroke slicing params

### What it does

For each connected source component:

1. Sort components by closeness of `component["size"]` to one output-cell area.
2. Decide whether the component is "stroke-like" from:
  - `linearity`
  - `major_span`
  - `minor_span`
3. Create seed centers:
  - centroid and edge peak for regular regions
  - principal-axis marching seeds for stroke-like regions
4. For each seed:
  - build a local window mask
  - count `footprint_count`
  - compute `area_ratio = footprint_count * sample_area / cell_area`
  - reject tiny windows
5. Accept the window and choose one representative source pixel:
  - edge peak for non-strokes with edge content
  - center-nearest pixel for strokes or flat regions
6. Project the accepted window center back to one output coord with `_project_source_point_to_output_coord(...)`
7. Append one proposal into that output cell's region bucket
8. Mark the consumed source footprint as unavailable within that component using `remaining[in_window] = False`

### Important variables

- `remaining`
- `seed_queue`
- `accepted_any`
- `stroke_component`
- `stroke_major_half`, `stroke_minor_half`
- `rep_linear`
- `area_ratio`
- `coverage`
- `coord_x`, `coord_y`
- `flat_index`

### Important consequence

This stage is still controlled by the inferred lattice through:

- `cell_w`, `cell_h`
- `target_width`, `target_height`
- `phase_x`, `phase_y`
- the projection from source window center to output coord

So changing the fixed lattice changes the actual tile cutting behavior, not just the later scoring.

## Stage 8: Per-Cell Candidate Selection

File: `src/repixelizer/tile_graph.py`

Helpers:

- `_select_source_region_candidates(...)`
- `build_tile_graph_model(...)`

### `build_tile_graph_model(...)` main flow

1. Resolve geometry prior inputs if hybrid mode is active.
2. Check the process-local tile-graph cache.
3. Build `source_reference`.
4. Compute `cell_w`, `cell_h`.
5. Decide `source_region_stride`.
6. Build `sampled_rgba` and `sampled_edge`.
7. Segment components.
8. Extract region tiles into `region_buckets`.
9. For each output coord:
  - decide whether it is an edge cell from `source_reference.edge_strength`
  - collect region candidates from `region_buckets[flat_index]`
  - downselect them with `_select_source_region_candidates(...)`
  - if no region candidates survive, fall back to `sharp` and optionally `edge_peak`
10. Build candidate arrays and expected neighbor deltas.

### `_select_source_region_candidates(...)`

For one output cell, this ranks region proposals by:

- closeness of `area_ratio` to `1.0`
- closeness to `reference_rgba`
- closeness to `edge_reference_rgba`
- `edge_peak`

It then keeps only a small cap:

- normal cells: `tile_graph_max_candidates_per_coord` (default `2`)
- edge cells: `tile_graph_edge_candidates_per_coord` (default `6`)

### Important consequence

Even if source-region extraction found the right candidate, it can still be dropped here if:

- too many proposals land in the same output cell
- the candidate is not close enough to the lattice-conditioned references

## Stage 9: What Actually Lives In `TileGraphModel`

After candidate selection, the model stores:

- `candidate_rgba`
  - literal source-pixel colors, one row per candidate
- `candidate_coords`
  - which output cell each candidate belongs to
- `candidate_area_ratio`
  - window size match score proxy
- `candidate_coverage`
  - clipped area coverage proxy
- `candidate_deltas`
  - sampled right/down/left/up expected deltas one cell away in source space
- `reference_mean_rgba`
  - lattice-conditioned per-cell mean
- `reference_sharp_rgba`
  - lattice-conditioned per-cell exemplar
- `reference_edge_rgba`
  - source pixel at the per-cell edge peak
- `edge_strength`

So the final discrete solver is not operating on raw tiles alone.

It is operating on:

- source-owned candidate colors
- scored against lattice-conditioned references
- connected with deltas sampled from source-space positions offset by the inferred cell size

## Stage 10: Choice Grid And Unary Cost

Helpers:

- `_build_choice_grid(...)`
- `_tile_graph_unary_cost(...)`
- `_tile_graph_unary_cost_torch(...)`

### `_build_choice_grid(...)`

Transforms CSR-style cell candidate storage into dense tensors:

- `choice_indices`: `(target_h, target_w, max_choices)`
- `choice_mask`: same shape, marks real entries vs padding

### Unary cost per candidate

For each output cell and each candidate:

1. `sharp_error`
2. `mean_error`
3. `edge_error`
4. If edge cell:
  - `color_error = min(sharp_error, edge_error) + mean_error * tile_graph_edge_mean_weight`
5. Else:
  - `color_error = sharp_error * nonedge_sharp_weight + mean_error * nonedge_mean_weight`
6. Add:
  - `area_error`
  - `alpha_error`
  - `coverage_error`
7. Optionally add hybrid geometry error.

Important implication:

- the first assignment is not "pick the source region candidate that owns the tile"
- it is "pick the lowest lattice-conditioned unary cost among the allowed candidates for this output coord"

## Stage 11: Initial Assignment

File: `src/repixelizer/tile_graph.py`

In `optimize_tile_graph(...)`:

1. Build model.
2. Build dense choice tensors.
3. Compute `unary_cost_t`.
4. Choose:
   - `initial_choice_t = argmin(unary_cost_t, dim=2)`
   - `selected_t = choice_indices_t[...]`
5. Save:
   - `initial_selected_t = selected_t.clone()`

That initial assignment becomes:

- `initial_rgba = _assignment_rgba(model, initial_selected)`

This is where the fixed `126x126` badge is already broken.

Measured on the bad run:

- `tile_graph_initial_source_fidelity = 0.500884`
- `tile_graph_final_source_fidelity = 0.499863`

So the badness is already in `initial_choice_t`.

## Stage 12: Pairwise Refinement

Still in `optimize_tile_graph(...)`

After the unary argmin, the solver runs a checkerboard parity update loop for `tile_graph_iterations`.

What it adds:

- right-neighbor pair penalty
- left-neighbor pair penalty
- down-neighbor pair penalty
- up-neighbor pair penalty

These penalties compare:

- observed delta between two chosen candidate colors
- expected delta sampled for each candidate in that direction

The solver alternates black/white checkerboard parities so updates are easy to parallelize.

Important finding:

- this loop is not the main cause of the fixed-126 corruption
- it barely changes the broken initial assignment

## Stage 13: Final Revert Guard

At the end of `optimize_tile_graph(...)`:

1. Build `initial_rgba` from initial choices.
2. Build `target_rgba` from refined choices.
3. Score both with `source_lattice_consistency_breakdown(...)`.
4. If refined is worse, revert to initial.

This is why recent tile-graph passes stopped smearing as badly:

- the solver is prevented from making a decent initial assignment worse

But it cannot rescue a bad initial assignment.

## Stage 14: Cleanup And Diagnostics

Files:

- `src/repixelizer/discrete.py`
- `src/repixelizer/diagnostics.py`

### Cleanup

`cleanup_pixels(...)` defaults to `iterations=0`.

That means:

- cleanup is currently a no-op unless explicitly changed
- it is not causing the observed corruption in this badge case

### Diagnostics

`summarize_run(...)` records:

- `source_fidelity.snap_initial`
- `source_fidelity.solver_target`
- `source_fidelity.final_output`
- `phase_rerank_candidates`
- `loss_history`

## What The Broken Fixed-126 Run Is Actually Doing

Directly measured on `tests/fixtures/real/ai-badge-cleaned.png` with pinned `126x126` / `(0.0, -0.2)`:

- source shape: `1254x1254`
- inferred cell size: about `9.95 x 9.95`
- `source_region_stride = 2`
- connected components after segmentation: `2499`
- output cells: `15876`
- edge cells by `source_reference.edge_strength`: `1610`
- output cells with any opaque sampled source pixel under the full-size lattice mapping: `8167`
- output cells with any extracted source-region bucket after the empty-cell fill pass: `8169`
- occupied output cells still missing extracted region buckets: `0`
- output cells with no extracted region bucket are now background-only cells by design: `7707`

What that means:

- connected-components still ignore pixels below the alpha threshold, so background-only cells do not get source-region buckets from this stage
- an earlier draft of this document overstated the loss by comparing sampled pixels against `lattice_indices(...)` on the downsampled grid instead of projecting sampled source coordinates back onto the full-size lattice; that comparison was wrong and has now been corrected
- the missing-bucket bug on occupied cells was real and is now fixed
- the fixed `126x126` output is still broken anyway, so empty extracted buckets were not the dominant cause of the collapse

So if the pinned lattice is poor for tile-graph, the fallback path is poisoned immediately.

## Where Collapse Can First Appear

In order of appearance:

1. `infer_fixed_lattice(...)`
   - locks in a size/phase that all later steps must obey
2. `build_source_lattice_reference(...)`
   - hard-assigns source pixels into inferred cells
   - creates `mean_rgba`, `sharp_rgba`, `edge_peak` from those cells
3. `_extract_source_region_tiles(...)`
   - cuts source components using windows sized by the inferred cell dimensions
   - projects accepted windows back onto output coords using the inferred phase
4. `_select_source_region_candidates(...)`
   - truncates each output cell down to a tiny local candidate set
5. initial unary argmin
   - chooses the final initial candidate from lattice-conditioned references

The fixed-126 badge collapse is already visible at step 5, but it can be caused by bad state introduced in steps 2 through 4.

## Pondering It

### 1. The current tile-graph path is not lattice-agnostic

Fixing the lattice size changes all of these at once:

- source-pixel to cell assignment
- per-cell exemplar and edge anchors
- source-region window size
- source-region projection target
- stride used for region extraction
- fallback candidate identity

So in the current implementation, a pinned size can absolutely collapse the result.

That is not a pipeline bug. It is a design property.

### 2. The current implementation is still not the pure intended algorithm

The intended idea was closer to:

- find atomic source tiles first
- consume them
- build adjacency from them
- place them on the output grid

The current implementation is instead:

- infer a lattice first
- build lattice-conditioned source references
- cut source regions using windows sized by that lattice
- project those windows into output cells using that lattice
- rank them against lattice-conditioned references

That is a different algorithm.

### 3. The fixed-126 failure is mostly an initial-assignment failure, not a solver failure

The measured gap:

- initial: `0.500884`
- final: `0.499863`

is tiny.

So the parity refinement loop is not "destroying" a good layout. It is starting from a bad one.

### 4. The missing-bucket bug was real, but it was not the main collapse

The extraction stage did have a real bug:

- it could emit source-region windows by center projection and still leave some occupied output cells without a source-region candidate

That is now fixed by a final overlap-based fill pass, and with the corrected full-size lattice mapping:

- occupied output cells missing extracted buckets: `0`

But the fixed `126x126` output is still effectively unchanged after that fix.

That means the dominant collapse is not "foreground cells had no extracted region bucket at all."
It is happening later, through the way lattice-conditioned references and unary ranking choose between the candidates that do exist.

### 5. The fixed-126 run is harsher because it flips extraction stride

On this badge:

- `126x126` uses `source_region_stride = 2`
- the stronger `162x162` tile-graph badge run used `source_region_stride = 1`

That means the broken fixed-126 run is not just using bigger cell windows.

It is also:

- sampling the source more coarsely during component labeling
- sampling the source more coarsely during region cutting
- making representative-pixel choice from a coarser source sample

That makes thin details easier to miss before the solver even starts.

### 6. Candidate truncation is still severe

Even when multiple region proposals land in the same output cell:

- non-edge cells keep at most `2`
- edge cells keep at most `6`

So a real source-owned candidate can still be dropped if the lattice-conditioned references like other candidates better.

## What I Think This Means

The most honest current description of tile-graph is:

"A lattice-conditioned source-region candidate generator plus a local discrete solver."

It is not yet:

"A source-owned tile extraction system that merely uses the lattice as a final placement scaffold."

That distinction explains why fixing the lattice can collapse the result so dramatically.

## The Most Important Current Findings

1. The fixed-lattice pipeline path is correct.
   The corruption is not being added by the wrapper.

2. The extraction stage really did need an exhaustiveness fix.
   Occupied output cells are no longer silently losing their source-region bucket under the corrected full-size lattice mapping.

3. The tile-graph initial assignment is still bad on the broken `126x126` case even after that fix.
   So the remaining problem lives in lattice-conditioned reference building, candidate truncation, and unary selection before pairwise refinement has much chance to help.