repixelizer/docs/tile-graph-algorithm-map.md

Tile-Graph Algorithm Map

Why this map exists

The tile-graph path is trying to do one simple, stubborn thing:

Take a fake-pixel mural, cut it into honest one-cell tiles that really belong to the source image, learn which tiles naturally sit beside which, and then lay those tiles back down on a clean output lattice.

That is the dream.

When the output looks wrong, this map is here to answer one question:

Where, exactly, did the machine start lying?

One-sentence machine

The current tile-graph machine is:

source image -> edge scout -> atomic opaque regions -> one-cell tile cuts -> per-cell candidate buckets -> learned tile adjacency -> local discrete assignment

Or in more visual language:

• the source image is the mural
• connected components are the paint islands
• extracted tiles are the stones we chip out of that mural
• candidate buckets are the bins beside each slot in the final mosaic
• the solver is the mason choosing which chipped stone goes in each slot while trying to keep neighboring stones compatible

What was cut away

The machine used to carry several ideas that did not belong:

• subsampling the mural before extraction with source_region_stride
• ranking region candidates by resemblance to a lattice portrait (sharp_rgba, edge_rgba)
• papering over missing foreground tiles by injecting sharp_pixel or edge_pixel
• pretending a learned tile graph existed while actually using sampled color deltas from one cell away in source space
• building and hauling a dedicated tile-graph source-reference object through the core loop

Those are gone.

That matters because they were smuggling continuous-solver assumptions back into a path that is supposed to be source-owned and discrete.

The living machine

Stage 0: Pipeline chooses a lattice

Function:

• run_pipeline(...) in src/repixelizer/pipeline.py

Inputs:

• source_rgba
• chosen InferenceResult
  • target_width
  • target_height
  • phase_x
  • phase_y

What it means:

The pipeline chooses the grid we are going to commit to. This is not the tile-graph solver yet. This is just choosing the size of the chessboard before we decide which pieces go on it.

Important truth:

• if the lattice is wrong, every later step is forced to cut the mural against the wrong ruler
• but once the lattice is pinned, the tile-graph path now runs directly on that lattice without phase-rerank probes or hybrid sidecars

Stage 1: Edge scout

Function:

• analyze_tile_graph_source(...) in src/repixelizer/analysis.py

Output:

• TileGraphSourceAnalysis.edge_map

What it means:

This stage is the scout walking over the mural with a lantern, marking where the sharp cliffs are. It is not deciding colors. It is only pointing out where the paint changes abruptly.

Important variable:

• edge_map[y, x]
  • a per-source-pixel edge strength in [0, 1]

Why it exists:

• region extraction needs to know where a component has its sharpest internal detail
• candidate budgeting uses edge strength to give difficult cells more room

Stage 2: Source pixels are projected onto the output lattice

Function:

• build_tile_graph_model(...) in src/repixelizer/tile_graph.py

Key variables:

• cell_w
• cell_h
• projected_coord_x
• projected_coord_y
• projected_flat_index

What it means:

Every source pixel is asked:

"If this final grid is the one we believe in, which output cell do you live under?"

This is like dropping a transparent graph-paper sheet over the mural and writing the output cell index on every source pixel.

Derived summaries:

• cell_counts_flat
  • how many source pixels fall into each output cell
• cell_mean_rgba_flat
  • the average source color already living under that cell
• cell_alpha_support_flat
  • the strongest alpha seen in that cell
• cell_alpha_mean_flat
  • the average alpha in that cell
• cell_edge_strength_flat
  • the strongest edge signal seen in that cell

These summaries are not a tile portrait from a separate source-reference system. They are direct summaries of the actual source pixels already assigned to that output cell under the chosen lattice.

Stage 3: Atomic opaque regions

Function:

• _segment_atomic_source_regions(...)
• CPU path: _segment_atomic_source_regions_cpu(...)

Inputs:

• full-resolution source_rgba
• full-resolution edge_map
• alpha/color join thresholds

What it means:

Now the mural is split into paint islands.

Pixels join the same island only if:

• they are opaque enough
• their premultiplied colors are similar enough
• their alpha is similar enough

This is not a "cluster by vibes" step. It is literal connected-component labeling over the full-resolution mural.

Important consequence:

• every opaque source pixel belongs to some component
• if an occupied output cell later ends up with no real extracted tile, that is not normal; it is a bug or a failure in cutting/projection logic

Stage 4: Cut one-cell tiles out of each component

Function:

• _extract_source_region_tiles(...)

Inputs:

• components
• flat_rgba
• flat_edge
• flat_x
• flat_y
• lattice geometry: cell_w, cell_h, phase_x, phase_y

What it means:

This is the quarry.

For each paint island, we try to chip out cell-sized stones.

There are two modes:

• ordinary components:
  • start from the centroid and edge peak
  • accept a one-cell window if it contains enough of the component
  • march outward one cell at a time
• elongated stroke-like components:
  • estimate the component’s principal axis
  • seed along that axis
  • cut tiles in bands so a long stroke is not immediately turned into a rectangular blob

Important variables inside each extracted tile:

• rep_linear
  • the literal source pixel index chosen to represent the tile
• rep_rgba
  • the tile color
• area_ratio
  • how much component area the tile owns relative to one output cell
• coverage
  • area_ratio clipped to [0, 1]
• edge_peak
  • strongest edge value inside that tile footprint
• source_center_x, source_center_y
  • where this tile lives in source space
• coord_x, coord_y, flat_index
  • which output cell this tile projects into
• component_id
  • which paint island it came from

Natural-language picture:

The component is a strip of stained glass. We are cutting out little panes, each about one output cell wide, and tagging where each pane came from.

Stage 5: Fill empty occupied cells honestly

Still inside:

• _extract_source_region_tiles(...)

What it means:

After the first pass of tile cutting, some output cells may still have no tile in their bucket even though component pixels really do overlap them.

To fix that, the builder does an overlap-based empty-cell fill:

• for each component, look at every output cell it physically overlaps
• if that bucket is still empty, add one overlap tile there

This matters because:

• a foreground cell is allowed to have multiple candidates
• a foreground cell is not allowed to have zero real extracted candidates

Background-only cells are different:

• they have no opaque source ownership
• they get a transparent candidate later in model building

Stage 6: Learn adjacency from extracted tiles

Still inside:

• _extract_source_region_tiles(...)

What it means:

Each extracted tile now looks one cell away in all four directions and asks:

"Among the tiles that actually landed in that neighboring output slot, which one looks like my real neighbor back in source space?"

For each direction, we search the neighboring bucket and choose the best observed neighbor by:

• source-center displacement closeness
• same-component tie preference
• area fit
• coverage
• edge strength

Stored per tile:

• neighbor_rgba[4, 4]
• neighbor_mask[4]

This is the first honest adjacency graph the tile-graph path has had.

It is not:

• sampled one-cell-away colors from the source image
• a lattice portrait
• a guessed delta field

It is:

• "this extracted stone most naturally sat beside that extracted stone"

Stage 7: Build per-cell candidate buckets

Function:

• build_tile_graph_model(...)
• helper: _select_source_region_candidates(...)

What it means:

Each output cell gets a bucket of legal stones.

Candidate selection now keeps only tile-owned signals:

• area fit
• coverage
• edge peak

It does not use:

• sharp_rgba similarity
• edge_rgba similarity
• injected sharp/edge fallback pixels

Important rule:

• if a cell has real source-owned region tiles, those are the truth
• if a cell has no opaque source support, it gets a transparent candidate
• if a cell has opaque source support but no extracted region candidate, the build fails loudly

That last rule is deliberate. The old machine would quietly make something up. The new one would rather stop the line than lie.

Additional background option:

• if cell_alpha_mean_flat < 0.98, a transparent candidate is also added

This lets the solver choose "leave this slot empty" in partially occupied or fringe cells instead of being forced to place an opaque stone just because a tiny sliver of source detail touched the slot.

Stage 8: The solver model

Dataclass:

• TileGraphModel

Current essential fields:

• candidate_rgba
• candidate_area_ratio
• candidate_coverage
• candidate_edge_peak
• candidate_neighbor_rgba
• candidate_neighbor_mask
• cell_candidate_offsets
• cell_candidate_indices
• cell_mean_rgba
• cell_alpha_mean
• cell_edge_strength

What it means:

This is the parts tray beside the machine.

• the first block describes the legal stones
• the second block says which stones belong to which output slot
• the last block is the local source evidence already living under each slot

Stage 9: Unary cost

Function:

• _tile_graph_unary_cost_torch(...)

Current unary terms:

• color_error
  • how far the candidate color is from the actual mean source color already under that output cell
• area_error
  • how far the candidate footprint is from one cell
• alpha_error
  • how far the candidate alpha is from the actual mean alpha under that cell
• coverage_error
  • how incomplete the candidate’s source ownership is
• edge_error
  • how far the candidate’s edge peak is from the strongest edge already under that cell

Natural-language picture:

This is the bouncer at each slot saying:

"Does this stone even look like it belongs in the patch of mural that sits under this square?"

Important difference from the old machine:

• this is local source support
• not a separate lattice portrait pretending to know the answer in advance

Stage 10: Pairwise adjacency cost

Functions:

• _pair_penalty_selected_torch(...)
• _pair_penalty_option_right_torch(...)
• _pair_penalty_option_left_torch(...)
• _pair_penalty_option_down_torch(...)
• _pair_penalty_option_up_torch(...)

What it means:

Now the mason cares about seams.

If tile A says it naturally expects tile B to the right, then choosing a very different right neighbor should cost something.

The current pairwise cost compares:

• the actually chosen neighbor color
• against the neighbor color learned from extracted tiles

This is now a real observed compatibility signal.

It is no longer:

• "the source image one cell away from this center happened to look like X, so maybe that is the right delta"

Stage 11: Checkerboard parity refinement

Function:

• optimize_tile_graph(...)

What it does:

1. build the model
2. compute unary cost
3. choose an initial argmin candidate per cell
4. alternate parity updates over the grid
5. keep the initial assignment if the refinement step makes source-lattice fidelity worse

Natural-language picture:

The mason lays down a first draft, then alternates black squares and white squares of the chessboard, trying to improve local seam compatibility without moving everything at once.

Important truth:

• this is still a local greedy/discrete smoother
• it is not a global optimal graph solver
• if the initial candidate buckets are wrong, parity updates will not save the image

Data dictionary

flat_rgba_np

• full-resolution source pixels flattened into [pixel, rgba]

projected_flat_index

• for each source pixel, which output cell it belongs to under the chosen lattice

components

• connected opaque paint islands in the source

region_buckets[flat_index]

• all extracted real tiles currently legal for one output cell

candidate_rgba

• literal colors the solver is allowed to place

candidate_neighbor_rgba

• per-candidate remembered neighbor colors from real extracted-tile adjacency

cell_mean_rgba

• actual average source color already living under each output cell

cell_alpha_mean

• actual average source alpha already living under each output cell

cell_edge_strength

• strongest edge signal already living under each output cell

What this machine is trying to maximize

In plain language:

• choose only real, source-owned stones
• prefer stones that genuinely fit the patch of mural beneath the slot
• prefer neighboring stones that were actually observed beside each other
• never silently hallucinate a foreground tile just because the lattice wanted one

What would make the machine lie now

If the output is still wrong after this pass, the likely failure points are no longer the old ones.

The remaining likely lies are:

• region cutting still slices a component into the wrong one-cell panes
• candidate truncation still throws away the only good tile for a difficult slot
• learned adjacency still chooses the wrong observed neighbor when multiple neighbors are plausible
• the parity solver is still too local and can settle into a bad arrangement even with legal tiles
• the lattice itself is wrong, which means the whole quarry is being cut against the wrong ruler

Current reality check

Pinned cleaned badge run:

• target: 126x126
• phase: (0.0, -0.2)
• mode: tile-graph
• phase rerank: off

Current result after the latest cut:

• initial source-fidelity: 0.1814
• final source-fidelity: 0.1814
• the solver kept the initial assignment because refinement did not improve it

That is still not good enough, but it is dramatically more honest than the earlier architectural-collapse run around 0.5055.

In other words:

• the machine is no longer babbling nonsense
• it is still not carving the right stones often enough

Big-picture judgment

Does the current machine make sense?

Yes, much more than before.

The data now flows in one believable direction:

1. observe the mural
2. find real paint islands
3. cut one-cell stones out of those islands
4. learn which stones really sat beside which
5. choose among those stones for each output slot

That is a coherent mental model.

What is still missing is not conceptual honesty.

What is still missing is quality in the stone cutting and compatibility choice.