From 96cb6d20f1d40b0f53c3d419e6cd2dbf3ed3810b Mon Sep 17 00:00:00 2001 From: soul Date: Mon, 19 Oct 2026 13:01:22 +0200 Subject: [PATCH] Research: prior art for agent-authored stand-in hulls (render-critique loops, part libraries, self-judging limits) Co-Authored-By: Claude Opus 5.5 --- ...-agent-authored-standin-hulls-prior-art.md | 101 ++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 docs/research/2026-10-19-agent-authored-standin-hulls-prior-art.md diff --git a/docs/research/2026-10-19-agent-authored-standin-hulls-prior-art.md b/docs/research/2026-10-19-agent-authored-standin-hulls-prior-art.md new file mode 100644 index 000000000..9e6c6eb5d --- /dev/null +++ b/docs/research/2026-10-19-agent-authored-standin-hulls-prior-art.md @@ -0,0 +1,101 @@ +# Prior art for agent-authored stand-in hulls (aetheria-release, 2026-10-19) + +Serves: the stand-in hull prompt bench for the demo cast and freelancer factions +(operator 2026-10-19: "Let's see if we can figure out a prompt that gets Opus to +spit out some decent stand-in hulls for the demo cast (including ships from +non-present factions to be piloted by other freelancers)"). Gathered by Eyes +pass `eyes-ag7-llm-cad-prior-art`, session `self-2026-10-19-ag7`. Companion to +`2026-10-18-agent-built-hull-art-prior-art.md` and +`2026-10-18-faction-form-language-prior-art.md`. + +Evidence levels: **read** (HTML or abstract page read), **snippet** (search +excerpt only). + +## Loops and their measured gains + +- **SceneCraft** (ICML 2024, arXiv 2403.01248, read). Removing the render-review + inner loop dropped constraint score 88.9 to 26.1, the largest loss in its + ablation. Removing the learned code-skill library dropped it 88.9 to 64.5 and + CLIP 69.8 to 48.3. Human eval put its aesthetics win rate at 74.5% against + BlenderGPT's 14.5%. It measures scene layout from given assets, not part form. +- **CADCodeVerify** (ICLR 2025, arXiv 2410.05340, read). Two refinement rounds; + no gain observed beyond round 2. Point-cloud distance 0.156 to 0.126. Dropping + the reference views or the few-shot questions made it worse. The VLM's own + verification answers were correct only 64.6% and 68.2% of the time; human + gold feedback (0.120) beat VLM feedback (0.137) on a 50-sample subset. +- **BlenderGym** (CVPR 2025, arXiv 2504.01786, read). VLM verifier agreement + with human pairwise judgments 0.66 (Claude 3.5 Sonnet) against 0.79 + human-to-human; Claude was the only verifier that consistently gave full + reasoning. Scarce compute favours generation, abundant compute favours + verification: a scaled small verifier over 32 Claude candidates beat unscaled + GPT-4o and Claude. Generators hallucinate differences, and edits mismatch the + diagnosis in scripts over about 80 lines. Metrics miss physical + implausibility such as floating objects. +- **LL3M** (arXiv 2508.08228, read). Planner, retrieval, coder, critic and + verifier agents; the critic sees 5 renders, the verifier checks each fix was + applied, refinement loops until the verifier approves. Blender-API retrieval + raised complex-operation use 1.20 to 5.86 and cut execution errors 26%. + Without retrieval the coder sometimes deleted offending lines to silence + errors. Without shared prior code the coder regenerates a substantially + different asset. Disconnected handles survive auto-refinement; VLMs struggle + with spatial artifacts. Agent ablations are qualitative only; no user study. +- **Procedura** (arXiv 2608.26238, read). Named parts with typed mates, per-part + compile, mate and connectivity gates, and a vision critic that fixes one + diagnosed issue at a time. Ablation on MechBench-36 (judge score): full 0.828, + no refine 0.800, no planning 0.791, no 3D feedback 0.819; dropping planning + caused the biggest aesthetics drop, 0.827 to 0.775. A floater at 70% span was + 1.9% of volume, so a volume check misses it. Case study: 24 to 3 to 2 HIGH + issues over three critic cycles. CSG suits hard-surface better than organic. +- **P3D-Bench** (arXiv 2606.11152, abstract). Multi-part code generation is much + harder than single-part: models get identity right, dimensions, part count + and part geometry wrong. +- **ModelRift Pantheon benchmark** (vendor blog, one run per cell, read). Opus + 4.7 scored 3.0/5 with "uniform colour and weight"; Sonnet 4.6 3.4/5 with the + cleanest massing. Explicit measured dimensions helped; human annotations drawn + on the render beat text corrections. A preview PNG looked better than the + exported STL, which had non-manifold surfaces. + +## Self-judgement + +- Huang et al., ICLR 2024 (arXiv 2310.01798, snippet): intrinsic + self-correction without external feedback often fails and can hurt; reported + gains partly came from oracle labels or weak first prompts. +- Panickssery et al., NeurIPS 2024 (arXiv 2404.13076, snippet): LLM evaluators + recognise and favour their own generations; text domain. +- Multimodal self-judging leans on text priors over the image (arXiv 2406.11280, + 2508.20655, snippets). + +## Failure modes in CAD code + +- BenchCAD and ASSEMCAD (snippets): successful renders still show floating + components; union without face glue gives floating subpieces; polar patterns + are an error cluster. ASSEMCAD mitigates with safe-union and overlap + validators. +- Shape-grammar ships (CESCG 2016; 2022 Hacettepe thesis; snippets): primitive + sets plus rules give similar outlines with varied detail; unconstrained + derivation tends toward unstructured results. + +## Game-art practice (criteria a critic can check) + +- Stellaris Dev Diary 230 (search excerpt): silhouette and proportions read + first at distance; big shapes settled before small; shared motifs unify a + family. +- Fractured Space art direction, Creative Bloq (search excerpt): an arrowhead + silhouette reads as a threat across the map; three manufacturers each get a + distinct vocabulary (stacked cylinders, brutalist cubes, naval modular + shells); human figures dropped in as scale checks. +- "Moth" analysis, John Liberto, ArtStation (search excerpt): 3 large, 4 mid + and many small volumes; the large volumes dictate the silhouette. +- Primary/secondary/tertiary shapes (steveyang.readthedocs.io, read, after Neil + Blevins): primary defines silhouette, secondary individuality, tertiary story. + +## Gaps + +No controlled study of design-language or style-spec conditioning for +LLM-authored CAD; no measurement of a parts library against floating-part or +symmetry errors (nearest: SceneCraft's skill library, LL3M's retrieval); no +sweep of render-critique round counts for aesthetic quality; no direct +measurement of a model praising its own render (inferred from text-domain +work); no source on boxy "lego" look or scale confusion in LLM 3D code; no +authoritative game-art source for 60/30/10 ship proportions. 3D-PreMise +(2401.06437) and MeshCoder (2508.14879) seen as abstracts only.