Regenerate the shipped mod ship.cc files off the retired HullData slots; blind element ids only where they are irrelevant
AetherDb ship-authoring regenerate rewrites a package under the current schema and requires it to read back identical. ElementIdUpgradeTests gains content preservation and a shipped-packages-writable check; ShipModCatalogTests compares mod records unblinded where the same packages are composed twice. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
a517c97928
commit
b191e3b30f
6 changed files with 133 additions and 18 deletions
BIN
GameData/Mods/aetheria.nakovalnya/ship.cc
(Stored with Git LFS)
BIN
GameData/Mods/aetheria.nakovalnya/ship.cc
(Stored with Git LFS)
Binary file not shown.
BIN
GameData/Mods/aetheria.wren/ship.cc
(Stored with Git LFS)
BIN
GameData/Mods/aetheria.wren/ship.cc
(Stored with Git LFS)
Binary file not shown.
|
|
@ -1,6 +1,6 @@
|
|||
# Element ids and the CultLib pin (abd1a92e)
|
||||
# Element ids and the CultLib pin (cultlib-unity-v3.0.0)
|
||||
|
||||
CultLib 3.0 (unreleased; headless pin `abd1a92e`, `Directory.Build.props`) gives every element of an object list an identity. Every
|
||||
CultLib 3.0.0 (Unity tag `cultlib-unity-v3.0.0`, commit `7225835d`, which is the headless pin in `Directory.Build.props`; CultMath `cultmath-unity-v0.5.1`, `f1edfa45`) gives every element of an object list an identity. Every
|
||||
element type reachable through a list of a `[CultDocument]` type carries one string member with `[Key(n)]` and `[CultElementId]`;
|
||||
the registry refuses to open otherwise, naming the type and list.
|
||||
|
||||
|
|
@ -20,15 +20,22 @@ the registry refuses to open otherwise, naming the type and list.
|
|||
|
||||
## The bump is one-way
|
||||
|
||||
A store holding element ids is format v3 (single file) or v5 (directory). The previous pin, `b8fc5eb0`, refuses a v3 store loudly
|
||||
A store holding element ids is format v3 (single file) or v5 (directory). The previous pin (`cultlib-unity-v2.0.1`, `b8fc5eb0`) refuses a v3 store loudly
|
||||
(`NotSupportedException: Store format cultcache.store.v3 is not readable`; checked by Soul on PR 12). CultLib #98 (`7002c252`) records
|
||||
that some intermediate pins read a single-file v3 store silently and drop the ids on the next write. So: upgrade readers before
|
||||
writers, keep a pre-upgrade copy of any real store (`GameData/Aetheria.cc` and the mod packages are tracked in git; player saves and
|
||||
run stores are not), and do not roll the pin back after a store has been written under the new one.
|
||||
|
||||
Shipped mod packages (`GameData/Mods/*/ship.cc`) hold values in retired `HullData` slots 21 and 22. They read, but the new CultLib
|
||||
refuses to write over them (`CultSchemaConflictException`), so an in-place rewrite of those two files fails until their schema ids are
|
||||
declared compatible or the files are regenerated. A rewritten catalog is about 15% larger: every behavior row is 38 slots, mostly nil.
|
||||
The shipped mod packages (`GameData/Mods/aetheria.wren/ship.cc`, `aetheria.nakovalnya/ship.cc`) once held values in the retired
|
||||
`HullData` slots 21 and 22 (Wwise sound bank and audio stats), which the new CultLib refuses to write over
|
||||
(`CultSchemaConflictException`). They were regenerated with `AetherDb ship-authoring regenerate <ship.cc> apply`, which loads the
|
||||
records, writes them into a fresh file under the current schema, requires them to read back identical and replaces the file. Both
|
||||
now hold persisted element ids and nil in slots 21 and 22; `EveryShippedModPackageCanBeWrittenOver` fails if a package regresses.
|
||||
A mod file that fails the same way is regenerated the same way. A rewritten catalog is about 15% larger: every behavior row is 38
|
||||
slots, mostly nil.
|
||||
|
||||
The Unity manifest (`Packages/manifest.json`) still pins `caching-unity-v2.0.0` / `cultlib-unity-v2.0.1`; those move when CultLib
|
||||
tags a Unity release of 3.0. Until then the Unity editor cannot compile `[CultElementId]` (the plugin DLLs at those tags lack the attribute): this branch is headless-only and must not merge to master before a Unity-consumable CultLib exists.
|
||||
The Unity manifest (`Packages/manifest.json`) pins `org.gamecult.cultlib` at `cultlib-unity-v3.0.0` and `org.gamecult.cultmath` at
|
||||
`cultmath-unity-v0.5.1` (the version v3.0.0 depends on). `org.gamecult.caching.unity` stays at `caching-unity-v2.0.0`: it is editor
|
||||
tooling, unchanged in v3.0.0. The v3.0.0 plugin `GameCult.Caching.dll` exports `CultElementIdAttribute` (sealed, fields and
|
||||
properties, one optional `derivedFrom` argument), the shape `ServerShared` uses. The Unity editor compile of this branch has not
|
||||
been run; the headless build and tests are the evidence so far.
|
||||
|
|
|
|||
|
|
@ -1,10 +1,12 @@
|
|||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Linq;
|
||||
using System.Threading.Tasks;
|
||||
using GameCult.Caching;
|
||||
using Xunit;
|
||||
|
||||
// The CultLib pin that requires element ids (abd1a92e) reads the stores written before it (b8fc5eb0 and older), which carry none.
|
||||
// CultLib 3.0 requires element ids and reads the stores written before it (2.0 and older), which carry none.
|
||||
// The shipped GameData stores are those stores, byte for byte: they have never been rewritten under the new pin.
|
||||
// Upgrading is one-way: once a store is written under the new pin it holds element ids, and an older pin can read it only by
|
||||
// dropping them (CultLib #98). Readers upgrade before writers.
|
||||
|
|
@ -41,6 +43,31 @@ public class ElementIdUpgradeTests
|
|||
}
|
||||
}
|
||||
|
||||
// Upgrading assigns ids and changes nothing else: every record of the old-format file decodes, once the id members are
|
||||
// blank, to the same bytes before and after the mint, flush and reopen.
|
||||
[Fact]
|
||||
public async Task UpgradingKeepsEveryFieldOfEveryRecordAndOnlyAddsIds()
|
||||
{
|
||||
using var directory = new TempDirectory();
|
||||
var copy = Path.Combine(directory.Path, "Aetheria.cc");
|
||||
File.Copy(Shipped, copy);
|
||||
var registry = TestCatalog.Registry();
|
||||
Dictionary<string, string> Content(string path) =>
|
||||
GameCult.Caching.MessagePack.CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path)).Records
|
||||
.ToDictionary(record => record.Key, record => record.SchemaId + ":" + Convert.ToHexString(ElementIdBlind.Payload(registry, record.SchemaId, record.Payload)));
|
||||
var before = Content(copy);
|
||||
|
||||
using (var cache = AetheriaStores.Open(copy, catalogWritable: true, registry: TestCatalog.Registry()))
|
||||
{
|
||||
Assert.True(cache.MintElementIds() > 0);
|
||||
await cache.FlushAsync();
|
||||
}
|
||||
|
||||
var after = Content(copy);
|
||||
Assert.Equal(before.Count, after.Count);
|
||||
foreach (var record in before) Assert.True(after[record.Key] == record.Value, $"{record.Key} changed content when its element ids were minted");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void EveryShippedModPackageStillLoads()
|
||||
{
|
||||
|
|
@ -53,4 +80,45 @@ public class ElementIdUpgradeTests
|
|||
Assert.NotNull(visual);
|
||||
}
|
||||
}
|
||||
|
||||
// A package whose file still holds a retired slot reads but cannot be written over (CultSchemaConflictException).
|
||||
[Fact]
|
||||
public void EveryShippedModPackageCanBeWrittenOver()
|
||||
{
|
||||
foreach (var package in Directory.GetDirectories(Path.Combine(RunStartTests.FindRepoRoot(), "GameData", "Mods")))
|
||||
{
|
||||
using var directory = new TempDirectory();
|
||||
var copy = Path.Combine(directory.Path, "ship.cc");
|
||||
File.Copy(Path.Combine(package, "ship.cc"), copy);
|
||||
var (hull, visual, products) = ShipAuthoringStore.Load(copy);
|
||||
using (var cache = ShipAuthoringStore.Open(copy, writable: true))
|
||||
{
|
||||
ShipAuthoringStore.Write(cache, hull, visual);
|
||||
foreach (var product in products) ShipAuthoringStore.WriteProduct(cache, visual.Id, product);
|
||||
cache.FlushAsync().GetAwaiter().GetResult();
|
||||
}
|
||||
Assert.Equal(visual.Id, ShipAuthoringStore.Load(copy).Visual.Id);
|
||||
}
|
||||
}
|
||||
|
||||
// The tool that regenerated the shipped packages: a dry run touches nothing, apply keeps every record's content.
|
||||
[Fact]
|
||||
public void RegenerateDryRunLeavesTheFileAndApplyKeepsTheContent()
|
||||
{
|
||||
using var directory = new TempDirectory();
|
||||
var copy = Path.Combine(directory.Path, "ship.cc");
|
||||
File.Copy(Path.Combine(RunStartTests.FindRepoRoot(), "GameData", "Mods", "aetheria.wren", "ship.cc"), copy);
|
||||
var bytes = File.ReadAllBytes(copy);
|
||||
var (hull, visual, products) = ShipAuthoringStore.Load(copy);
|
||||
|
||||
Assert.Equal(0, ShipAuthoringCommands.Run(new[] { "regenerate", copy }));
|
||||
Assert.Equal(bytes, File.ReadAllBytes(copy));
|
||||
Assert.Equal(0, ShipAuthoringCommands.Run(new[] { "regenerate", copy, "apply" }));
|
||||
|
||||
var (hull2, visual2, products2) = ShipAuthoringStore.Load(copy);
|
||||
Assert.Equal(ElementIdBlind.Pack(hull), ElementIdBlind.Pack(hull2));
|
||||
Assert.Equal(ElementIdBlind.Pack(visual), ElementIdBlind.Pack(visual2));
|
||||
Assert.Equal(products.Length, products2.Length);
|
||||
Assert.Equal(new[] { copy }, Directory.GetFiles(directory.Path).Where(file => !file.EndsWith(".lock")).ToArray());
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -624,7 +624,8 @@ public sealed class ShipModCatalogTests : IDisposable
|
|||
ShipModCatalog.Compose(Shipped, second, secondMods);
|
||||
ShipModCatalog.Compose(Shipped, Path.Combine(_directory.Path, "Derived3", "Aetheria.modded.cc"), firstMods);
|
||||
|
||||
Assert.Equal(Records(Derived), Records(second));
|
||||
// Two packages written separately carry separately minted ids: same content, ids blank.
|
||||
Assert.Equal(Records(Derived, blindMods: true), Records(second, blindMods: true));
|
||||
Assert.Equal(Records(Derived), Records(Path.Combine(_directory.Path, "Derived3", "Aetheria.modded.cc")));
|
||||
Assert.Contains(ShipModCatalog.HullKey("mod.c").Value, Records(Derived).Keys);
|
||||
}
|
||||
|
|
@ -793,10 +794,15 @@ public sealed class ShipModCatalogTests : IDisposable
|
|||
ShipModCatalog.PackageOf(cache, hull, Mods)).Message);
|
||||
}
|
||||
|
||||
// Every stored record's key, schema and payload bytes. StoredAt is a write timestamp, not record content.
|
||||
private static Dictionary<string, string> Records(string path) =>
|
||||
// Every stored record's key, schema and payload bytes. StoredAt is a write timestamp, not record content. The shipped
|
||||
// catalog holds no element ids, so each open mints new random ones for the records Compose copies from it; those are
|
||||
// always compared with ids blank. A mod package's records carry the ids its ship.cc persisted and are compared as stored,
|
||||
// unless the two sides are packages written separately (blindMods), where each write minted its own.
|
||||
private static Dictionary<string, string> Records(string path, bool blindMods = false) =>
|
||||
GameCult.Caching.MessagePack.CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path)).Records
|
||||
.ToDictionary(record => record.Key, record => record.SchemaId + ":" + Convert.ToHexString(ElementIdBlind.Payload(TestCatalog.Registry(), record.SchemaId, record.Payload)));
|
||||
.ToDictionary(record => record.Key, record => record.SchemaId + ":" + Convert.ToHexString(
|
||||
!blindMods && ShipModCatalog.IsModKey(new CultRecordKey(record.Key)) ? record.Payload
|
||||
: ElementIdBlind.Payload(TestCatalog.Registry(), record.SchemaId, record.Payload)));
|
||||
|
||||
private void AssertOnlyTheDerivedFile()
|
||||
{
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
using GameCult.Caching;
|
||||
using GameCult.Caching.MessagePack;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Globalization;
|
||||
|
|
@ -8,7 +9,7 @@ using System.Linq;
|
|||
// Uses the existing database executable; mod authoring does not earn another binary target.
|
||||
public static class ShipAuthoringCommands
|
||||
{
|
||||
private const string Usage = "ship-authoring create <ship.cc> <stable-id> <name> [--like <shipped hull name>] | starter <mods-dir> <stable-id> <name> --like <shipped hull name> | inspect <ship.cc> | centre <ship.cc> | validate <ship.cc> | product <ship.cc> <maker> (--name <name> [--description <text>] [--not-offered] | --remove) | price <ship.cc> <price> | stats <ship.cc> [--mass m] [--durability d] [--armor a] [--drag x] [--top-speed v] | compose <shipped.cc> <derived.cc> <mods-dir>";
|
||||
private const string Usage = "ship-authoring create <ship.cc> <stable-id> <name> [--like <shipped hull name>] | starter <mods-dir> <stable-id> <name> --like <shipped hull name> | regenerate <ship.cc> [apply] | inspect <ship.cc> | centre <ship.cc> | validate <ship.cc> | product <ship.cc> <maker> (--name <name> [--description <text>] [--not-offered] | --remove) | price <ship.cc> <price> | stats <ship.cc> [--mass m] [--durability d] [--armor a] [--drag x] [--top-speed v] | compose <shipped.cc> <derived.cc> <mods-dir>";
|
||||
|
||||
private static readonly string[] StatFlags = { "--mass", "--durability", "--armor", "--drag", "--top-speed" };
|
||||
|
||||
|
|
@ -84,6 +85,39 @@ public static class ShipAuthoringCommands
|
|||
Console.WriteLine($"Wrote a flyable starter package {path}; replace its stand-in model through Blender and Package, then set price and product.");
|
||||
return 0;
|
||||
}
|
||||
if (args[0] == "regenerate" && (args.Length == 2 || args.Length == 3 && args[2] == "apply"))
|
||||
{
|
||||
// A package written before a record dropped a persisted slot cannot be written over (CultSchemaConflictException,
|
||||
// retired slots stay read-only). Its decoded records go into a fresh file under today's schema, which must read
|
||||
// back identical, and the fresh file replaces the old only with `apply`. Retired slots are the only content lost.
|
||||
var (hull, ship, products) = ShipAuthoringStore.Load(args[1]);
|
||||
var fresh = Path.GetFullPath(args[1]) + ".regenerating";
|
||||
File.Delete(fresh);
|
||||
try
|
||||
{
|
||||
using (var cache = ShipAuthoringStore.Open(fresh, writable: true))
|
||||
{
|
||||
ShipAuthoringStore.Write(cache, hull, ship);
|
||||
foreach (var product in products) ShipAuthoringStore.WriteProduct(cache, ship.Id, product);
|
||||
cache.FlushAsync().Wait();
|
||||
}
|
||||
var (hull2, ship2, products2) = ShipAuthoringStore.Load(fresh);
|
||||
ShipAuthoringStore.ValidateLoaded(hull2, ship2, products2);
|
||||
var options = CultDocumentMessagePackSerialization.OptionsFor(typeof(HullData).Assembly);
|
||||
byte[] Pack<T>(T value) => MessagePack.MessagePackSerializer.Serialize(value, options);
|
||||
var same = Pack(hull).AsSpan().SequenceEqual(Pack(hull2)) && Pack(ship).AsSpan().SequenceEqual(Pack(ship2)) &&
|
||||
products.Length == products2.Length &&
|
||||
products.OrderBy(p => p.Manufacturer.Key.Value, StringComparer.Ordinal).Zip(
|
||||
products2.OrderBy(p => p.Manufacturer.Key.Value, StringComparer.Ordinal),
|
||||
(a, b) => Pack(a).AsSpan().SequenceEqual(Pack(b))).All(equal => equal);
|
||||
if (!same) throw new InvalidOperationException($"{ship.Id}: the regenerated file does not read back as the records that went in; {args[1]} is untouched.");
|
||||
Console.WriteLine($"{ship.Id}: {new FileInfo(args[1]).Length} bytes -> {new FileInfo(fresh).Length} bytes, {hull.Hardpoints.Count} hardpoints, {ship.Anchors.Count} anchors, {products.Length} products read back identical.");
|
||||
if (args.Length == 3) File.Move(fresh, args[1], overwrite: true);
|
||||
else Console.WriteLine("Dry run; add apply to replace the file.");
|
||||
}
|
||||
finally { File.Delete(fresh); File.Delete(fresh + ".lock"); }
|
||||
return 0;
|
||||
}
|
||||
if (args[0] == "price" && args.Length == 3)
|
||||
{
|
||||
// Judged before the file is touched: a refusal leaves it byte-identical.
|
||||
|
|
|
|||
Loading…
Reference in a new issue