From 9634ab5b190815edb2e613c99934e115bca5fee7 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 14 Oct 2026 21:45:19 +0200 Subject: [PATCH] Research: header flags one writer sets for content another cannot see (ext4, SQLite, git extensions, protobuf, Avro/Parquet), for variants element ids Co-Authored-By: Claude Opus 5.5 --- docs/research/element-id-header-flags.md | 49 ++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 docs/research/element-id-header-flags.md diff --git a/docs/research/element-id-header-flags.md b/docs/research/element-id-header-flags.md new file mode 100644 index 00000000..30016c3d --- /dev/null +++ b/docs/research/element-id-header-flags.md @@ -0,0 +1,49 @@ +# Prior art: a header flag one writer sets and content another writer cannot see + +Serves: `variants:question:non-csharp-writers-mark-ids` (raised in +`variants:follow_up:foreign-ids-under-v1-header`, campaign `variants`). Gathered by +Imagination, session self-2026-10-14b-eureka, 2026-10-14. Facts with pointers; the +recommendation lives in the question, not here. + +The CultLib case: the store header (`cultcache.store.v1` / `v3`) says whether any record +holds an element id, so a reader older than ids refuses the store instead of rewriting +elements without them. An element id is an ordinary keyed string in the payload, so a +runtime that does not know which member is the id (cultcache-py, -ts, -rs today) cannot +tell from content whether to mark. The persistence contract +(`src/GameCult.Caching/Contracts/cultcache-persistence-format.md:424-427` at 6afa8ed9) +says those runtimes keep a `v3` header they found and do not decide. + +## How established formats split the job + +| System | Who sets the flag | What an implementation that does not understand the feature does | Pointer | +|---|---|---|---| +| ext4 superblock | The code that first uses a feature sets its bit in `s_feature_compat`, `_ro_compat` or `_incompat`. | Unknown `incompat` bit: refuse to mount (fsck stops too). Unknown `ro_compat` bit: mount read-only, never write. Unknown `compat` bit: read and write freely; fsck should not modify. | https://www.kernel.org/doc/html/latest/filesystems/ext4/super.html (feature fields) | +| SQLite file header | The writer's journal mode sets the write/read version bytes (offsets 18, 19); the schema format number (offset 44) is set by the writer of the newer schema features. | Read version 1-2 with write version > 2: open read-only. Read version > 2: neither read nor write. | https://www.sqlite.org/fileformat.html section 1.3.3, 1.3.10 | +| git repository format | A repository declares `core.repositoryformatversion = 1` and the `extensions.*` it uses; the tool enabling a feature writes its extension key. | Any unimplemented `extensions.*` key in a v1 repo: the operation must not proceed (read or write). | https://git-scm.com/docs/repository-version | +| Protocol Buffers | No flag. New fields are self-delimited by tag. | Parsers keep unknown fields and write them back (proto2 always; proto3 again since 3.5), so an old reader does not drop a newer writer's data on rewrite. | https://protobuf.dev/programming-guides/proto3/#unknowns | +| Avro container files | The writer embeds its full schema in the file header. | Reader resolves writer schema against its own; a writer field the reader lacks is ignored, so it is lost if the reader rewrites. | https://avro.apache.org/docs/current/specification/ ("Schema Resolution") | +| Parquet | Writer stamps `created_by` (application and version) in FileMetaData. | Readers keep a list of known-bad writer versions and distrust their metadata (parquet-mr ignores min/max statistics from writers before 1.8.0, PARQUET-251). | https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift (`created_by`); https://issues.apache.org/jira/browse/PARQUET-251 | + +## Patterns, as facts + +- Flag-based formats (ext4, SQLite, git) put the duty to set the flag on the writer that + introduces the content. None of them has a writer that carries feature content while + not knowing it is that feature: an implementation that does not understand a feature + either does not write (ro_compat, SQLite read-only) or does not touch the store (git). +- Formats without a flag (protobuf, Avro) handle the older reader by preservation + (protobuf keeps unknown fields) or accept loss (Avro drops unknown writer fields on + rewrite). CultCache's C# 1.x readers skip unknown keyed slots and do not preserve + them, which is why the `v3` header exists. +- Parquet's `created_by` is a third option: readers judge content by who wrote it and + compensate for writers known to get a flag or statistic wrong. +- No system surveyed asks a reader to infer a feature flag from content it cannot + recognise; where content cannot be recognised, the burden is on writers. + +## Mechanism facts from this pass (probes at 6afa8ed9 on Yggdrasil) + +- A C# cache that pulls the Python-written `py-store/leaf-record.cc` (v1, element ids + `000000000021/22`) holds that record with `HoldsIds = true`, `IdsInMemoryOnly = false`; + a C# write that adds an unrelated record copies it forward and leaves the header `v1`. + The ids survive in C#; a pre-id reader with a type lacking the id slot would drop them. +- cultcache-py, -ts and -rs read `v3` and keep it on rewrite; none has a declaration of + which member is an element id.