Encoder B loads from snapshot directories, as the reference vector did
Measured on Yggdrasil: loading by repo id leaves the model's _name_or_path equal to the Llama repo id, so prepare_for_tokenization wraps the text in the chat header and the vector matches assembly A (cosine 1.0, 0.909 to the reference B). Loading from the snapshot paths reproduces the reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
6535cda025
commit
ffa2c7bf58
3 changed files with 31 additions and 12 deletions
|
|
@ -50,6 +50,11 @@ def read_snapshot_revision(hf_home: Path, repo_id: str) -> str:
|
|||
return ref.read_text().strip()
|
||||
|
||||
|
||||
def snapshot_path(hf_home: Path, repo_id: str) -> Path:
|
||||
"""The directory of the snapshot read_snapshot_revision names: assembly B loads from these, never from repo ids."""
|
||||
return Path(hf_home) / "hub" / ("models--" + repo_id.replace("/", "--")) / "snapshots" / read_snapshot_revision(hf_home, repo_id)
|
||||
|
||||
|
||||
def encoder_identity(*, assembly: str, device: str, revisions: dict, dtype: str, torch_version: str, transformers_version: str,
|
||||
peft_version: str, accelerate_version: str, kimodo_commit: str, loader_sha256: str) -> str:
|
||||
"""The canonical string a vector is keyed under: every input that can change it, in a fixed field order.
|
||||
|
|
|
|||
Loading…
Reference in a new issue