BP-3 3: bound every echoed field, and defer under the answer's own status
A reply echoes the request's message_id (and on failures its service_id, operation and payload_schema; on chunk answers its chunk_hash). answer now refuses any of them over MAX_ECHOED_FIELD_BYTES (256) at the door with a transport Error that carries none, so every reply, including the largest Deferred and ResponseTooLarge, fits the window. A deferred answer's envelope carries the status of the answer it stands for. The retention docs name the missing per-peer budget and that expiry runs only when a frame arrives. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
15fc24d6c6
commit
18de1bc6d7
3 changed files with 165 additions and 13 deletions
|
|
@ -11,7 +11,15 @@
|
|||
//! again and serving any of its chunks both touch it. When the stored bytes
|
||||
//! exceed the budget the body touched longest ago is evicted first. A chunk
|
||||
//! that two bodies share is stored once and lives until the last of them goes.
|
||||
//! Time is an argument, so nothing here reads a clock.
|
||||
//! Time is an argument, so nothing here reads a clock; expiry therefore runs
|
||||
//! only when a frame arrives, and an idle daemon holds an expired body until
|
||||
//! the next one.
|
||||
//!
|
||||
//! Limit: there is no per-session or per-peer budget, so any client can evict
|
||||
//! another client's body mid-fetch: four 64 MiB reads flush the 256 MiB store,
|
||||
//! and the loser's next chunk is answered `found: false`. That is acceptable
|
||||
//! for the organ's trusted WireGuard clients, where an instance identity is
|
||||
//! declared attribution and not authentication.
|
||||
|
||||
use std::cell::Cell;
|
||||
use std::collections::{BTreeSet, HashMap};
|
||||
|
|
|
|||
|
|
@ -183,6 +183,40 @@ pub fn schema_registry() -> Result<CultNetSchemaRegistry> {
|
|||
Ok(registry)
|
||||
}
|
||||
|
||||
/// The longest client-supplied string a reply may echo, in bytes. A reply
|
||||
/// echoes the request's `message_id` (and, on a failure, its `service_id`,
|
||||
/// `operation` and `payload_schema`; on a chunk answer, its `chunk_hash`), so an
|
||||
/// unbounded field would make the reply unbounded, and the hub would refuse a
|
||||
/// send the client then waits on forever. Every echoed field is refused past
|
||||
/// this at the door, with a transport `Error` that carries none of them. With
|
||||
/// it, every reply is within the window: the largest, a deferred answer or a
|
||||
/// refusal, is one 256-byte id, a real operation name, the daemon's own runtime
|
||||
/// id and, for a deferred answer, a manifest of at most
|
||||
/// `MAX_DEFERRED_BODY_BYTES / DEFERRED_CHUNK_BYTES` chunk references. Real ids
|
||||
/// are UUID-sized.
|
||||
pub const MAX_ECHOED_FIELD_BYTES: usize = 256;
|
||||
|
||||
/// The first field of a request that a reply would echo and that is over
|
||||
/// `MAX_ECHOED_FIELD_BYTES`.
|
||||
fn overlong_echo(message: &CultNetMessage) -> Option<&'static str> {
|
||||
let echoed: Vec<(&'static str, &str)> = match message {
|
||||
CultNetMessage::OperationRequest { message_id, service_id, operation, payload_schema, .. } => {
|
||||
vec![
|
||||
("message_id", message_id),
|
||||
("service_id", service_id),
|
||||
("operation", operation),
|
||||
("payload_schema", payload_schema),
|
||||
]
|
||||
}
|
||||
CultNetMessage::SchemaCatalogRequest { message_id, .. } => vec![("message_id", message_id)],
|
||||
CultNetMessage::ContentChunkRequest { message_id, chunk_hash, record_key, .. } => {
|
||||
vec![("message_id", message_id), ("chunk_hash", chunk_hash), ("record_key", record_key)]
|
||||
}
|
||||
_ => return None,
|
||||
};
|
||||
echoed.into_iter().find(|(_, value)| value.len() > MAX_ECHOED_FIELD_BYTES).map(|(name, _)| name)
|
||||
}
|
||||
|
||||
/// One message in, one message out. Nothing here reaches a mind except through
|
||||
/// `Daemon::handle`, and an envelope that does not decode never gets that far.
|
||||
/// A chunk request is answered from `bodies` alone, on the session it came in
|
||||
|
|
@ -196,6 +230,13 @@ pub fn answer<S: MindStore, I: IndexSink<S>>(
|
|||
) -> CultNetMessage {
|
||||
let runtime_id = daemon.runtime_id();
|
||||
bodies.expire(now);
|
||||
if let Some(field) = overlong_echo(&message) {
|
||||
return CultNetMessage::Error {
|
||||
error: format!("{MIND_SERVICE_ID} refuses a {field} over {MAX_ECHOED_FIELD_BYTES} bytes"),
|
||||
code: None,
|
||||
details: None,
|
||||
};
|
||||
}
|
||||
match &message {
|
||||
CultNetMessage::OperationRequest { message_id, operation, .. } => match decode_request(&message) {
|
||||
Ok((message_id, request)) => {
|
||||
|
|
@ -256,8 +297,12 @@ fn encode_or_fail(
|
|||
/// A reply that fits is returned as it is. A larger one whose payload, the
|
||||
/// named MessagePack of the response, is at most `MAX_DEFERRED_BODY_BYTES` is
|
||||
/// packed by CultNet's content plane, retained in `bodies`, and answered with
|
||||
/// `Deferred` and the manifest. That answer is a small fixed-shape envelope and
|
||||
/// is never itself measured or deferred. A larger payload is refused by name
|
||||
/// `Deferred` and the manifest. That answer is bounded, not fixed-shape: the id
|
||||
/// it echoes is at most `MAX_ECHOED_FIELD_BYTES` and its manifest at most 256
|
||||
/// references, so it always fits one send and is never itself measured or
|
||||
/// deferred. Its envelope status is the deferred answer's own, read from the
|
||||
/// reply before deferral, so routing on status is the same whole or deferred. A
|
||||
/// larger payload is refused by name
|
||||
/// with its own size and the deferral bound: nothing is truncated or paginated,
|
||||
/// and the caller narrows its own request. Both answers ride the response
|
||||
/// schema like every other, so a client parses no second vocabulary.
|
||||
|
|
@ -284,8 +329,8 @@ fn deliver(
|
|||
if encoded <= MAX_RESPONSE_BYTES {
|
||||
return reply;
|
||||
}
|
||||
let payload = match &reply {
|
||||
CultNetMessage::OperationResponse { payload, .. } => STANDARD.decode(payload),
|
||||
let (payload, answer_status) = match &reply {
|
||||
CultNetMessage::OperationResponse { payload, status, .. } => (STANDARD.decode(payload), status.clone()),
|
||||
_ => return not_encodable("only an operation response is deferred".into()),
|
||||
};
|
||||
let payload = match payload {
|
||||
|
|
@ -311,7 +356,12 @@ fn deliver(
|
|||
)
|
||||
.expect("the deferral constants are a valid pack");
|
||||
bodies.retain(&manifest, chunks, now);
|
||||
encode_or_fail(message_id, operation, &HuginnMindResponse::Deferred(DeferredAnswer { manifest }), runtime_id)
|
||||
let mut deferred =
|
||||
encode_or_fail(message_id, operation, &HuginnMindResponse::Deferred(DeferredAnswer { manifest }), runtime_id);
|
||||
if let CultNetMessage::OperationResponse { status, .. } = &mut deferred {
|
||||
*status = answer_status;
|
||||
}
|
||||
deferred
|
||||
}
|
||||
|
||||
/// Until `stopping`: expire what timed out, resend what was not acknowledged,
|
||||
|
|
@ -924,9 +974,12 @@ mod tests {
|
|||
fn an_answer_over_the_deferral_bound_is_refused_by_name() {
|
||||
let payload_of = |len: u64| STANDARD.encode(vec![0_u8; len as usize]);
|
||||
let mut bodies = fresh_bodies();
|
||||
// The longest id a request may carry: the reply echoes it, so the
|
||||
// largest deferred answer and the largest refusal are measured at it.
|
||||
let id = "m".repeat(MAX_ECHOED_FIELD_BYTES);
|
||||
|
||||
let at = reply_with_payload("m-b", "view", "huginn-yggdrasil", payload_of(MAX_DEFERRED_BODY_BYTES));
|
||||
let deferred = deliver(at, "m-b", "view", "huginn-yggdrasil", &mut bodies, now());
|
||||
let at = reply_with_payload(&id, "view", "huginn-yggdrasil", payload_of(MAX_DEFERRED_BODY_BYTES));
|
||||
let deferred = deliver(at, &id, "view", "huginn-yggdrasil", &mut bodies, now());
|
||||
let manifest = deferred_manifest(&deferred);
|
||||
assert_eq!(manifest.size_bytes as u64, MAX_DEFERRED_BODY_BYTES);
|
||||
assert_eq!(manifest.chunks.len() as u64, MAX_DEFERRED_BODY_BYTES / DEFERRED_CHUNK_BYTES as u64);
|
||||
|
|
@ -934,18 +987,18 @@ mod tests {
|
|||
assert_eq!(bodies.body_count(), 1);
|
||||
drop(bodies);
|
||||
|
||||
let over = reply_with_payload("m-b", "view", "huginn-yggdrasil", payload_of(MAX_DEFERRED_BODY_BYTES + 1));
|
||||
let over = reply_with_payload(&id, "view", "huginn-yggdrasil", payload_of(MAX_DEFERRED_BODY_BYTES + 1));
|
||||
let mut empty = fresh_bodies();
|
||||
let refused = deliver(over, "m-b", "view", "huginn-yggdrasil", &mut empty, now());
|
||||
let refused = deliver(over, &id, "view", "huginn-yggdrasil", &mut empty, now());
|
||||
let (correlation, answered) = decode_response(&refused).unwrap();
|
||||
assert_eq!(correlation, "m-b");
|
||||
assert_eq!(correlation, id);
|
||||
let HuginnMindResponse::Refused(MindRefusal::ResponseTooLarge { bytes, limit }) =
|
||||
answered.expect("a refusal is an answer, not an envelope failure")
|
||||
else {
|
||||
panic!("one byte over the deferral bound was not refused: {refused:?}");
|
||||
};
|
||||
assert_eq!((bytes, limit), (MAX_DEFERRED_BODY_BYTES + 1, MAX_DEFERRED_BODY_BYTES));
|
||||
assert!(encoded_len(&refused) <= MAX_FRAGMENT_BYTES as u64, "the refusal is one packet");
|
||||
assert!(encoded_len(&refused) <= MAX_RESPONSE_BYTES, "the refusal fits one send at the longest id");
|
||||
assert_eq!(empty.body_count(), 0, "a refused answer retains nothing");
|
||||
}
|
||||
|
||||
|
|
@ -1040,6 +1093,96 @@ mod tests {
|
|||
);
|
||||
}
|
||||
|
||||
/// Every field a reply echoes is bounded at the door, so no reply can
|
||||
/// outgrow the window: a request whose `message_id`, `service_id`,
|
||||
/// `operation`, `payload_schema` or `chunk_hash` is over the bound is
|
||||
/// answered with a transport `Error` that fits and carries none of them,
|
||||
/// and touches no mind; a field of exactly the bound is served as usual.
|
||||
/// The 1.3 MB id is the one that made a `Whoami` answer echo past the
|
||||
/// window, deferred it, and left the client waiting on a send the hub
|
||||
/// refused.
|
||||
#[test]
|
||||
fn an_overlong_echoed_field_is_refused_at_the_door_with_an_answer_that_fits() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let mut daemon = Daemon::open(root.path(), &slug(INSTANCE)).unwrap();
|
||||
let registry = schema_registry().unwrap();
|
||||
let mut bodies = fresh_bodies();
|
||||
let huge = "x".repeat(1_300_000);
|
||||
let over = "x".repeat(MAX_ECHOED_FIELD_BYTES + 1);
|
||||
let at = "x".repeat(MAX_ECHOED_FIELD_BYTES);
|
||||
let whoami = |message_id: &str, service_id: &str, operation: &str, payload_schema: &str| {
|
||||
let payload = STANDARD.encode(rmp_serde::to_vec_named(&HuginnMindRequest::Whoami).unwrap());
|
||||
CultNetMessage::OperationRequest {
|
||||
message_id: message_id.into(),
|
||||
service_id: service_id.into(),
|
||||
operation: operation.into(),
|
||||
payload_schema: payload_schema.into(),
|
||||
payload_encoding: "messagepack-base64".into(),
|
||||
payload,
|
||||
source_runtime_id: None,
|
||||
target_runtime_id: None,
|
||||
}
|
||||
};
|
||||
let chunk = |message_id: &str, chunk_hash: &str, record_key: &str| CultNetMessage::ContentChunkRequest {
|
||||
message_id: message_id.into(),
|
||||
chunk_hash: chunk_hash.into(),
|
||||
record_key: record_key.into(),
|
||||
expected_size_bytes: 1,
|
||||
};
|
||||
let refused = [
|
||||
("huge message_id", whoami(&huge, MIND_SERVICE_ID, "whoami", MIND_REQUEST_SCHEMA)),
|
||||
("message_id", whoami(&over, MIND_SERVICE_ID, "whoami", MIND_REQUEST_SCHEMA)),
|
||||
("service_id", whoami("m", &huge, "whoami", MIND_REQUEST_SCHEMA)),
|
||||
("operation", whoami("m", MIND_SERVICE_ID, &huge, MIND_REQUEST_SCHEMA)),
|
||||
("payload_schema", whoami("m", MIND_SERVICE_ID, "whoami", &huge)),
|
||||
(
|
||||
"catalog message_id",
|
||||
CultNetMessage::SchemaCatalogRequest {
|
||||
message_id: huge.clone(),
|
||||
include_schema_json: Some(true),
|
||||
schema_ids: None,
|
||||
kinds: None,
|
||||
},
|
||||
),
|
||||
("chunk message_id", chunk(&huge, "abc", "")),
|
||||
("chunk_hash", chunk("m", &huge, "")),
|
||||
("record_key", chunk("m", "abc", &huge)),
|
||||
];
|
||||
for (case, message) in refused {
|
||||
let reply = answer(&mut daemon, ®istry, &mut bodies, message, now());
|
||||
let CultNetMessage::Error { error, .. } = &reply else {
|
||||
panic!("{case}: expected a transport error, got {reply:?}");
|
||||
};
|
||||
assert!(error.contains("over 256 bytes"), "{case}: {error}");
|
||||
assert!(encoded_len(&reply) <= MAX_FRAGMENT_BYTES as u64, "{case}: the refusal is one packet");
|
||||
}
|
||||
assert_eq!(documents(&mut daemon), 0);
|
||||
|
||||
let reply = answer(&mut daemon, ®istry, &mut bodies, whoami(&at, MIND_SERVICE_ID, "whoami", MIND_REQUEST_SCHEMA), now());
|
||||
assert_eq!(decode_response(&reply).unwrap().0, at, "a field of exactly the bound is served");
|
||||
let reply = answer(&mut daemon, ®istry, &mut bodies, chunk(&at, &at, &at), now());
|
||||
assert!(!found(&reply), "an unknown chunk is answered found: false, not refused at the door");
|
||||
}
|
||||
|
||||
/// The envelope's status is the answer's own whether it is sent whole or
|
||||
/// deferred, so a client that routes on status routes the same way.
|
||||
#[test]
|
||||
fn a_deferred_answer_keeps_the_status_of_the_answer_it_stands_for() {
|
||||
for status in ["rejected", "accepted"] {
|
||||
let payload = STANDARD.encode(vec![0_u8; MAX_RESPONSE_BYTES as usize]);
|
||||
let mut reply = reply_with_payload("m-s", "view", "huginn-yggdrasil", payload);
|
||||
let CultNetMessage::OperationResponse { status: carried, .. } = &mut reply else { unreachable!() };
|
||||
*carried = status.into();
|
||||
let mut bodies = fresh_bodies();
|
||||
let deferred = deliver(reply, "m-s", "view", "huginn-yggdrasil", &mut bodies, now());
|
||||
let CultNetMessage::OperationResponse { status: envelope, .. } = &deferred else {
|
||||
panic!("expected an operation response, got {deferred:?}");
|
||||
};
|
||||
assert_eq!(envelope, status);
|
||||
deferred_manifest(&deferred);
|
||||
}
|
||||
}
|
||||
|
||||
/// One cut spec's own reference, by its cut label.
|
||||
fn spec_ref(daemon: &mut Daemon<OwnedRedbMessagePackBackingStore, NoIndex>, cut: &str) -> PipelineRef {
|
||||
let selection = Selection {
|
||||
|
|
|
|||
|
|
@ -125,7 +125,8 @@ impl HuginnMindResponse {
|
|||
/// The envelope's `status`, in the C# reference's vocabulary: `rejected`
|
||||
/// exactly for a refusal, `accepted` otherwise. `Conflict` and
|
||||
/// `AlreadyAdmitted` are answers about the mind, not refusals of the
|
||||
/// request.
|
||||
/// request. A `Deferred` is nominally `accepted`; the daemon sets the
|
||||
/// envelope of a deferred answer to the status of the answer it stands for.
|
||||
pub fn status(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Refused(_) | Self::Admit(PipelineAdmissionOutcome::Refused(_)) => "rejected",
|
||||
|
|
|
|||
Loading…
Reference in a new issue