diff --git a/CHANGELOG.md b/CHANGELOG.md index 489e0a431..fedd731bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,11 +10,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). * Per-direction virtqueue configuration through `SandboxConfiguration` and `SandboxBuilder`, with allocations included in scratch sizing. * Shared virtqueue framing with a 12-byte `MsgHeader` and external byte values. +* `MailboxValue` defines the transport mailbox's allowed wire values. +* `virtq::canonical::validate_canon_prefix` validates a bounded available + descriptor prefix. * `ExternalValueSource` implementations for `RecvChain` and `Segments`. * Producer batch completion without notification and segmented payload assembly and extraction without flattening. +* Retained guest `Bytes` and `ByteChunks` preserve contents and pointers across + snapshot capture, restore, and cloning. ### Changed +* Guest transport aliases follow buffer ownership. Restored buffers keep + captured backing while their scratch slots become reusable. * `Sandbox` is the primary initialized sandbox type. `MultiUseSandbox` remains as a deprecated alias. * `Sandbox` lives in the private `sandbox::initialized` module and is reached @@ -32,8 +39,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). the configured level is above `OFF` rather than whether the tracing state was allocated. * **Breaking:** Virtqueue rings and pools occupy host-owned scratch before page - tables. Snapshots use ABI 5 and config schema v3. Existing snapshots must be - regenerated. + tables. Retained-buffer checkpoints use ABI 6 and config schema v3. Existing + snapshots must be regenerated. * Host virtqueue access uses checked copies and atomics across mapped scratch. Snapshot admission checks geometry, canonical rings, and distinct, aligned H2G pool slots. @@ -53,7 +60,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). rejects snapshots without transport state. * Running snapshots checkpoint dirty virtqueues before capture. Ordinary calls keep their deferred result path. -* Reject snapshot capture while guest-owned transport buffers are retained. * Use the reclaimed stack pages to raise the default G2H and H2G pools to 12 and 8 pages. * `hyperlight_guest_bin::exception::arch`, previously available on diff --git a/Cargo.lock b/Cargo.lock index 7b78932f0..25b18a44b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1702,6 +1702,7 @@ dependencies = [ "flatbuffers", "hyperlight-common", "hyperlight-guest-tracing", + "itertools 0.15.0", "serde_json", "tracing", ] diff --git a/docs/snapshot-versioning.md b/docs/snapshot-versioning.md index 368b39e9f..b86f72ca0 100644 --- a/docs/snapshot-versioning.md +++ b/docs/snapshot-versioning.md @@ -38,6 +38,9 @@ A snapshot carries four independently evolvable version markers: and requires a transport layer. Config v1 and v2 are incompatible with the current ABI. +ABI 6 requires checkpoint readiness after guest initialization and permits +H2G prefill to omit retained slots. Earlier snapshots must be regenerated. + The `OCI_LAYOUT_VERSION` constant is pinned by the OCI image-layout spec at `1.0.0`. diff --git a/docs/virtio-host-guest-communication.md b/docs/virtio-host-guest-communication.md index 3b1529a58..468d241fa 100644 --- a/docs/virtio-host-guest-communication.md +++ b/docs/virtio-host-guest-communication.md @@ -5,9 +5,6 @@ packed virtqueues. It uses the packed ring layout and ownership rules, but it is not a discoverable VIRTIO device. Queue configuration, arena placement, and notification behavior are part of the Hyperlight ABI. -This document describes the fixed-pool runtime, which rejects snapshots with -retained buffers. - ## Architecture The guest is the driver (producer) for both queues. The host is the device @@ -45,7 +42,7 @@ optional writable response capacity. ## Transport arena -Both rings, the checkpoint mailbox, and both pools occupy one fixed prefix of +Both rings, the transport mailbox, and both pools occupy one fixed prefix of guest scratch memory. ```text @@ -170,12 +167,16 @@ On the first VM entry, the guest: 1. Reads the published configuration. 2. Reconstructs `TransportArena`. 3. Converts each transport GPA into its scratch GVA. -4. Creates both packed ring producers and slot pools. -5. Prefills H2G with one writable descriptor per available H2G slot, bounded - by queue size. -6. Publishes the resulting `GuestContext`. +4. Constructs a `GuestContext` and installs it with `set_global_context`. + H2G receives one writable descriptor per available slot, bounded by queue size. + +Dispatch starts with `transport::maybe_refresh`, before logging or tracing. +Runtime operations borrow the context through `transport::with_ctx`. -The host consumers observe the descriptors after guest initialization. +After user initialization and trace flushing, the guest calls +`transport::prepare_snapshot`, then halts. The host resets its consumers and +requires `MailboxValue::CheckpointComplete`. Initial snapshots include values +retained during initialization. ## Wire format @@ -362,6 +363,11 @@ drains and acknowledges them during the same VM exit. Guest `SlotPool` instances own all transport buffers. Pool clones share one allocation bitmap with each producer. +Each producer pairs its backend with that pool. Completion leases carry the +original slot addresses and full capacities. Backing owns alias allocation and +paging. The backend holds only scratch bounds. +Guest completion mapping checks scratch bounds. Allocation ownership and +initialized lengths follow the `BufferMap` safety contract. ```text Free -> allocated -> published -> completed -> owner-backed Bytes -> Free @@ -375,15 +381,29 @@ external `ByteChunks`: * G2H host responses can become owner-backed guest `Bytes`. * `VecBytes` values copy into a contiguous `Vec`. * Multiple `Bytes` clones or slices backed by one owner keep one slot live. -* The slot returns to the pool when the final owner drops. +* The final owner unmaps its alias and returns the virtual range for reuse. + Its lease releases the scratch slot only in the slot's current generation. Producer reset releases allocations still owned by queue bookkeeping. After both producers reset and before H2G prefill, every live pool slot belongs to -guest retained `Bytes`. +guest retained `Bytes`. H2G prefills free slots up to the queue size. Checkpoint preparation requires stopped host consumers with no live chain handles. The host resets both consumers before processing more queue traffic. +### Retained virtual addresses + +Both pools share one guest-global alias allocator. Its state is captured with +the mappings and keeps retained ranges reserved across pool generations. +Each completed `GuestMapping` owns a separate page-aligned range. Mapping occurs +when the guest receives the buffer. Capture, restore, and cloning preserve its +pointer and contents. + +Retained aliases reach captured memory after restore. The first transport +entry advances the pool generations and recycles slots outside posted chains. +Retained values keep their aliases until their final owner drops. Reusing +freed virtual ranges bounds page-table growth by the alias high-water mark. + ### Trust boundary The host treats guest rings, descriptors, headers, FlatBuffers, and payload @@ -403,7 +423,8 @@ lengths as untrusted. The transport arena lives in scratch and is not captured as ordinary guest memory. Guest producer and pool bookkeeping is normal guest state, while ring and pool bytes live in scratch. Snapshot capture needs a canonical transport -state. +state. Retained aliases map live payload pages outside the scratch map, so +capture copies them as ordinary memory. `Sandbox` tracks whether queue traffic occurred after the last canonical boundary. A cached or clean snapshot needs no VM entry. A dirty @@ -412,29 +433,33 @@ snapshot uses this flow: ```text Host Guest | | - | mailbox = u64::MAX | + | mailbox = CheckpointPending | | H2G SnapshotCheckpoint ------------>| | enter VM | | | reclaim completed G2H work | | reset G2H producer | | reset H2G producer - | | count live pool slots - | | publish mailbox count - | | prefill H2G + | | prefill free H2G slots + | | mailbox = CheckpointComplete |<------------------------------------| halt | reset both consumers | - | read mailbox | - | capture memory and rings | - | validate ring images | + | require CheckpointComplete | + | read and validate ring images | + | capture memory | ``` +Checkpoint preparation keeps retained leases and aliases intact without +payload copying. Capture leaves the source queues and allocator ready for +continued use, including when memory capture fails. + The canonical state is: * G2H is empty at cursor zero. -* H2G starts at cursor zero with one writable descriptor per complete free - slot in the configured pool, bounded by queue size. Each descriptor names a - distinct, configured-size slot aligned relative to the pool start. -* Guest producer and pool bookkeeping matches the rings. +* H2G starts at cursor zero with one writable descriptor per free slot, + bounded by queue size. Each descriptor names a distinct, configured-size + slot aligned relative to the pool start. Available descriptors form a prefix + followed by zeroed descriptors. +* Guest producer and pool bookkeeping matches the rings and current leases. * Driver and device event suppression is normalized. * Host consumers start at cursor zero. @@ -442,8 +467,9 @@ The snapshot stores normal guest memory plus the two canonical ring images. Construction and loading validate the ring images against the finalized layout. The layout and copied ring images remain immutable. The OCI representation places ring images in the -[transport layer](./snapshot-oci-format.md). Pool payload bytes, the mailbox, -and host consumer cursors are not stored. +[transport layer](./snapshot-oci-format.md). Retained payloads use the ordinary +memory layer through their aliases. The mailbox and host consumer cursors are +not stored. ### Restore @@ -453,53 +479,52 @@ state against the layout. Admitted images and their layout remain immutable. Transport admission precedes changes to sandbox status, the cached snapshot, and memory mappings. -Restore writes the arena GPA metadata and both ring images into fresh scratch. -It attaches new host consumers at cursor zero. Normal guest memory restores -the matching producer and pool bookkeeping. Restore does not need a preparatory -VM entry. +Restore writes the arena GPA metadata, a `CheckpointComplete` mailbox value, +and both ring images into fresh scratch. It attaches new host consumers at +cursor zero. Normal guest memory restores the matching producers, pools, +leases, and aliases. +Initialized `restore` and `from_snapshot` are ready for the first H2G request +before guest entry. Pre-initialization snapshots use normal guest startup. -## Retention mailbox +The first request fits the H2G capacity posted at checkpoint. Retained slots +reduce pool capacity but hold no ring descriptors. Framing, slot rounding, ring +size, and the external-byte control reserve still apply. -The mailbox is one `u64` in the ring to pool alignment gap. It is outside both -rings and pools. Both sides derive its address from trusted arena geometry. -The host accesses it before VM entry and after guest halt. +The first transport entry after restore recycles slots held by captured +leases. Posted chains keep their reservations, preserving the submitted +request. Descriptors and cursors stay unchanged. Retained aliases keep their +captured mappings without payload copying. -The mailbox avoids a G2H checkpoint response. G2H can remain empty in the -canonical image even when retained G2H slots reduce available capacity. +Source continuation keeps the pool generation and live scratch leases intact. +Checkpointing alone does not detach retained payloads from scratch. -Before a dirty checkpoint, the host writes `u64::MAX` as a pending marker. -After producer reset, the guest writes: +Result completion prefills free H2G slots. This requires no preparatory guest +entry or application warmup for the checkpoint-posted capacity. -```text -g2h_producer.pool().num_live() + h2g_producer.pool().num_live() -``` +## Transport mailbox -The host reads the value after a successful guest halt and after resetting -both consumers. - -* `u64::MAX` is a fatal incomplete checkpoint. -* Zero permits snapshot capture. -* A nonzero count rejects capture without poisoning the sandbox. +The mailbox holds a `MailboxValue` encoded as one `u64` in the ring to pool +alignment gap. It is outside both rings and pools. Both sides derive its +address from trusted arena geometry. The host accesses it before VM entry and +after guest halt. -A nonzero rejection leaves the queues usable and keeps transport dirty. -Guest code can release retained values and retry the snapshot. +The mailbox avoids a G2H checkpoint response. G2H can remain empty in the +canonical image even when retained G2H slots reduce available capacity. -The count only answers whether retained slots exist. It does not contain pool -identity, addresses, or initialized lengths. Retained pool payloads cannot be -restored because pool bytes are absent from the snapshot. +Before a dirty checkpoint, the host writes `CheckpointPending`. +The guest writes `CheckpointComplete` after producer reset and free-slot H2G prefill. +The host reads the value after a successful guest halt and after resetting +both consumers. -To preserve transport-backed data across snapshots, copy it into -guest-heap-owned storage, such as a `Vec`, and release all transport-backed -views before capture. This preserves the data at the cost of a payload copy. -Cloning `Bytes` only shares the original buffer and does not remove the -restriction. +* `CheckpointPending` (`0`) is a fatal incomplete checkpoint. +* `CheckpointComplete` (`1`) permits snapshot capture. +* Every other value is a fatal invalid status. ## Placement and relocation limitations The fixed-pool runtime places both rings, the mailbox, and both pools in one host-owned arena at the scratch base. The guest reconstructs that layout from -host metadata. Canonical capture and attachment require the published arena -address to match the configured address. +host metadata. Descriptors, pool owners, and producer state contain absolute GVAs. Restore adopts the snapshot's scratch size, queue geometry, and transport addresses, @@ -508,12 +533,6 @@ even when the target sandbox was created with a different layout. Transport capacity is fixed when the sandbox is created. Runtime queue resize and VIRTIO feature negotiation are not supported. -## Future work - -The current runtime rejects snapshots with retained transport-backed buffers. -Planned work aims to preserve guest-held `Bytes` and `ByteChunks` across capture, -restore, and cloning using guest-allocated pools. - ## Source map * Shared framing: [`src/hyperlight_common/src/transport.rs`](../src/hyperlight_common/src/transport.rs) diff --git a/src/hyperlight_common/src/arch/aarch64/layout.rs b/src/hyperlight_common/src/arch/aarch64/layout.rs index 83a0fdcac..6f0deed18 100644 --- a/src/hyperlight_common/src/arch/aarch64/layout.rs +++ b/src/hyperlight_common/src/arch/aarch64/layout.rs @@ -6,6 +6,8 @@ pub const SCRATCH_TOP_GVA: usize = 0x0000_ffff_ffff_dfff; pub const SNAPSHOT_PT_GVA_MIN: usize = 0x0000_8000_0000_0000; pub const SNAPSHOT_PT_GVA_MAX: usize = 0x0000_80ff_ffff_ffff; +pub const VIRTQ_BUFFER_GVA_START: u64 = 0x0000_fc00_0000_0000; +pub const VIRTQ_BUFFER_GVA_END: u64 = 0x0000_fd00_0000_0000; pub const SCRATCH_TOP_GPA: usize = 0x0000_000f_ffff_bfff; pub const IO_PAGE_GVA: u64 = 0x0000_ffff_ffff_e000; @@ -16,6 +18,8 @@ pub const WHP_GITS_TRANSLATOR_BASE_GPA: u64 = 0xeff6_8000; pub const WHP_GICR_BASE_GPA: u64 = 0xeffe_e000; pub const WHP_GICD_BASE_GPA: u64 = 0xffff_0000; +const _: () = assert!(VIRTQ_BUFFER_GVA_END <= 1 << 48); + pub const fn io_page() -> Option<(crate::vmem::PhysAddr, crate::vmem::VirtAddr)> { Some((IO_PAGE_GPA, IO_PAGE_GVA)) } diff --git a/src/hyperlight_common/src/arch/amd64/layout.rs b/src/hyperlight_common/src/arch/amd64/layout.rs index c4759d004..f60e6ddf3 100644 --- a/src/hyperlight_common/src/arch/amd64/layout.rs +++ b/src/hyperlight_common/src/arch/amd64/layout.rs @@ -11,6 +11,8 @@ pub const SCRATCH_TOP_GVA: usize = 0xffff_ffff_ffff_efff; pub const SNAPSHOT_PT_GVA_MIN: usize = 0xffff_8000_0000_0000; pub const SNAPSHOT_PT_GVA_MAX: usize = 0xffff_80ff_ffff_ffff; +pub const VIRTQ_BUFFER_GVA_START: u64 = 0xffff_fc00_0000_0000; +pub const VIRTQ_BUFFER_GVA_END: u64 = 0xffff_fd00_0000_0000; /// We assume 36-bit IPAs for now, since every amd64 processor /// supports at least 36 bits. Almost all of them support at least 40 @@ -18,6 +20,11 @@ pub const SNAPSHOT_PT_GVA_MAX: usize = 0xffff_80ff_ffff_ffff; /// ever memory-constrained. pub const SCRATCH_TOP_GPA: usize = 0x0000_000f_ffff_ffff; +const _: () = { + assert!(VIRTQ_BUFFER_GVA_START >> 47 == 0x1ffff); + assert!((VIRTQ_BUFFER_GVA_END - 1) >> 47 == 0x1ffff); +}; + pub fn io_page() -> Option<(u64, u64)> { None } diff --git a/src/hyperlight_common/src/layout.rs b/src/hyperlight_common/src/layout.rs index 6983fd07e..43b74b3fa 100644 --- a/src/hyperlight_common/src/layout.rs +++ b/src/hyperlight_common/src/layout.rs @@ -9,7 +9,8 @@ use core::num::{NonZeroU16, NonZeroUsize}; mod arch; pub use arch::{ - SCRATCH_TOP_GPA, SCRATCH_TOP_GVA, SNAPSHOT_PT_GVA_MAX, SNAPSHOT_PT_GVA_MIN, io_page, + SCRATCH_TOP_GPA, SCRATCH_TOP_GVA, SNAPSHOT_PT_GVA_MAX, SNAPSHOT_PT_GVA_MIN, + VIRTQ_BUFFER_GVA_END, VIRTQ_BUFFER_GVA_START, io_page, }; #[cfg(target_arch = "aarch64")] pub use arch::{WHP_GICD_BASE_GPA, WHP_GICR_BASE_GPA, WHP_GITS_TRANSLATOR_BASE_GPA}; @@ -102,6 +103,14 @@ const _: () = { assert!(SCRATCH_TOP_H2G_POOL_PAGES_OFFSET == 0x60); assert!(SCRATCH_TOP_H2G_BUFFER_SIZE_OFFSET == 0x68); assert!(SCRATCH_TOP_EXN_STACK_OFFSET == 0x70); + + assert!(VIRTQ_BUFFER_GVA_START < VIRTQ_BUFFER_GVA_END); + assert!((VIRTQ_BUFFER_GVA_START as usize).is_multiple_of(crate::vmem::PAGE_SIZE)); + assert!((VIRTQ_BUFFER_GVA_END as usize).is_multiple_of(crate::vmem::PAGE_SIZE)); + assert!(VIRTQ_BUFFER_GVA_START > SNAPSHOT_PT_GVA_MAX as u64); + + assert!(VIRTQ_BUFFER_GVA_END < scratch_base_gva(16 * 1024 * 1024 * 1024)); + assert!((SCRATCH_TOP_GPA as u64) < VIRTQ_BUFFER_GVA_END - VIRTQ_BUFFER_GVA_START); }; /// Exclusive upper GPA boundary for dynamic scratch allocations. @@ -112,14 +121,14 @@ pub const fn scratch_allocator_limit_gpa() -> u64 { pub fn scratch_base_gpa(size: usize) -> u64 { (SCRATCH_TOP_GPA - size + 1) as u64 } -pub fn scratch_base_gva(size: usize) -> u64 { +pub const fn scratch_base_gva(size: usize) -> u64 { (SCRATCH_TOP_GVA - size + 1) as u64 } -/// Compute the minimum scratch region size needed for a sandbox. +/// Compute fixed scratch overhead for a sandbox, saturating at [`usize::MAX`]. /// -/// `transport_len` includes both rings and buffer pools. -/// The result saturates at [`usize::MAX`]. +/// `transport_len` covers rings and buffer pools. Snapshot page tables and +/// runtime allocations need additional scratch. pub fn min_scratch_size(transport_len: usize) -> usize { arch::min_scratch_size() .and_then(|fixed| fixed.checked_add(transport_len)) @@ -396,11 +405,12 @@ mod tests { } #[test] - fn minimum_scratch_includes_ring_arena_and_pools() { + fn minimum_scratch_includes_transport() { let fixed = arch::min_scratch_size().unwrap(); - let transport_len = (1 + 8 + 4) * crate::vmem::PAGE_SIZE; - assert_eq!(fixed + transport_len, min_scratch_size(transport_len)); + for transport_len in [0, (1 + 8 + 4) * crate::vmem::PAGE_SIZE, 5 << 20] { + assert_eq!(fixed + transport_len, min_scratch_size(transport_len)); + } } #[test] diff --git a/src/hyperlight_common/src/transport.rs b/src/hyperlight_common/src/transport.rs index 8e8cd46f1..6f7445e14 100644 --- a/src/hyperlight_common/src/transport.rs +++ b/src/hyperlight_common/src/transport.rs @@ -55,6 +55,40 @@ impl TryFrom for MsgKind { } } +/// Allowed values in the transport mailbox. +#[repr(u64)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, bytemuck::CheckedBitPattern, bytemuck::NoUninit)] +pub enum MailboxValue { + /// Guest preparation is outstanding. + CheckpointPending = 0, + /// Both queues are reset and free H2G slots are prefilled. + CheckpointComplete = 1, +} + +impl MailboxValue { + /// Return the mailbox wire representation. + pub fn as_bytes(&self) -> &[u8] { + bytemuck::bytes_of(self) + } + + /// Return wire representation as a raw integer. + pub fn raw(self) -> u64 { + self as u64 + } +} + +impl TryFrom for MailboxValue { + type Error = u64; + + fn try_from(value: u64) -> Result { + match value { + 0 => Ok(Self::CheckpointPending), + 1 => Ok(Self::CheckpointComplete), + other => Err(other), + } + } +} + /// Wire header for all virtqueue messages. #[derive(Debug, Clone, Copy, PartialEq, Eq, bytemuck::CheckedBitPattern, bytemuck::NoUninit)] #[repr(C)] @@ -428,9 +462,32 @@ pub const fn size_prefixed_len(payload_len: usize) -> Option { #[cfg(test)] mod tests { + use bytemuck::checked::try_pod_read_unaligned; + use super::*; use crate::flatbuffer_wrappers::ExternalValueSink; + /// Mailbox words have fixed encodings and reject undefined values. + #[test] + fn mailbox_value_wire_contract() { + for (value, raw) in [ + (MailboxValue::CheckpointPending, 0u64), + (MailboxValue::CheckpointComplete, 1), + ] { + assert_eq!(value.raw(), raw); + assert_eq!(value.as_bytes(), raw.to_ne_bytes()); + assert_eq!(MailboxValue::try_from(raw), Ok(value)); + + let bits = try_pod_read_unaligned::(value.as_bytes()).unwrap(); + assert_eq!(bits, value); + } + + for raw in [2u64, 3, 0xa000, u64::MAX] { + assert_eq!(MailboxValue::try_from(raw), Err(raw)); + assert!(try_pod_read_unaligned::(&raw.to_ne_bytes()).is_err()); + } + } + #[test] fn header_contains_framing_fields() { let header = MsgHeader::new(MsgKind::Response, 0x1234_5678, 4096); diff --git a/src/hyperlight_common/src/virtq/buffer.rs b/src/hyperlight_common/src/virtq/buffer.rs index c66fe80d0..cecdbd82f 100644 --- a/src/hyperlight_common/src/virtq/buffer.rs +++ b/src/hyperlight_common/src/virtq/buffer.rs @@ -245,10 +245,12 @@ impl Buf for SegmentsBuf<'_> { } } -/// An exclusively owned buffer allocation returned to its pool on drop. +/// A buffer allocation returned on drop while its pool generation is current. pub struct BufferLease { /// The pool that allocated the buffer. pool: SlotPool, + /// The pool generation at the time of allocation. + generation: u64, /// The buffer's start address and full allocation capacity. allocation: Allocation, } @@ -256,7 +258,13 @@ pub struct BufferLease { impl BufferLease { /// Create a new buffer lease from a pool and allocation. pub fn new(pool: SlotPool, allocation: Allocation) -> Self { - Self { pool, allocation } + let generation = pool.generation(); + + Self { + pool, + allocation, + generation, + } } /// The buffer's start address and full allocation capacity. @@ -267,6 +275,10 @@ impl BufferLease { impl Drop for BufferLease { fn drop(&mut self) { + if self.generation != self.pool.generation() { + return; + } + if let Err(error) = self.pool.dealloc(self.allocation.addr) { log::error!("Failed to release a virtqueue buffer: {error}"); debug_assert!(false, "BufferLease deallocation failed: {error}"); @@ -302,6 +314,42 @@ mod tests { pool.dealloc(reused.addr).unwrap(); } + #[test] + fn retired_lease_does_not_release_a_reused_slot() { + let pool = SlotPool::new(SlotLayout::new(0x1000, 4096, 1).unwrap()).unwrap(); + let allocation = pool.alloc(4096).unwrap(); + let retained = BufferLease::new(pool.clone(), allocation); + + // SAFETY: This test has no views or queue reservations. + unsafe { pool.retire_generation(core::iter::empty()) }.unwrap(); + let reused = pool.alloc(4096).unwrap(); + assert_eq!(reused.addr, allocation.addr); + + drop(retained); + assert_eq!(pool.num_free(), 0); + drop(BufferLease::new(pool.clone(), reused)); + assert_eq!(pool.num_free(), 1); + } + + #[test] + fn generation_recovery_preserves_queue_reservations() { + let pool = SlotPool::new(SlotLayout::new(0x1000, 4096, 2).unwrap()).unwrap(); + let reserved = pool.alloc(4096).unwrap(); + let captured = pool.alloc(4096).unwrap(); + let retained = BufferLease::new(pool.clone(), captured); + + // SAFETY: No view exists. The iterator names the only queue reservation. + unsafe { pool.retire_generation(core::iter::once(reserved.addr)) }.unwrap(); + assert_eq!(pool.num_free(), 1); + assert_eq!(pool.alloc(4096).unwrap().addr, captured.addr); + + drop(retained); + assert_eq!(pool.num_free(), 0); + pool.dealloc(reserved.addr).unwrap(); + pool.dealloc(captured.addr).unwrap(); + assert_eq!(pool.num_free(), 2); + } + #[test] fn segments_cursor_advances_across_segments() { let segments = Segments::new([ diff --git a/src/hyperlight_common/src/virtq/pool/fuzz.rs b/src/hyperlight_common/src/virtq/pool/fuzz.rs index 86637be30..a83055d93 100644 --- a/src/hyperlight_common/src/virtq/pool/fuzz.rs +++ b/src/hyperlight_common/src/virtq/pool/fuzz.rs @@ -147,26 +147,6 @@ fn check_slot_pool_invariants( return Err("free + live != total slots"); } - // Free-slot enumeration must match the reported count and be strictly ordered. - let mut free = Vec::new(); - pool.for_each_free(|allocation| free.push(allocation)); - - if free.len() != pool.num_free() { - return Err("free-slot visitation is inconsistent"); - } - - if free.windows(2).any(|pair| pair[0].addr >= pair[1].addr) { - return Err("free-slot visitation is inconsistent"); - } - - // Free slots cannot also be live and must report their tier's full capacity. - if free.iter().any(|allocation| { - expected_live.contains_key(&allocation.addr) - || slot_capacity(pool, allocation.addr) != Some(allocation.len as usize) - }) { - return Err("free-slot visitation is inconsistent"); - } - // Reported geometry must agree with the stored tier layouts. let (lower, upper) = pool.layouts(); let expected_base = lower.map_or(upper.base_addr(), SlotLayout::base_addr); diff --git a/src/hyperlight_common/src/virtq/pool/slot.rs b/src/hyperlight_common/src/virtq/pool/slot.rs index e0c690e0a..84ff6b2fe 100644 --- a/src/hyperlight_common/src/virtq/pool/slot.rs +++ b/src/hyperlight_common/src/virtq/pool/slot.rs @@ -186,6 +186,28 @@ impl Tier { Ok(()) } + /// Keep one queue-owned slot unavailable during generation recovery. + fn reserve_addr(&mut self, addr: u64) -> Result<(), AllocError> { + let slot = self.slot_of(addr)?; + + if self.allocated.contains(slot) { + return Err(AllocError::InvalidArg); + } + + self.allocated.insert(slot); + Ok(()) + } + + /// Rebuild the free list from the recovered ownership bitmap. + fn rebuild_free(&mut self) { + self.free.clear(); + + for slot in self.allocated.zeroes() { + self.free + .push(self.base_addr + slot as u64 * u64::from(self.slot_size)); + } + } + fn allocation_len(&self, addr: u64) -> Result { self.live_slot_of(addr)?; Ok(self.slot_size as usize) @@ -207,15 +229,6 @@ impl Tier { ); } - fn for_each_free(&self, f: &mut impl FnMut(Allocation)) { - for slot in self.allocated.zeroes() { - f(Allocation { - addr: self.base_addr + slot as u64 * u64::from(self.slot_size), - len: self.slot_size, - }); - } - } - fn layout(&self) -> SlotLayout { SlotLayout { base_addr: self.base_addr, @@ -228,6 +241,7 @@ impl Tier { struct Inner { lower: Option, upper: Tier, + generation: u64, } impl Inner { @@ -236,6 +250,7 @@ impl Inner { return Ok(Self { lower: None, upper: Tier::from_layout(upper)?, + generation: 0, }); }; @@ -258,12 +273,14 @@ impl Inner { return Ok(Self { lower: None, upper: Tier::from_layout(layout)?, + generation: 0, }); } Ok(Self { lower: Some(Tier::from_layout(lower)?), upper: Tier::from_layout(upper)?, + generation: 0, }) } @@ -294,6 +311,43 @@ impl Inner { self.upper.dealloc_addr(addr) } + /// Start a new generation and rebuild allocation state from `reserved`. + fn retire_generation( + &mut self, + reserved: impl Iterator + Clone, + ) -> Result<(), AllocError> { + let generation = self.generation.checked_add(1).ok_or(AllocError::Overflow)?; + + for addr in reserved.clone() { + self.allocation_len(addr)?; + } + + self.generation = generation; + self.upper.allocated.clear(); + + if let Some(lower) = &mut self.lower { + lower.allocated.clear(); + } + + for addr in reserved { + if let Some(lower) = &mut self.lower + && lower.contains(addr) + { + lower.reserve_addr(addr)?; + } else { + self.upper.reserve_addr(addr)?; + } + } + + self.upper.rebuild_free(); + + if let Some(lower) = &mut self.lower { + lower.rebuild_free(); + } + + Ok(()) + } + fn allocation_len(&self, addr: u64) -> Result { if let Some(lower) = &self.lower && lower.contains(addr) @@ -343,6 +397,11 @@ impl Inner { fn layouts(&self) -> (Option, SlotLayout) { (self.lower.as_ref().map(Tier::layout), self.upper.layout()) } + + /// Byte span of both tiers, including any gap. + fn byte_len(&self) -> usize { + (self.upper.layout().end_addr() - self.base_addr()) as usize + } } /// A buffer pool with one or two fixed-slot tiers. @@ -356,6 +415,25 @@ pub struct SlotPool { } impl SlotPool { + /// Generation owning the current scratch allocations. + pub(crate) fn generation(&self) -> u64 { + self.inner.borrow().generation + } + + /// Retire existing leases and rebuild the pool around the supplied reservations. + /// + /// # Safety + /// + /// Retired views must already use backing independent of these scratch slots. + /// `reserved` must name every other live allocation exactly once. + /// No allocation or peer access may race this call. + pub(crate) unsafe fn retire_generation( + &self, + reserved: impl Iterator + Clone, + ) -> Result<(), AllocError> { + self.inner.borrow_mut().retire_generation(reserved) + } + /// Create a single-tier recycling pool from exact slot placement. pub fn new(layout: SlotLayout) -> Result { Self::from_layouts(None, layout) @@ -382,17 +460,6 @@ impl SlotPool { self.inner.borrow().live_addrs() } - /// Visit every free slot in lower-then-upper index order. - /// - /// The callback must not allocate or free slots in this pool. - pub fn for_each_free(&self, mut f: impl FnMut(Allocation)) { - let inner = self.inner.borrow(); - if let Some(lower) = &inner.lower { - lower.for_each_free(&mut f); - } - inner.upper.for_each_free(&mut f); - } - /// Return the lower and upper tier layouts. pub fn layouts(&self) -> (Option, SlotLayout) { self.inner.borrow().layouts() @@ -469,6 +536,11 @@ impl SlotPool { self.inner.borrow_mut().alloc(len) } + /// Byte span from the first slot to the last, including any gap between tiers. + pub fn byte_len(&self) -> usize { + self.inner.borrow().byte_len() + } + #[cfg(test)] pub(crate) fn strong_count(&self) -> usize { Rc::strong_count(&self.inner) diff --git a/src/hyperlight_common/src/virtq/pool/tests.rs b/src/hyperlight_common/src/virtq/pool/tests.rs index b47e3904a..2dba194e1 100644 --- a/src/hyperlight_common/src/virtq/pool/tests.rs +++ b/src/hyperlight_common/src/virtq/pool/tests.rs @@ -26,6 +26,7 @@ fn test_slot_pool_preserves_exact_base() { let pool = SlotPool::new(layout).unwrap(); assert_eq!(pool.base_addr(), 0x80001); + assert_eq!(pool.byte_len(), 8192); assert_eq!(pool.count(), 2); assert_eq!(pool.slot_addr(0), Some(0x80001)); assert_eq!(pool.slot_addr(1), Some(0x81001)); @@ -41,6 +42,7 @@ fn test_tiered_slot_pool_reports_layouts() { assert_eq!(lower, Some(SlotLayout::new(0x80001, 0x100, 2).unwrap())); assert_eq!(upper, SlotLayout::new(0x90001, 0x1000, 2).unwrap()); assert_eq!(pool.base_addr(), 0x80001); + assert_eq!(pool.byte_len(), 0x12000); assert_eq!(pool.slot_size(), 0x1000); assert_eq!(pool.count(), 4); assert_eq!(pool.slot_addr(0), Some(0x80001)); @@ -61,6 +63,7 @@ fn test_tiered_slot_pool_combines_contiguous_equal_sized_layouts() { (None, SlotLayout::new(0x80000, 0x100, 5).unwrap()) ); assert_eq!(pool.base_addr(), 0x80000); + assert_eq!(pool.byte_len(), 0x500); assert_eq!(pool.slot_size(), 0x100); assert_eq!(pool.count(), 5); assert_eq!(pool.num_free_lower(), 0); @@ -132,6 +135,9 @@ fn test_slot_layout_rejects_overflowing_ranges() { let layout = SlotLayout::new(u64::MAX - 8, 4, 2).unwrap(); assert_eq!(layout.end_addr(), u64::MAX); + + let pool = SlotPool::new(layout).unwrap(); + assert_eq!(pool.byte_len(), 8); } #[test] @@ -246,34 +252,6 @@ fn test_tiered_slot_pool_live_addrs_are_deterministic() { ); } -#[test] -fn free_slots_include_full_capacities_and_preserve_allocation_order() { - let pool = make_tiered_slot_pool(2, 2); - let lower = pool.alloc(128).unwrap(); - let upper = pool.alloc(1024).unwrap(); - let mut free = Vec::new(); - pool.for_each_free(|allocation| free.push((allocation.addr, allocation.len))); - assert_eq!(free, [(0x80000, 256), (0x90000, 4096)]); - assert_eq!(pool.live_addrs(), [lower.addr, upper.addr]); - - pool.dealloc(lower.addr).unwrap(); - let repeated = pool.alloc(128).unwrap(); - assert_eq!(repeated.addr, lower.addr); - pool.dealloc(repeated.addr).unwrap(); - pool.dealloc(upper.addr).unwrap(); - free.clear(); - pool.for_each_free(|allocation| free.push((allocation.addr, allocation.len))); - assert_eq!( - free, - [ - (0x80000, 256), - (0x80100, 256), - (0x90000, 4096), - (0x91000, 4096) - ] - ); -} - #[test] fn test_slot_pool_dealloc_out_of_range() { let pool = make_slot_pool(4, 4096); diff --git a/src/hyperlight_common/src/virtq/producer.rs b/src/hyperlight_common/src/virtq/producer.rs index 0bddd4b1c..403178811 100644 --- a/src/hyperlight_common/src/virtq/producer.rs +++ b/src/hyperlight_common/src/virtq/producer.rs @@ -240,6 +240,30 @@ where self.inner.mem() } + /// Reclaim retained scratch slots while preserving in-flight chains. + /// + /// # Safety + /// + /// Every live allocation outside this producer's in-flight chains must + /// belong to a returned view with backing independent of reused scratch. + /// No unsent chains or builders may remain. Peer accesses must be stopped. + pub unsafe fn reclaim_slots(&mut self) -> Result<(), VirtqError> { + if !self.pending.is_empty() { + return Err(VirtqError::InvalidState); + } + + let reserved = self + .inflight + .live + .iter() + .flat_map(|inflight| inflight.chain.buffers.iter().map(|buffer| buffer.addr)); + + // SAFETY: In-flight records preserve all non-retired allocations. + // The caller guarantees independent backing and excludes untracked chains. + unsafe { self.pool.retire_generation(reserved)? }; + Ok(()) + } + /// Begin building a descriptor chain for submission. /// /// The builder captures the current free-descriptor budget. diff --git a/src/hyperlight_common/src/virtq/ring/canonical.rs b/src/hyperlight_common/src/virtq/ring/canonical.rs index cb56f8da3..96e3cee7b 100644 --- a/src/hyperlight_common/src/virtq/ring/canonical.rs +++ b/src/hyperlight_common/src/virtq/ring/canonical.rs @@ -61,12 +61,12 @@ pub enum ImageError { /// Reading the shared ring image failed. #[error(transparent)] Ring(#[from] RingError), - /// The caller supplied an impossible available-descriptor prefix length. - #[error("available descriptor count {available} exceeds ring capacity {capacity}")] + /// The available-descriptor prefix exceeds its allowed limit. + #[error("available descriptor count {available} exceeds limit {capacity}")] DescCount { - /// Number of descriptors expected to be available. + /// Number of descriptors supplied or discovered as available. available: usize, - /// Descriptor-table capacity. + /// Maximum allowed number of available descriptors. capacity: usize, }, /// An event-suppression structure is not the canonical enabled value. @@ -260,6 +260,51 @@ where Ok(chains) } +/// Validate a canonical image with at most `max_avail_descs` available descriptors. +/// +/// The first zero descriptor ends the prefix. [`validate_canon_image`] then +/// checks the prefix and zero tail. Keep the image unchanged during both passes. +/// +/// # Errors +/// +/// Returns [`ImageError`] if the prefix exceeds the limit, a descriptor cannot +/// be read, or [`validate_canon_image`] rejects the image or its buffers. +pub fn validate_canon_prefix( + mem: &M, + layout: Layout, + max_avail_descs: usize, + validate_buf: F, +) -> Result, ImageError> +where + M: MemOps, + F: FnMut(u16, BufferElement) -> bool, +{ + let cap = layout.desc_table_len() as usize; + // SAFETY: `Layout` validates the table base, alignment, and descriptor count. + let table = unsafe { DescTable::from_raw_parts(layout.desc_table_addr(), cap) }; + let empty = Descriptor::zeroed(); + let mut avail_descs = cap; + + for pos in 0..cap { + let idx = u16::try_from(pos).map_err(|_| RingError::InvalidState)?; + let addr = table.desc_addr(idx).ok_or(RingError::InvalidState)?; + let desc = mem + .read_val::(addr) + .map_err(|_| RingError::mem_err(MemOp::ReadDesc, addr))?; + + if desc == empty { + avail_descs = pos; + break; + } + } + + if avail_descs > max_avail_descs { + return Err(ImageError::desc_count(avail_descs, max_avail_descs)); + } + + validate_canon_image(mem, layout, avail_descs, validate_buf) +} + fn read_canon_avail_desc( mem: &M, table: &DescTable, @@ -595,4 +640,44 @@ mod tests { }) )); } + + #[test] + fn canon_prefix_rejects_incomplete_chains_and_gaps() { + let ring = make_ring(4); + let mut producer = make_producer(&ring); + producer + .submit_available(&writable_chain(0x1000, &[64, 64])) + .unwrap(); + + ring.write_desc(1, Descriptor::zeroed()); + let err = validate_canon_prefix(&ring.mem(), ring.layout(), ring.len(), |_, _| true); + + assert!(matches!( + err, + Err(ImageError::Desc { + index: 0, + reason: DescError::ChainContinues, + }) + )); + + for avail in 0..=1 { + let ring = make_ring(4); + let mut producer = make_producer(&ring); + + for _ in 0..avail { + producer.submit_one(0x1000, 64, true).unwrap(); + } + + ring.write_desc(3, Descriptor::new(0x2000, 64, 0, DescFlags::empty())); + let err = validate_canon_prefix(&ring.mem(), ring.layout(), ring.len(), |_, _| true); + + assert!(matches!( + err, + Err(ImageError::Desc { + index: 3, + reason: DescError::ExpectedZero, + }) + )); + } + } } diff --git a/src/hyperlight_guest/Cargo.toml b/src/hyperlight_guest/Cargo.toml index 08becd29a..95295ca3c 100644 --- a/src/hyperlight_guest/Cargo.toml +++ b/src/hyperlight_guest/Cargo.toml @@ -17,6 +17,7 @@ serde_json = { version = "1.0", default-features = false, features = ["alloc"] } hyperlight-common = { workspace = true, default-features = false } flatbuffers = { version= "25.12.19", default-features = false } tracing = { version = "0.1.44", default-features = false, features = ["attributes"] } +itertools = { version = "0.15.0", default-features = false } [target.'cfg(target_arch = "x86_64")'.dependencies] hyperlight-guest-tracing = { workspace = true, default-features = false, optional = true } diff --git a/src/hyperlight_guest/src/arch/aarch64/paging.rs b/src/hyperlight_guest/src/arch/aarch64/paging.rs index f06e6ad97..b8d1dd60b 100644 --- a/src/hyperlight_guest/src/arch/aarch64/paging.rs +++ b/src/hyperlight_guest/src/arch/aarch64/paging.rs @@ -129,8 +129,15 @@ pub unsafe fn modify_mapping( } } +/// Walk the page containing `gva`. pub fn virt_to_phys(gva: vmem::VirtAddr) -> impl Iterator { - unsafe { vmem::virt_to_phys::<_>(GuestMappingOperations::new(), gva, 1) } + virt_to_phys_range(gva, 1) +} + +/// Walk page mappings intersecting the range, skipping unmapped pages. +pub fn virt_to_phys_range(gva: vmem::VirtAddr, len: u64) -> impl Iterator { + // SAFETY: The operations read valid page tables in the current guest context. + unsafe { vmem::virt_to_phys::<_>(GuestMappingOperations::new(), gva, len) } } pub fn phys_to_virt(gpa: vmem::PhysAddr) -> Option<*mut u8> { diff --git a/src/hyperlight_guest/src/arch/amd64/paging.rs b/src/hyperlight_guest/src/arch/amd64/paging.rs index a2bed7a4c..4c95e9793 100644 --- a/src/hyperlight_guest/src/arch/amd64/paging.rs +++ b/src/hyperlight_guest/src/arch/amd64/paging.rs @@ -144,8 +144,15 @@ pub unsafe fn modify_mapping( } } +/// Walk the page containing `gva`. pub fn virt_to_phys(gva: vmem::VirtAddr) -> impl Iterator { - unsafe { vmem::virt_to_phys::<_>(GuestMappingOperations::new(), gva, 1) } + virt_to_phys_range(gva, 1) +} + +/// Walk page mappings intersecting the range, skipping unmapped pages. +pub fn virt_to_phys_range(gva: vmem::VirtAddr, len: u64) -> impl Iterator { + // SAFETY: The operations read valid page tables in the current guest context. + unsafe { vmem::virt_to_phys::<_>(GuestMappingOperations::new(), gva, len) } } pub fn phys_to_virt(gpa: vmem::PhysAddr) -> Option<*mut u8> { diff --git a/src/hyperlight_guest/src/paging.rs b/src/hyperlight_guest/src/paging.rs index 5c27a4c8d..067411360 100644 --- a/src/hyperlight_guest/src/paging.rs +++ b/src/hyperlight_guest/src/paging.rs @@ -7,7 +7,7 @@ #[cfg_attr(target_arch = "aarch64", path = "arch/aarch64/paging.rs")] mod arch; -pub use arch::{map_region, modify_mapping, phys_to_virt, virt_to_phys}; +pub use arch::{map_region, modify_mapping, phys_to_virt, virt_to_phys, virt_to_phys_range}; /// Barriers that other code may need to use when updating page tables pub mod barrier { /// Call this function when a virtual address has had its diff --git a/src/hyperlight_guest/src/transport/backing.rs b/src/hyperlight_guest/src/transport/backing.rs new file mode 100644 index 000000000..a6c7d5d4b --- /dev/null +++ b/src/hyperlight_guest/src/transport/backing.rs @@ -0,0 +1,342 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright 2026 The Hyperlight Authors. + +//! Reusable virtual ranges for owner-backed transport aliases. + +use alloc::vec::Vec; +use core::cell::RefCell; +use core::ops::Range; + +use hyperlight_common::layout::{VIRTQ_BUFFER_GVA_END, VIRTQ_BUFFER_GVA_START}; +use hyperlight_common::virtq::AllocError; +use hyperlight_common::vmem::{BasicMapping, Mapping, MappingKind, PAGE_SIZE}; +use itertools::Itertools; + +use super::SyncWrap; +use crate::paging; + +/// Reservations are captured with their aliases and survive pool generations. +static ALLOCATOR: SyncWrap> = SyncWrap(RefCell::new(AliasAllocator::new())); +// Permissions for transport alias mappings. +const PERMISSIONS: BasicMapping = BasicMapping { + readable: true, + writable: true, + executable: false, +}; + +/// First-fit free ranges followed by a page-aligned bump frontier. +struct AliasAllocator { + /// Sorted, disjoint free ranges with adjacent ranges coalesced. + free: Vec>, + next: u64, +} + +impl AliasAllocator { + /// Reserve addresses in the dedicated transport alias arena. + const fn new() -> Self { + Self { + free: Vec::new(), + next: VIRTQ_BUFFER_GVA_START, + } + } + + /// Reuse freed addresses before extending the page-table footprint. + fn alloc(&mut self, len: u64) -> Result, AllocError> { + if len == 0 || !len.is_multiple_of(PAGE_SIZE as u64) { + return Err(AllocError::InvalidArg); + } + + if let Some(index) = self + .free + .iter() + .position(|range| range.end - range.start >= len) + { + let start = self.free[index].start; + let end = start + len; + self.free[index].start = end; + + if end == self.free[index].end { + self.free.remove(index); + } + + return Ok(start..end); + } + + let end = self.next.checked_add(len).ok_or(AllocError::Overflow)?; + + if end > VIRTQ_BUFFER_GVA_END { + return Err(AllocError::NoSpace); + } + + let range = self.next..end; + self.next = end; + Ok(range) + } + + /// Validate an owned, unmapped range and merge it with adjacent free ranges. + fn dealloc(&mut self, range: Range) { + assert!(range.start >= VIRTQ_BUFFER_GVA_START && range.end <= self.next); + assert!(range.start < range.end); + assert!(range.start.is_multiple_of(PAGE_SIZE as u64)); + assert!(range.end.is_multiple_of(PAGE_SIZE as u64)); + + // Only the insertion point's neighbors can overlap the returned range. + let index = self.free.partition_point(|free| free.start < range.start); + assert!(index == 0 || self.free[index - 1].end <= range.start); + assert!(index == self.free.len() || range.end <= self.free[index].start); + + let join_prev = index > 0 && self.free[index - 1].end == range.start; + let join_next = index < self.free.len() && range.end == self.free[index].start; + + match (join_prev, join_next) { + (true, true) => self.free[index - 1].end = self.free.remove(index).end, + (true, false) => self.free[index - 1].end = range.end, + (false, true) => self.free[index].start = range.start, + (false, false) => self.free.insert(index, range), + } + } +} + +/// Require complete source coverage by readable, writable basic pages. +fn validate_source_pages( + source: Range, + pages: impl Iterator, +) -> Result<(), AllocError> { + let mut next_source = source.start; + + for page in pages { + let writable = matches!( + page.kind, + MappingKind::Basic(perm) if perm.readable && perm.writable + ); + + if !writable + || page.virt_base != next_source + || page.len != PAGE_SIZE as u64 + || page.len > source.end.saturating_sub(next_source) + { + return Err(AllocError::InvalidArg); + } + + next_source += page.len; + } + + if next_source != source.end { + return Err(AllocError::InvalidArg); + } + + Ok(()) +} + +/// Yield coalesced aliases for the same pages checked by [`validate_source_pages`]. +fn coalesce_mappings( + source: u64, + start: u64, + pages: impl Iterator, +) -> impl Iterator { + let runs = pages.coalesce(|mut run, page| { + if run.phys_base.checked_add(run.len) != Some(page.phys_base) { + return Err((run, page)); + } + + run.len += page.len; + Ok(run) + }); + + runs.map(move |run| Mapping { + virt_base: start + (run.virt_base - source), + kind: MappingKind::Basic(PERMISSIONS), + ..run + }) +} + +/// Update owned alias entries and invalidate cleared translations. +/// +/// # Safety +/// +/// The caller must own the alias range and exclude all views during updates. +/// Mapped backing must remain live for later views. +/// Serialize paging and share alias leaves across accessing roots. +unsafe fn update_alias(mapping: Mapping) { + // SAFETY: The caller owns these entries and provides valid backing. + unsafe { + paging::map_region( + mapping.phys_base, + mapping.virt_base as *mut u8, + mapping.len, + mapping.kind, + ); + } + + if mapping.kind == MappingKind::Unmapped { + paging::barrier::downgrade_in_place(mapping.virt_base..mapping.virt_base + mapping.len); + } +} + +/// Reserve an alias and map it to initialized scratch pages. +/// +/// # Safety +/// +/// The scratch range must be page-aligned, live, and initialized. +pub(crate) unsafe fn map(scratch: u64, len: u64) -> Result, AllocError> { + if !scratch.is_multiple_of(PAGE_SIZE as u64) { + return Err(AllocError::InvalidArg); + } + + let mut allocator = ALLOCATOR.0.borrow_mut(); + let alias = allocator.alloc(len)?; + let end = scratch.checked_add(len).ok_or(AllocError::Overflow)?; + + let pages = paging::virt_to_phys_range(scratch, len); + let res = validate_source_pages(scratch..end, pages); + + if let Err(error) = res { + allocator.dealloc(alias); + return Err(error); + } + + let pages = paging::virt_to_phys_range(scratch, len); + let start = alias.start; + + for mapping in coalesce_mappings(scratch, start, pages) { + // SAFETY: The reservation is unused and source pages have valid backing. + unsafe { update_alias(mapping) }; + } + + paging::barrier::first_valid_same_ctx(); + Ok(alias) +} + +/// Unmap an owned alias and return its virtual range for reuse. +/// +/// # Safety +/// +/// The caller must own a range returned by [`map`] and exclude all remaining +/// views. Run on the serialized guest vCPU. +/// Serialize paging and share alias leaves across accessing roots. +pub(crate) unsafe fn unmap(alias: &Range) { + // SAFETY: The caller exclusively owns these leaves and has released views. + unsafe { + update_alias(Mapping { + phys_base: 0, + virt_base: alias.start, + len: alias.end - alias.start, + kind: MappingKind::Unmapped, + }); + } + + ALLOCATOR.0.borrow_mut().dealloc(alias.clone()); +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn alias_walk_coalesces_contiguous_physical_pages() { + let page = PAGE_SIZE as u64; + let source = 0x10_0000; + let alias = VIRTQ_BUFFER_GVA_START; + + let kind = MappingKind::Basic(PERMISSIONS); + let first = Mapping { + phys_base: page, + virt_base: source, + len: page, + kind, + }; + + let pages = [ + first, + Mapping { + phys_base: 2 * page, + virt_base: source + page, + ..first + }, + Mapping { + phys_base: 8 * page, + virt_base: source + 2 * page, + ..first + }, + ]; + + validate_source_pages(source..source + 3 * page, pages.iter().copied()).unwrap(); + + let updates: Vec<_> = coalesce_mappings(source, alias, pages.iter().copied()) + .map(|m| (m.virt_base, m.phys_base, m.len, m.kind)) + .collect(); + + assert_eq!( + updates, + [ + (alias, page, 2 * page, kind), + (alias + 2 * page, 8 * page, page, kind), + ] + ); + } + + #[test] + fn aliases_reuse_and_coalesce_freed_ranges() { + let mut allocator = AliasAllocator::new(); + let page = PAGE_SIZE as u64; + let first = allocator.alloc(page).unwrap(); + let middle = allocator.alloc(2 * page).unwrap(); + let last = allocator.alloc(page).unwrap(); + let frontier = allocator.next; + + allocator.dealloc(middle.clone()); + let reused = allocator.alloc(page).unwrap(); + + assert_eq!(reused, middle.start..middle.start + page); + + allocator.dealloc(first.clone()); + allocator.dealloc(last.clone()); + allocator.dealloc(reused); + + assert_eq!(allocator.alloc(4 * page).unwrap(), first.start..last.end); + assert_eq!(allocator.next, frontier); + } + + #[test] + fn exhausted_alias_space_still_reuses_freed_ranges() { + let mut allocator = AliasAllocator::new(); + let whole = allocator + .alloc(VIRTQ_BUFFER_GVA_END - VIRTQ_BUFFER_GVA_START) + .unwrap(); + + let err = allocator.alloc(PAGE_SIZE as u64); + assert!(matches!(err, Err(AllocError::NoSpace))); + + allocator.dealloc(whole.clone()); + + let reused = allocator.alloc(PAGE_SIZE as u64).unwrap(); + assert_eq!(reused.start, whole.start); + } + + /// Failed reservations preserve both reusable ranges and the frontier. + #[test] + fn invalid_alias_lengths_preserve_allocator_state() { + let mut allocator = AliasAllocator::new(); + let page = PAGE_SIZE as u64; + let first = allocator.alloc(page).unwrap(); + let _retained = allocator.alloc(page).unwrap(); + + allocator.dealloc(first.clone()); + let free = allocator.free.clone(); + let frontier = allocator.next; + + for len in [0, page - 1, page + 1] { + let err = allocator.alloc(len); + assert!(matches!(err, Err(AllocError::InvalidArg))); + } + + let ov = allocator.alloc(u64::MAX / page * page); + assert!(matches!(ov, Err(AllocError::Overflow))); + + let ns = allocator.alloc(VIRTQ_BUFFER_GVA_END - VIRTQ_BUFFER_GVA_START); + assert!(matches!(ns, Err(AllocError::NoSpace))); + assert_eq!(allocator.free, free); + assert_eq!(allocator.next, frontier); + assert_eq!(allocator.alloc(page).unwrap(), first); + } +} diff --git a/src/hyperlight_guest/src/transport/context.rs b/src/hyperlight_guest/src/transport/context.rs index 950a12625..f9219d2d7 100644 --- a/src/hyperlight_guest/src/transport/context.rs +++ b/src/hyperlight_guest/src/transport/context.rs @@ -12,17 +12,21 @@ use hyperlight_common::flatbuffer_wrappers::function_types::{ FunctionCallResult, ParameterValue, ReturnType, ReturnValue, }; use hyperlight_common::flatbuffer_wrappers::util::estimate_flatbuffer_capacity; +use hyperlight_common::layout::SCRATCH_TOP_GVA; use hyperlight_common::outb::OutBAction; -use hyperlight_common::transport::{EncodedMessage, ExternalValues, MsgHeader, MsgKind}; +use hyperlight_common::transport::{ + EncodedMessage, ExternalValues, MailboxValue, MsgHeader, MsgKind, +}; use hyperlight_common::virtq::{ - AllocError, G2H_LOWER_SLOT_COUNT, G2H_LOWER_SLOT_SIZE, Layout, MemOps, Notifier, QueueStats, - SendChain, SlotLayout, SlotPool, Token, UsedChain, VirtqError, VirtqProducer, + AllocError, G2H_LOWER_SLOT_COUNT, G2H_LOWER_SLOT_SIZE, Layout, Notifier, QueueStats, SendChain, + SlotLayout, SlotPool, Token, UsedChain, VirtqError, VirtqProducer, }; +use hyperlight_common::vmem::PAGE_SIZE; use super::{GuestMemOps, codec}; -use crate::bail; use crate::error::{GuestErrorContext, Result}; use crate::exit::out32; +use crate::{bail, layout}; /// Exits to the host to process available virtqueue work. #[derive(Clone, Copy)] @@ -97,9 +101,11 @@ pub struct GuestContext { g2h_producer: G2hProducer, /// Host-to-guest driver. h2g_producer: H2gProducer, + /// Snapshot lineage whose scratch slots belong to the current pool generation. + generation: u64, /// Size of each prefilled H2G buffer. h2g_slot_size: usize, - /// Snapshot checkpoint mailbox GVA. + /// Transport mailbox GVA. mbx_gva: u64, /// Correlation ID assigned to the next host-function request. next_cid: u32, @@ -116,20 +122,28 @@ impl GuestContext { /// and pools. Scratch must remain mapped while the context or any returned /// buffer views exist. pub unsafe fn new(g2h: QueueConfig, h2g: QueueConfig, mbx_gva: u64) -> Result { + // SAFETY: Generation has been initialized by host. + let generation = unsafe { layout::snapshot_generation_gva().read_volatile() }; + let g2h_pool = g2h_pool(g2h.pool_gva, g2h.pool_pages, g2h.buffer_size) .with_context(|| "failed to create G2H pool")?; - // SAFETY: The caller supplies the guest execution and scratch lifetime requirements. - let mem = unsafe { GuestMemOps::for_scratch() }; + // SAFETY: The caller owns the pool's scratch pages. + let mem = unsafe { pool_mem(&g2h_pool) } + .with_context(|| "failed to initialize G2H pool access")?; let g2h_producer = VirtqProducer::new(g2h.layout, mem, OutbNotifier, g2h_pool); let h2g_pool = h2g_pool(h2g.pool_gva, h2g.pool_pages, h2g.buffer_size) .with_context(|| "failed to create H2G slot pool")?; + // SAFETY: The caller owns this disjoint pool. + let mem = unsafe { pool_mem(&h2g_pool) } + .with_context(|| "failed to initialize H2G pool access")?; // H2G prefill supplies writable buffers for host-initiated messages. let h2g_producer = VirtqProducer::new(h2g.layout, mem, NoopNotifier, h2g_pool); let mut ctx = Self { g2h_producer, h2g_producer, + generation, h2g_slot_size: h2g.buffer_size, mbx_gva, next_cid: 1, @@ -320,20 +334,20 @@ impl GuestContext { self.prefill_h2g() } - /// Canonicalize both queues and publish the retained allocation count. + /// Reset stopped queues while retained values keep their pool slots. /// - /// # Safety + /// The mailbox is complete only after free H2G slots are prefilled. /// - /// All host consumer chain handles must be dropped. Both consumers must - /// stay stopped until they are reset or replaced. + /// # Safety /// - /// ```compile_fail,E0133 - /// # use hyperlight_guest::transport::GuestContext; - /// fn checkpoint(context: &mut GuestContext) { - /// context.prepare_snapshot().unwrap(); - /// } - /// ``` + /// All host consumer chain handles must be dropped. + /// Both consumers must stay stopped until they are reset or replaced. pub unsafe fn prepare_snapshot(&mut self) -> Result<()> { + let pending = MailboxValue::CheckpointPending.raw(); + // SAFETY: The host has initialized the mailbox. + unsafe { (self.mbx_gva as *mut u64).write_volatile(pending) }; + + // Read all acks from host self.g2h_producer .reclaim() .with_context(|| "G2H snapshot reclaim failed")?; @@ -348,22 +362,40 @@ impl GuestContext { .with_context(|| "H2G snapshot reset failed")?; } - // Only guest-retained allocations remain between reset and H2G prefill. - let guest_owned = self - .g2h_producer - .pool() - .num_live() - .checked_add(self.h2g_producer.pool().num_live()) - .ok_or(VirtqError::InvalidState)?; - let guest_owned = u64::try_from(guest_owned).map_err(|_| VirtqError::InvalidState)?; + self.prefill_h2g()?; - // Retained snapshots will publish readiness after sanitization and H2G prefill. - self.g2h_producer - .memory() - .write(self.mbx_gva, &guest_owned.to_le_bytes()) - .map_err(|_| VirtqError::MemoryWriteError)?; + let compl = MailboxValue::CheckpointComplete.raw(); + // SAFETY: The host has initialized the writable mailbox. + unsafe { (self.mbx_gva as *mut u64).write_volatile(compl) }; - self.prefill_h2g() + Ok(()) + } + + /// Reclaim retained scratch slots after restore without changing the rings. + /// + /// Retained owners keep their captured aliases. Posted chains keep their + /// scratch reservations, including a request written before guest entry. + pub fn maybe_refresh(&mut self) { + // SAFETY: Construction requires initialized guest scratch metadata. + let generation = unsafe { layout::snapshot_generation_gva().read_volatile() }; + + if self.generation == generation { + return; + } + + // SAFETY: A changed generation denotes restore with independent captured + // backing. Canonical checkpoints leave only posted chains as reservations. + // Guest entry excludes peer access. + unsafe { + self.g2h_producer + .reclaim_slots() + .expect("G2H pool recovery failed"); + self.h2g_producer + .reclaim_slots() + .expect("H2G pool recovery failed"); + } + + self.generation = generation; } /// Send a log message via the G2H queue. @@ -380,9 +412,6 @@ impl GuestContext { } /// Publish one writable H2G chain for each currently free slot. - /// - /// Retained external values reduce the number of available receive buffers - /// until their final owner drops. fn prefill_h2g(&mut self) -> Result<()> { let mut batch = self.h2g_producer.batch(); @@ -520,6 +549,26 @@ fn pool_len(pages: usize) -> result::Result { .ok_or(AllocError::Overflow) } +/// Validate pool placement and bind scratch access. +/// +/// # Safety +/// +/// Run on the initialized guest vCPU and own the pool's scratch. +unsafe fn pool_mem(pool: &SlotPool) -> result::Result { + let scratch_gva = layout::scratch_base_gva(); + let scratch_end = SCRATCH_TOP_GVA as u64 + 1; + + let base = pool.base_addr(); + let end = base + pool.byte_len() as u64; + + if !base.is_multiple_of(PAGE_SIZE as u64) || base < scratch_gva || end > scratch_end { + return Err(AllocError::InvalidArg); + } + + // SAFETY: The validated pool lies in live scratch on the serialized guest vCPU. + unsafe { Ok(GuestMemOps::new(scratch_gva, scratch_end)) } +} + /// Build the uniform H2G pool. /// /// Each slot becomes one independent preposted receive buffer. diff --git a/src/hyperlight_guest/src/transport/mem.rs b/src/hyperlight_guest/src/transport/mem.rs index 3196a8508..9d4deba61 100644 --- a/src/hyperlight_guest/src/transport/mem.rs +++ b/src/hyperlight_guest/src/transport/mem.rs @@ -3,24 +3,28 @@ //! Guest-side [`MemOps`] implementation for virtqueue access. -use core::mem::{ManuallyDrop, align_of, size_of}; +use core::marker::PhantomData; +use core::mem::{align_of, size_of}; +use core::ops::Range; use core::ptr::NonNull; use core::sync::atomic::{AtomicU16, Ordering}; use hyperlight_common::virtq::{BufferLease, BufferMap, MemOps}; +use hyperlight_common::vmem::PAGE_SIZE; -use crate::layout; +use super::backing; -#[cfg(test)] -extern crate std; +// Mark types not [`Send`] +type NotThreadSafe = PhantomData<*mut ()>; -/// Fixed-pool memory access within an initialized guest runtime. +/// Bounded scratch memory access. /// /// Copies reject buffers that overlap the accessed scratch range. #[derive(Clone, Copy, Debug)] pub(crate) struct GuestMemOps { scratch_gva: u64, scratch_end: u64, + _nts: NotThreadSafe, } /// Invalid guest virtqueue memory access. @@ -28,33 +32,18 @@ pub(crate) struct GuestMemOps { pub struct GuestMemError; impl GuestMemOps { - /// # Safety - /// - /// Guest scratch metadata must be initialized. Execution must stay within - /// the single-vCPU guest. Scratch must remain mapped while this accessor - /// or any returned views exist. - pub(super) unsafe fn for_scratch() -> Self { - // SAFETY: The caller guarantees initialized guest scratch metadata. - let scratch_len = unsafe { layout::scratch_size_gva().read_volatile() }; - // SAFETY: The caller supplies the guest execution and backing lifetime guarantees. - unsafe { Self::from_raw_parts(layout::scratch_base_gva(), scratch_len) } - } - - /// Create an accessor for a scratch virtual address range. + /// Bind memory access to live scratch bounds. /// /// # Safety /// - /// The range must remain mapped while this accessor or its returned views - /// exist. Peer access must follow descriptor ownership. Production callers - /// must uphold the guest execution requirements of [`Self::for_scratch`]. - unsafe fn from_raw_parts(scratch_gva: u64, scratch_len: u64) -> Self { - let scratch_end = scratch_gva - .checked_add(scratch_len) - .expect("scratch end overflow"); - + /// Scratch must be live, initialized, and writable. + /// All access and destruction must stay on the serialized guest vCPU. + /// Serialize paging and share alias leaf entries across accessing roots. + pub(super) unsafe fn new(scratch_gva: u64, scratch_end: u64) -> Self { Self { scratch_gva, scratch_end, + _nts: PhantomData, } } @@ -84,53 +73,52 @@ impl BufferMap for GuestMemOps { lease: BufferLease, written: usize, ) -> Result { - // SAFETY: The caller owns the initialized prefix. The accessor's - // constructor guarantees backing remains mapped while returned views exist. - let data = NonNull::from(unsafe { self.as_slice(lease.allocation().addr, written)? }); + let alloc = lease.allocation(); + self.ptr(alloc.addr, alloc.len as usize)?; + + let off = alloc.addr % PAGE_SIZE as u64; + let len = (off + written.max(1) as u64).div_ceil(PAGE_SIZE as u64) * PAGE_SIZE as u64; + + // SAFETY: The lease owns initialized bytes in live scratch pages. + // Guest entry serializes paging. + let alias = unsafe { backing::map(alloc.addr - off, len) }.map_err(|_| GuestMemError)?; + // SAFETY: The reserved arena is nonnull and the mapping is live. + let data = unsafe { NonNull::new_unchecked((alias.start + off) as *mut u8) }; + Ok(GuestMapping { - data, - lease: ManuallyDrop::new(lease), - #[cfg(test)] - creator: std::thread::current().id(), + data: NonNull::slice_from_raw_parts(data, written), + alias, + _lease: lease, }) } } -/// An initialized scratch view held by its original allocation lease. +/// An alias view backed by its allocation lease. pub(crate) struct GuestMapping { data: NonNull<[u8]>, - lease: ManuallyDrop, - #[cfg(test)] - creator: std::thread::ThreadId, + alias: Range, + _lease: BufferLease, +} + +impl Drop for GuestMapping { + fn drop(&mut self) { + // SAFETY: The final Bytes owner has released every view. This owner + // exclusively reserves its alias pages on the serialized guest vCPU. + unsafe { backing::unmap(&self.alias) }; + } } -// SAFETY: The view is immutable. Production construction requires serialized -// guest execution and guest-lifetime backing. Native tests check the creator -// thread before releasing the Rc-backed lease. +// SAFETY: Construction requires access and destruction on the serialized guest +// vCPU. The lease keeps the alias backed for this owner's lifetime. unsafe impl Send for GuestMapping {} impl AsRef<[u8]> for GuestMapping { fn as_ref(&self) -> &[u8] { - // SAFETY: The lease protects the initialized view from reuse. + // SAFETY: The lease keeps this initialized alias range mapped. unsafe { self.data.as_ref() } } } -impl Drop for GuestMapping { - fn drop(&mut self) { - #[cfg(test)] - assert_eq!( - self.creator, - std::thread::current().id(), - "guest mapping dropped on another thread" - ); - - // SAFETY: The view is no longer exposed. Guest execution, or the native - // creator-thread check, serializes release of the Rc-backed lease. - unsafe { ManuallyDrop::drop(&mut self.lease) }; - } -} - // SAFETY: Every address is restricted to the scratch mapping. Payload // references rely on descriptor ownership, and ring flags use aligned atomics. unsafe impl MemOps for GuestMemOps { @@ -196,7 +184,6 @@ mod tests { use alloc::vec; use core::mem::size_of; - use hyperlight_common::flatbuffer_wrappers::function_types::Bytes; use hyperlight_common::virtq::{MemOps, SlotLayout, SlotPool}; use super::*; @@ -206,7 +193,8 @@ mod tests { const LEN: usize = 0x4000; let mut backing = vec![0u64; LEN / size_of::()]; let base = backing.as_mut_ptr() as usize as u64; - let mem = unsafe { GuestMemOps::from_raw_parts(base, LEN as u64) }; + // SAFETY: Backing stays initialized and mapped. No peer accesses it. + let mem = unsafe { GuestMemOps::new(base, base + LEN as u64) }; mem.write(base, &[1, 2, 3, 4]).unwrap(); let mut bytes = [0; 4]; @@ -225,7 +213,7 @@ mod tests { let mut backing = [1u8, 2, 3, 4]; let base = backing.as_mut_ptr() as u64; // SAFETY: Backing stays initialized and mapped. No peer accesses it. - let mem = unsafe { GuestMemOps::from_raw_parts(base, 4) }; + let mem = unsafe { GuestMemOps::new(base, base + 4) }; assert_eq!(mem.read(base, &mut backing), Err(GuestMemError)); assert_eq!(mem.read(base, &mut backing[1..]), Err(GuestMemError)); @@ -238,7 +226,7 @@ mod tests { let mut backing = [1u8, 2, 3, 4]; let base = backing.as_mut_ptr() as u64; // SAFETY: Backing stays initialized and mapped. No peer accesses it. - let mem = unsafe { GuestMemOps::from_raw_parts(base, 4) }; + let mem = unsafe { GuestMemOps::new(base, base + 4) }; assert_eq!(mem.write(base, &backing), Err(GuestMemError)); assert_eq!(mem.write(base, &backing[1..]), Err(GuestMemError)); @@ -251,7 +239,7 @@ mod tests { let mut backing = [1u8, 2, 3, 4]; let base = backing.as_mut_ptr() as u64; // SAFETY: Backing stays initialized and mapped. No peer accesses it. - let mem = unsafe { GuestMemOps::from_raw_parts(base, 4) }; + let mem = unsafe { GuestMemOps::new(base, base + 4) }; // Expose each copy address after splitting to preserve pointer provenance. let (left, right) = backing.split_at_mut(2); @@ -277,7 +265,7 @@ mod tests { let mut backing = [1u8, 2, 3, 4]; let base = backing.as_mut_ptr() as u64; // SAFETY: Backing stays initialized and mapped. No peer accesses it. - let mem = unsafe { GuestMemOps::from_raw_parts(base, 4) }; + let mem = unsafe { GuestMemOps::new(base, base + 4) }; mem.read(base, &mut backing[..0]).unwrap(); mem.write(base, &backing[..0]).unwrap(); @@ -289,38 +277,12 @@ mod tests { assert_eq!(backing, [1, 2, 3, 4]); } - #[test] - fn mapped_bytes_keep_the_slot_until_the_last_view_drops() { - let mut backing = [u64::from_ne_bytes(*b"dataTAIL")]; - let base = backing.as_mut_ptr() as u64; - // SAFETY: Backing remains live until all mapped views are dropped. - let mem = unsafe { GuestMemOps::from_raw_parts(base, 8) }; - let layout = SlotLayout::new(base, 8, 1).unwrap(); - let pool = SlotPool::new(layout).unwrap(); - let allocation = pool.alloc(8).unwrap(); - let lease = BufferLease::new(pool.clone(), allocation); - - // SAFETY: The allocation is initialized and exclusively leased. - let bytes = Bytes::from_owner(unsafe { mem.map_buffer(lease, 4) }.unwrap()); - assert_eq!(bytes.as_ref(), b"data"); - assert_eq!(bytes.as_ptr() as u64, base); - - let retained = bytes.slice(1..3); - drop(bytes); - assert_eq!(pool.num_free(), 0); - assert_eq!(retained.as_ref(), b"at"); - assert_eq!(retained.as_ptr() as u64, base + 1); - - drop(retained); - assert_eq!(pool.num_free(), 1); - } - #[test] fn failed_mapping_releases_its_lease() { let mut backing = [0u64; 2]; let base = backing.as_mut_ptr() as u64; // SAFETY: Backing remains mapped for the accessor's lifetime. - let mem = unsafe { GuestMemOps::from_raw_parts(base, 8) }; + let mem = unsafe { GuestMemOps::new(base, base + 8) }; let layout = SlotLayout::new(base + 8, 8, 1).unwrap(); let pool = SlotPool::new(layout).unwrap(); let allocation = pool.alloc(8).unwrap(); diff --git a/src/hyperlight_guest/src/transport/mod.rs b/src/hyperlight_guest/src/transport/mod.rs index e1e8e5b1f..df6f56138 100644 --- a/src/hyperlight_guest/src/transport/mod.rs +++ b/src/hyperlight_guest/src/transport/mod.rs @@ -3,8 +3,10 @@ //! Guest transport context and memory access. //! -//! Global context is installed once via [`set_global_context`] and accessed via [`with_context`]. +//! Global context is installed once via [`set_global_context`]. +//! Dispatch starts with [`maybe_refresh`]. Checkpointing uses [`prepare_snapshot`]. +mod backing; mod codec; pub mod context; pub mod mem; @@ -15,6 +17,8 @@ use core::sync::atomic::{AtomicU8, Ordering}; pub use context::{DispatchAction, GuestContext, QueueConfig}; pub(crate) use mem::GuestMemOps; +use crate::error::Result; + const UNINITIALIZED: u8 = 0; const INITIALIZED: u8 = 1; @@ -61,3 +65,22 @@ pub fn set_global_context(context: GuestContext) { ); *GLOBAL_CONTEXT.0.borrow_mut() = Some(context); } + +/// Prepare pool aliases before dispatch, logging, or tracing. +#[inline] +pub fn maybe_refresh() { + with_ctx(GuestContext::maybe_refresh); +} + +/// Reset stopped queues and publish snapshot readiness. +/// +/// # Safety +/// +/// The caller must satisfy [`GuestContext::prepare_snapshot`]'s requirements. +#[inline] +pub unsafe fn prepare_snapshot() -> Result<()> { + with_ctx(|ctx| { + // SAFETY: The caller upholds the checkpoint preconditions. + unsafe { ctx.prepare_snapshot() } + }) +} diff --git a/src/hyperlight_guest_bin/src/arch/amd64/layout.rs b/src/hyperlight_guest_bin/src/arch/amd64/layout.rs index 2652300a7..03c90043c 100644 --- a/src/hyperlight_guest_bin/src/arch/amd64/layout.rs +++ b/src/hyperlight_guest_bin/src/arch/amd64/layout.rs @@ -10,3 +10,8 @@ /// a VA that will survive the snapshot process. Since we don't have a /// useful virtual allocator yet, we just put them here... pub const PROC_CONTROL_GVA: u64 = 0xffff_fd00_0000_0000; + +const _: () = assert!( + hyperlight_common::layout::VIRTQ_BUFFER_GVA_END <= PROC_CONTROL_GVA, + "transport aliases overlap processor control" +); diff --git a/src/hyperlight_guest_bin/src/guest_function/call.rs b/src/hyperlight_guest_bin/src/guest_function/call.rs index 4c59e980f..dab42d8d8 100644 --- a/src/hyperlight_guest_bin/src/guest_function/call.rs +++ b/src/hyperlight_guest_bin/src/guest_function/call.rs @@ -71,6 +71,7 @@ pub(crate) fn call_guest_function(function_call: FunctionCall) -> Result transport::with_ctx(|ctx| ctx.send_h2g_result(cid, result)) - .expect("Failed to send function call result"), - None => transport::with_ctx(|ctx| { + Some((cid, result)) => transport::with_ctx(|ctx| { + ctx.send_h2g_result(cid, result) + .expect("Failed to send function call result"); + }), + None => { // SAFETY: Host chain accesses are complete. The checkpoint protocol // keeps consumers stopped until the host resets them. - unsafe { ctx.prepare_snapshot() } - }) - .expect("Failed to prepare snapshot transport"), + unsafe { transport::prepare_snapshot() } + .expect("Failed to prepare transport for snapshot"); + } } } diff --git a/src/hyperlight_guest_bin/src/lib.rs b/src/hyperlight_guest_bin/src/lib.rs index 6b1dd3dbd..7df64cefc 100644 --- a/src/hyperlight_guest_bin/src/lib.rs +++ b/src/hyperlight_guest_bin/src/lib.rs @@ -339,6 +339,11 @@ pub(crate) extern "C" fn generic_init( hyperlight_guest_tracing::flush(); } + // SAFETY: Host calls have returned with no live chain handles. The host + // resets consumers before the next dispatch refreshes aliases. + unsafe { hyperlight_guest::transport::prepare_snapshot() } + .expect("failed to prepare initial transport snapshot"); + dispatch_function as *const () as usize as u64 } diff --git a/src/hyperlight_guest_bin/src/transport.rs b/src/hyperlight_guest_bin/src/transport.rs index 59141502f..a5c0dd31c 100644 --- a/src/hyperlight_guest_bin/src/transport.rs +++ b/src/hyperlight_guest_bin/src/transport.rs @@ -5,8 +5,8 @@ use hyperlight_common::layout::{QueueDims, TransportArena}; use hyperlight_common::virtq::Layout; -use hyperlight_guest::transport::{GuestContext, QueueConfig}; -use hyperlight_guest::{layout, transport as guest_transport}; +use hyperlight_guest::layout; +use hyperlight_guest::transport::{self, GuestContext, QueueConfig}; use crate::paging::phys_to_virt; @@ -62,7 +62,7 @@ pub(crate) fn initialize() { } .expect("failed to create guest context"); - guest_transport::set_global_context(context); + transport::set_global_context(context); } fn scratch_gva(gpa: u64) -> u64 { diff --git a/src/hyperlight_host/benches/benchmarks.rs b/src/hyperlight_host/benches/benchmarks.rs index e1aac27ee..16555177b 100644 --- a/src/hyperlight_host/benches/benchmarks.rs +++ b/src/hyperlight_host/benches/benchmarks.rs @@ -14,9 +14,10 @@ use hyperlight_common::flatbuffer_wrappers::function_types::{Bytes, ParameterVal use hyperlight_common::flatbuffer_wrappers::util::estimate_flatbuffer_capacity; use hyperlight_common::transport::ExternalValues; use hyperlight_common::vmem::PAGE_SIZE; +use hyperlight_host::func::Registerable; use hyperlight_host::mem::shared_mem::ExclusiveSharedMemory; use hyperlight_host::sandbox::{Sandbox, SandboxConfiguration, UninitializedSandbox}; -use hyperlight_host::{GuestBinary, SandboxBuilder}; +use hyperlight_host::{GuestBinary, HostFunctions, SandboxBuilder}; use hyperlight_testing::sandbox_sizes::{LARGE_HEAP_SIZE, MEDIUM_HEAP_SIZE, SMALL_HEAP_SIZE}; use hyperlight_testing::{c_simple_guest_as_pathbuf, simple_guest_as_pathbuf}; @@ -41,7 +42,7 @@ impl SandboxSize { match self { Self::Default => builder, Self::Small => builder.heap_size(SMALL_HEAP_SIZE), - Self::Medium => builder.heap_size(MEDIUM_HEAP_SIZE).scratch_size(0x60000), + Self::Medium => builder.heap_size(MEDIUM_HEAP_SIZE).scratch_size(0x64000), Self::Large => builder.heap_size(LARGE_HEAP_SIZE).scratch_size(0x100000), } } @@ -340,6 +341,75 @@ fn snapshots_benchmark(c: &mut Criterion) { group.finish(); } +/// Measure sandbox creation from snapshots and restore latency. +/// Capture and host-function registration stay outside the timed regions. +fn retained_buffers_benchmark(c: &mut Criterion) { + let mut group = c.benchmark_group("retained_buffers"); + let host_funcs = || { + let mut fns = HostFunctions::default(); + fns.register_host_function("HostEchoByteChunks", |input: Vec| Ok(input)) + .unwrap(); + fns + }; + + let new_sandbox = || { + SandboxBuilder::from_file(simple_guest_as_pathbuf()) + .guest_log_level(tracing_core::LevelFilter::OFF) + .host_functions(host_funcs()) + .build() + .unwrap() + }; + + let mut bench = |name: &str, mut sandbox: Sandbox| { + let snapshot = sandbox.snapshot().unwrap(); + group.bench_function(BenchmarkId::new("from_snapshot", name), |b| { + b.iter_batched( + host_funcs, + |fns| Sandbox::from_snapshot(snapshot.clone(), fns, None).unwrap(), + criterion::BatchSize::PerIteration, + ); + }); + + group.bench_function(BenchmarkId::new("restore", name), |b| { + b.iter_custom(|iterations| { + let mut elapsed = Duration::ZERO; + + for _ in 0..iterations { + sandbox.call::("Echo", "hello".to_string()).unwrap(); + + let start = Instant::now(); + sandbox.restore(snapshot.clone()).unwrap(); + elapsed += start.elapsed(); + } + + elapsed + }); + }); + + group.bench_function(BenchmarkId::new("restore_and_first_request", name), |b| { + sandbox.call::("Echo", "hello".to_string()).unwrap(); + b.iter(|| { + sandbox.restore(snapshot.clone()).unwrap(); + sandbox.call::("Echo", "hello".to_string()).unwrap() + }); + }); + }; + + bench("none", new_sandbox()); + + let mut sandbox = new_sandbox(); + let payload = vec![Bytes::from(vec![0xa5; 2 * PAGE_SIZE])]; + + for func in ["RetainGuestByteChunks", "RetainHostByteChunks"] { + let retained: i32 = sandbox.call(func, payload.clone()).unwrap(); + assert_eq!(retained as usize, 2 * PAGE_SIZE); + } + + bench("both", sandbox); + + group.finish(); +} + // ============================================================================ // Benchmark Category: Guest Calls (Large Parameters) // ============================================================================ @@ -748,6 +818,7 @@ criterion_group! { sandbox_lifecycle_benchmark, guest_calls_benchmark, snapshots_benchmark, + retained_buffers_benchmark, guest_call_benchmark_large_param, function_call_codec_benchmark, sample_workloads_benchmark, diff --git a/src/hyperlight_host/src/mem/mgr.rs b/src/hyperlight_host/src/mem/mgr.rs index 664bcf1c6..6f7de60c4 100644 --- a/src/hyperlight_host/src/mem/mgr.rs +++ b/src/hyperlight_host/src/mem/mgr.rs @@ -7,7 +7,7 @@ use hyperlight_common::flatbuffer_wrappers::function_types::FunctionCallResult; use hyperlight_common::flatbuffer_wrappers::host_function_details::HostFunctionDetails; use hyperlight_common::flatbuffer_wrappers::util::estimate_flatbuffer_capacity; use hyperlight_common::log_level::GuestLogFilter; -use hyperlight_common::transport::{Buf, EncodedMessage, ExternalValues, MsgKind}; +use hyperlight_common::transport::{Buf, EncodedMessage, ExternalValues, MailboxValue, MsgKind}; use hyperlight_common::virtq::ReplyChain; use hyperlight_common::vmem::{self, PAGE_TABLE_SIZE}; #[cfg(crashdump)] @@ -388,6 +388,10 @@ impl SandboxMemoryManager { host_mgr.update_scratch_bookkeeping()?; if matches!(host_mgr.next_action, NextAction::Initialise(_)) { + let cp = MailboxValue::CheckpointPending.as_bytes(); + let mbx = host_mgr.layout.get_transport_arena().mbx_offset(); + + host_mgr.scratch_mem.copy_from_slice(cp, mbx)?; host_mgr.create_virtq_consumers()?; } @@ -630,21 +634,18 @@ impl SandboxMemoryManager { /// Publish an internal request for guest-side snapshot canonicalization. /// - /// The pending marker distinguishes a completed checkpoint with no retained - /// buffers from a guest that halted without publishing mailbox status. + /// The pending marker distinguishes a completed checkpoint from an early halt. pub(crate) fn begin_snapshot_checkpoint(&mut self) -> Result<()> { let offset = self.layout.get_transport_arena().mbx_offset(); - self.scratch_mem.write(offset, u64::MAX.to_le_bytes())?; + let cp = MailboxValue::CheckpointPending.as_bytes(); + self.scratch_mem.copy_from_slice(cp, offset)?; let message = EncodedMessage::new_snapshot_cp(); self.write_h2g_message(&message) } - /// Reset consumers before reading the retained count to keep rejected captures usable. - /// - /// Retained-buffer support will use the mailbox only for checkpoint and restore - /// completion (`u64::MAX` pending, zero ready after all preparation succeeds). - pub(crate) fn finish_snapshot_checkpoint(&mut self) -> Result { + /// Reset consumers and require the guest's completion status. + pub(crate) fn finish_snapshot_checkpoint(&mut self) -> Result<()> { let Some(g2h) = self.g2h_consumer.as_mut() else { return Err(new_error!("G2H consumer is not attached")); }; @@ -656,16 +657,20 @@ impl SandboxMemoryManager { g2h.reset()?; h2g.reset()?; - let offset = self.layout.get_transport_arena().mbx_offset(); - let guest_owned = u64::from_le_bytes(self.scratch_mem.read(offset)?); - - if guest_owned == u64::MAX { - return Err(HyperlightError::TransportError( - "Guest did not publish snapshot checkpoint status".to_string(), - )); + let status = self + .scratch_mem + .read::(self.layout.get_transport_arena().mbx_offset()) + .map_err(|error| HyperlightError::TransportError(error.to_string()))?; + + match MailboxValue::try_from(status) { + Ok(MailboxValue::CheckpointComplete) => Ok(()), + Ok(MailboxValue::CheckpointPending) => Err(HyperlightError::TransportError( + "Guest did not publish snapshot checkpoint status".into(), + )), + Err(_) => Err(HyperlightError::TransportError(format!( + "Guest published invalid snapshot checkpoint status {status:#x}" + ))), } - - Ok(guest_owned) } /// Restore base memory after the caller checks snapshot compatibility. @@ -731,6 +736,10 @@ impl SandboxMemoryManager { if let Some(virtq) = virtq { self.restore_virtq(virtq)?; } else if matches!(snapshot.next_action(), NextAction::Initialise(_)) { + self.scratch_mem.copy_from_slice( + MailboxValue::CheckpointPending.as_bytes(), + self.layout.get_transport_arena().mbx_offset(), + )?; self.create_virtq_consumers()?; } Ok((gsnapshot, gscratch)) @@ -1191,6 +1200,10 @@ mod tests { let mut mgr = manager(&case); let buffer = case.h2g_desc(0).addr; + let mbx = mgr.layout.get_transport_arena().mbx_offset(); + let cp = MailboxValue::CheckpointComplete.as_bytes(); + + mgr.scratch_mem.copy_from_slice(cp, mbx).unwrap(); mgr.begin_snapshot_checkpoint().unwrap(); let wire = case.h2g_buffer(0, buffer); @@ -1202,11 +1215,9 @@ mod tests { assert_eq!(header.payload_len, 0); assert_eq!(mgr.next_guest_cid, 1); - let mbx = mgr.layout.get_transport_arena().mbx_offset(); - assert_eq!( - mgr.scratch_mem.read::<[u8; 8]>(mbx).unwrap(), - u64::MAX.to_le_bytes() + MailboxValue::try_from(mgr.scratch_mem.read::(mbx).unwrap()).unwrap(), + MailboxValue::CheckpointPending ); } @@ -1225,18 +1236,33 @@ mod tests { #[test] fn reads_completed_snapshot_checkpoint_status() { - for retained in [0u64, 3] { - let case = TestCase::new(); - let mut mgr = manager(&case); - mgr.g2h_consumer = Some(case.g2h_consumer()); - mgr.begin_snapshot_checkpoint().unwrap(); + let case = TestCase::new(); + let mut mgr = manager(&case); + mgr.g2h_consumer = Some(case.g2h_consumer()); + mgr.begin_snapshot_checkpoint().unwrap(); + let mbx = mgr.layout.get_transport_arena().mbx_offset(); + mgr.scratch_mem + .copy_from_slice(MailboxValue::CheckpointComplete.as_bytes(), mbx) + .unwrap(); + + mgr.finish_snapshot_checkpoint().unwrap(); + let consumer = mgr.h2g_consumer.as_ref().unwrap(); + assert_eq!(consumer.avail_cursor().head(), 0); + assert_eq!(consumer.used_cursor().head(), 0); + } + + #[test] + fn invalid_checkpoint_status_is_fatal() { + for status in [2, 3, 0xa000, u64::MAX - 1, u64::MAX] { + let queue = TestCase::new(); + let mut mgr = manager(&queue); + mgr.g2h_consumer = Some(queue.g2h_consumer()); let mbx = mgr.layout.get_transport_arena().mbx_offset(); - mgr.scratch_mem.write(mbx, retained.to_le_bytes()).unwrap(); + mgr.scratch_mem.write::(mbx, status).unwrap(); + let error = mgr.finish_snapshot_checkpoint().unwrap_err(); - assert_eq!(mgr.finish_snapshot_checkpoint().unwrap(), retained); - let consumer = mgr.h2g_consumer.as_ref().unwrap(); - assert_eq!(consumer.avail_cursor().head(), 0); - assert_eq!(consumer.used_cursor().head(), 0); + assert!(matches!(error, HyperlightError::TransportError(_))); + assert!(error.is_poison_error(), "{error}"); } } diff --git a/src/hyperlight_host/src/mem/virtq/mod.rs b/src/hyperlight_host/src/mem/virtq/mod.rs index 06e11610c..c699306fa 100644 --- a/src/hyperlight_host/src/mem/virtq/mod.rs +++ b/src/hyperlight_host/src/mem/virtq/mod.rs @@ -10,7 +10,7 @@ //! untrusted guest requests and results into host-owned values before use. //! Shared wire framing lives in `hyperlight_common::transport`. //! -//! Snapshots require canonical rings with empty G2H and the initial H2G prefill. +//! Snapshots require canonical rings with empty G2H and H2G free-slot prefill. //! Each H2G chain contains one writable descriptor naming a distinct, //! configured-size slot aligned relative to the H2G pool start. @@ -25,7 +25,8 @@ pub(crate) use codec::{ get_host_function_call, read_guest_function_call_result, read_guest_log_data, read_message_header, try_write_response, }; -use hyperlight_common::virtq::canonical::validate_canon_image; +use hyperlight_common::transport::MailboxValue; +use hyperlight_common::virtq::canonical::{validate_canon_image, validate_canon_prefix}; use hyperlight_common::virtq::{Layout as VirtqLayout, Notifier, QueueStats, VirtqConsumer}; use mem::{HostMemOps, ImageMem}; @@ -139,11 +140,14 @@ impl VirtqSnapshot { scratch_mem: &HostSharedMemory, ) -> Result<(G2hConsumer, H2gConsumer)> { let (g2h_offset, h2g_offset) = ring_offsets(layout); + let mbx_offset = layout.get_transport_arena().mbx_offset(); + let cp = MailboxValue::CheckpointComplete.as_bytes(); write_published_arena_gpa(scratch_mem, layout.get_transport_arena().base_addr())?; scratch_mem.copy_from_slice(&self.g2h_ring, g2h_offset)?; scratch_mem.copy_from_slice(&self.h2g_ring, h2g_offset)?; + scratch_mem.copy_from_slice(cp, mbx_offset)?; create_consumers(layout, scratch_mem) } @@ -174,15 +178,15 @@ impl VirtqSnapshot { let buffer_size = layout.get_h2g_buffer_size(); let h2g_dims = layout.get_h2g_queue_dims(); - let h2g_prefill = usize::from(h2g_dims.size().get()).min(h2g_dims.pool_len() / buffer_size); + let max_prefill = usize::from(h2g_dims.size().get()).min(h2g_dims.pool_len() / buffer_size); let h2g_mem = ImageMem::new(h2g.desc_table_addr(), &self.h2g_ring); let (_, _, _, pool_offset, _) = layout.get_transport_arena().to_offsets(); let pool_start = g2h.desc_table_addr() + pool_offset as u64; let pool_end = pool_start + h2g_dims.pool_len() as u64; - let mut seen_slots = HashSet::with_capacity(h2g_prefill); + let mut seen_slots = HashSet::with_capacity(max_prefill); - let chains = validate_canon_image(&h2g_mem, h2g, h2g_prefill, |_, elem| { + let chains = validate_canon_prefix(&h2g_mem, h2g, max_prefill, |_, elem| { if !elem.writable || usize::try_from(elem.len).ok() != Some(buffer_size) { return false; } @@ -203,7 +207,7 @@ impl VirtqSnapshot { }) .map_err(|error| new_error!("invalid canonical H2G image: {error}"))?; - if chains.len() != h2g_prefill { + if chains.iter().any(|chain| chain.buffers().len() != 1) { return Err(new_error!( "H2G snapshot chains must contain one descriptor" )); diff --git a/src/hyperlight_host/src/mem/virtq/tests.rs b/src/hyperlight_host/src/mem/virtq/tests.rs index 3b7953818..0048fb79c 100644 --- a/src/hyperlight_host/src/mem/virtq/tests.rs +++ b/src/hyperlight_host/src/mem/virtq/tests.rs @@ -138,7 +138,11 @@ fn snapshots_and_restores_rings() { let allocator = layout.get_first_free_scratch_gpa(); let allocator_offset = restored.mem_size() - SCRATCH_TOP_ALLOCATOR_OFFSET as usize; + let mbx_offset = layout.get_transport_arena().mbx_offset(); + let cp = MailboxValue::CheckpointPending.as_bytes(); + restored.write::(allocator_offset, allocator).unwrap(); + restored.copy_from_slice(cp, mbx_offset).unwrap(); let (mut g2h, mut h2g) = captured.restore(&layout, &restored).unwrap(); let restored_snapshot = VirtqSnapshot::capture(&layout, &restored).unwrap(); @@ -152,6 +156,13 @@ fn snapshots_and_restores_rings() { assert_eq!(restored.read::(allocator_offset).unwrap(), allocator); assert_eq!(restored.read::<[u8; 16]>(spare_offset).unwrap(), [0; 16]); assert_eq!(pool_bytes, [0; 16]); + + let mbx = restored.read::(mbx_offset).unwrap(); + assert_eq!( + MailboxValue::try_from(mbx).unwrap(), + MailboxValue::CheckpointComplete + ); + assert!(g2h.poll(0).unwrap().is_none()); let (recv, reply) = h2g.poll(0).unwrap().unwrap(); h2g.complete(recv, reply).unwrap(); @@ -389,3 +400,26 @@ fn malformed_descriptors_fail_when_polled() { Err(VirtqError::RingError(RingError::BadChain)) )); } + +#[test] +fn restores_partially_prefilled_h2g_ring() { + for prefill in 0..=4 { + let case = TestCase::new(); + + for index in prefill..4 { + case.set_h2g_desc(index, Descriptor::new(0, 0, 0, DescFlags::empty())); + } + + let layout = memory_layout(); + let captured = VirtqSnapshot::capture(&layout, &case.scratch).unwrap(); + let restored = host_scratch(); + let (_, mut consumer) = captured.restore(&layout, &restored).unwrap(); + + for _ in 0..prefill { + let (request, reply) = consumer.poll(0).unwrap().unwrap(); + consumer.complete(request, reply).unwrap(); + } + + assert!(consumer.poll(0).unwrap().is_none()); + } +} diff --git a/src/hyperlight_host/src/sandbox/config.rs b/src/hyperlight_host/src/sandbox/config.rs index edd632d16..b5456c468 100644 --- a/src/hyperlight_host/src/sandbox/config.rs +++ b/src/hyperlight_host/src/sandbox/config.rs @@ -105,8 +105,7 @@ impl SandboxConfiguration { pub const INTERRUPT_VCPU_SIGRTMIN_OFFSET: u8 = 0; /// The default heap size of a hyperlight sandbox pub const DEFAULT_HEAP_SIZE: u64 = 131072; - // TODO: Reassess the scratch budget and runtime headroom. - /// Scratch backs the default heap after reserving the arena and page tables. + /// Scratch backs the default heap, transport pools, and page tables. /// The size is 16 KiB aligned for macOS hosts. pub const DEFAULT_SCRATCH_SIZE: usize = 0x58000; /// The default G2H virtqueue descriptor count. diff --git a/src/hyperlight_host/src/sandbox/initialized.rs b/src/hyperlight_host/src/sandbox/initialized.rs index 1b3778162..4e98c9124 100644 --- a/src/hyperlight_host/src/sandbox/initialized.rs +++ b/src/hyperlight_host/src/sandbox/initialized.rs @@ -324,6 +324,12 @@ impl Sandbox { // delivered through the scratch-memory request below. vm.initialise(peb_addr, seed, &mut hshm, &host_funcs, max_guest_log_level) .map_err(crate::hypervisor::hyperlight_vm::HyperlightVmError::Initialize)?; + if matches!( + snapshot.next_action(), + super::snapshot::NextAction::Initialise(_) + ) { + hshm.finish_snapshot_checkpoint()?; + } if matches!(snapshot.next_action(), super::snapshot::NextAction::Call(_)) { hshm.request_libc_rng_reseed(seed as u32)?; @@ -473,13 +479,7 @@ impl Sandbox { sbox.mem_mgr.begin_snapshot_checkpoint()?; sbox.dispatch_guest_call()?; - let guest_owned = sbox.mem_mgr.finish_snapshot_checkpoint()?; - if guest_owned != 0 { - // Fixed-pool payloads are outside the ordinary memory snapshot. - return Err(HyperlightError::Error(format!( - "Cannot snapshot while {guest_owned} transport buffers are retained" - ))); - } + sbox.mem_mgr.finish_snapshot_checkpoint()?; sbox.transport_dirty = false; Ok(()) @@ -1247,8 +1247,10 @@ mod tests { use std::sync::{Arc, Barrier}; use std::thread; + use hyperlight_common::flatbuffer_wrappers::function_call::{FunctionCall, FunctionCallType}; + use hyperlight_common::flatbuffer_wrappers::function_types::ParameterValue; use hyperlight_common::flatbuffer_wrappers::guest_error::ErrorCode; - use hyperlight_common::func::Bytes; + use hyperlight_common::func::{Bytes, SupportedReturnType}; use hyperlight_testing::sandbox_sizes::{LARGE_HEAP_SIZE, MEDIUM_HEAP_SIZE, SMALL_HEAP_SIZE}; use hyperlight_testing::{c_simple_guest_as_pathbuf, simple_guest_as_pathbuf}; @@ -1565,6 +1567,12 @@ mod tests { ); assert_eq!(sandbox.mem_mgr.snapshot_count, generation); assert!(sandbox.snapshot.is_none()); + + // Restore the canonical descriptor before continuing the source. + sandbox.mem_mgr.scratch_mem.write::(offset, 0).unwrap(); + let expected = vec![0x5a; 2 * hyperlight_common::vmem::PAGE_SIZE + 137]; + let actual: Vec = sandbox.call("EchoGuestVecBytes", expected.clone()).unwrap(); + assert_eq!(actual, expected); } #[test] @@ -1620,7 +1628,7 @@ mod tests { } #[test] - fn snapshots_reject_retained_transport_buffers_without_poisoning() { + fn snapshots_preserve_retained_transport_and_clear_abort_state() { let path = simple_guest_as_pathbuf(); let mut sandbox = UninitializedSandbox::new(GuestBinary::FilePath(path), None).unwrap(); sandbox @@ -1630,56 +1638,72 @@ mod tests { let mut sandbox = sandbox.evolve().unwrap(); let retained = vec![Bytes::from(vec![0xa5; 6 * 1024])]; - let retained_len: i32 = sandbox - .call("RetainGuestByteChunks", retained.clone()) - .unwrap(); + for (retain, release) in [ + ("RetainGuestByteChunks", "ReleaseGuestByteChunks"), + ("RetainHostByteChunks", "ReleaseHostByteChunks"), + ] { + let retained_len: i32 = sandbox.call(retain, retained.clone()).unwrap(); + assert_eq!(retained_len, 6 * 1024); + sandbox.mem_mgr.abort_buffer.extend_from_slice(&[0xAA; 8]); - assert_eq!(retained_len, 6 * 1024); - sandbox.mem_mgr.abort_buffer.extend_from_slice(&[0xAA; 8]); + let snapshot = sandbox.snapshot().unwrap(); + assert!(!sandbox.status().is_poisoned()); + assert!(!sandbox.transport_dirty); + assert!(sandbox.mem_mgr.abort_buffer.is_empty()); - let Err(error) = sandbox.snapshot() else { - panic!("snapshot with retained H2G buffers succeeded"); - }; + assert_eq!(sandbox.call::(release, ()).unwrap(), retained_len); - match error { - HyperlightError::Error(message) => { - assert!(message.contains("transport buffers are retained")) - } - err => unreachable!("unexpected snapshot error: {err:#}"), + sandbox.restore(snapshot).unwrap(); + assert_eq!(sandbox.call::(release, ()).unwrap(), retained_len); + sandbox.snapshot().unwrap(); + assert!(!sandbox.transport_dirty); } - assert!(!sandbox.status().is_poisoned()); - assert!(sandbox.transport_dirty); - assert!(sandbox.mem_mgr.abort_buffer.is_empty()); - - let released_len: i32 = sandbox.call("ReleaseGuestByteChunks", ()).unwrap(); - assert_eq!(released_len, retained_len); - - sandbox.snapshot().unwrap(); - assert!(!sandbox.transport_dirty); + } - let retained_len: i32 = sandbox.call("RetainHostByteChunks", retained).unwrap(); - assert_eq!(retained_len, 6 * 1024); - sandbox.mem_mgr.abort_buffer.extend_from_slice(&[0xAA; 8]); + #[test] + fn retained_restore_is_ready_for_first_request() { + let mut sandbox = SandboxBuilder::from_file(simple_guest_as_pathbuf()) + .build() + .unwrap(); + sandbox + .call::( + "RetainGuestByteChunks", + vec![Bytes::from(vec![0x5a; 6 * 1024])], + ) + .unwrap(); + sandbox.checkpoint_transport_for_snapshot().unwrap(); + let source_root = sandbox.vm.get_root_pt().unwrap(); + let snapshot = sandbox.snapshot().unwrap(); + assert_eq!(sandbox.vm.get_root_pt().unwrap(), source_root); - let Err(error) = sandbox.snapshot() else { - panic!("snapshot with retained G2H buffers succeeded"); - }; + sandbox.restore(snapshot.clone()).unwrap(); + let mut clone = + Sandbox::from_snapshot(snapshot, crate::HostFunctions::default(), None).unwrap(); + let expected = vec![0x3c; 3 * hyperlight_common::vmem::PAGE_SIZE + 117]; + let call = FunctionCall::new( + "EchoGuestVecBytes".into(), + Some(vec![ParameterValue::VecBytes(expected.clone())]), + FunctionCallType::Guest, + Vec::::TYPE, + ); - match error { - HyperlightError::Error(message) => { - assert!(message.contains("transport buffers are retained")) - } - err => unreachable!("unexpected snapshot error: {err:#}"), + for restored in [&mut sandbox, &mut clone] { + restored.transport_dirty = true; + let cid = restored.mem_mgr.write_guest_function_call(&call).unwrap(); + restored.dispatch_guest_call().unwrap(); + let value = restored + .mem_mgr + .read_h2g_result_from_g2h(cid) + .unwrap() + .into_inner() + .unwrap(); + assert_eq!(Vec::::from_value(value).unwrap(), expected); } - assert!(!sandbox.status().is_poisoned()); - assert!(sandbox.transport_dirty); - assert!(sandbox.mem_mgr.abort_buffer.is_empty()); - let released_len: i32 = sandbox.call("ReleaseHostByteChunks", ()).unwrap(); - assert_eq!(released_len, retained_len); - - sandbox.snapshot().unwrap(); - assert!(!sandbox.transport_dirty); + assert_eq!( + clone.call::("ReleaseGuestByteChunks", ()).unwrap(), + 6 * 1024 + ); } #[test] diff --git a/src/hyperlight_host/src/sandbox/snapshot/file/config.rs b/src/hyperlight_host/src/sandbox/snapshot/file/config.rs index 93bff944d..1427c53a5 100644 --- a/src/hyperlight_host/src/sandbox/snapshot/file/config.rs +++ b/src/hyperlight_host/src/sandbox/snapshot/file/config.rs @@ -941,7 +941,7 @@ mod schema_pin { const PINNED_CALL: &str = r#"{ "hyperlight_version": "x.y.z", "arch": "x86_64", - "abi_version": 5, + "abi_version": 6, "hypervisor": "mshv", "cpu_vendor": "intel", "stack_top_gva": 3735928559, @@ -1132,7 +1132,7 @@ mod schema_pin { const PINNED_CALL: &str = r#"{ "hyperlight_version": "x.y.z", "arch": "aarch64", - "abi_version": 5, + "abi_version": 6, "hypervisor": "mshv", "cpu_vendor": "intel", "stack_top_gva": 3735928559, diff --git a/src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs b/src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs index a7444f9c5..e745ef2af 100644 --- a/src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs +++ b/src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs @@ -21,7 +21,7 @@ pub(in crate::sandbox::snapshot) const MT_TRANSPORT_CURRENT: &str = MT_TRANSPORT /// ABI version for the snapshot memory blob. Bumped when the /// host-guest contract for the snapshot bytes changes. See /// docs/snapshot-versioning.md. -pub(in crate::sandbox::snapshot) const SNAPSHOT_ABI_VERSION: u32 = 5; +pub(in crate::sandbox::snapshot) const SNAPSHOT_ABI_VERSION: u32 = 6; /// OCI standard annotation key for a manifest's tag inside an image /// index. Set on the manifest descriptor in `index.json`, not on the diff --git a/src/hyperlight_host/src/sandbox/snapshot/file/transport.rs b/src/hyperlight_host/src/sandbox/snapshot/file/transport.rs index 980ff21d3..936bd0361 100644 --- a/src/hyperlight_host/src/sandbox/snapshot/file/transport.rs +++ b/src/hyperlight_host/src/sandbox/snapshot/file/transport.rs @@ -305,6 +305,7 @@ mod tests { bytes[HEADER_LEN] = 0; let h2g_offset = HEADER_LEN + snapshot.g2h_ring().len(); bytes[h2g_offset..].fill(0); + bytes[h2g_offset] = 1; let error = decode(&layout, &bytes).unwrap_err(); assert!( diff --git a/src/hyperlight_host/src/sandbox/snapshot/file_tests.rs b/src/hyperlight_host/src/sandbox/snapshot/file_tests.rs index d36b4b277..7e801ea92 100644 --- a/src/hyperlight_host/src/sandbox/snapshot/file_tests.rs +++ b/src/hyperlight_host/src/sandbox/snapshot/file_tests.rs @@ -2091,7 +2091,7 @@ fn legacy_config_versions_rejected() { &path, OciTag::new("latest").unwrap(), )); - assert_err_contains(err, "incompatible with snapshot ABI 5"); + assert_err_contains(err, "incompatible with snapshot ABI 6"); } } diff --git a/src/hyperlight_host/src/sandbox/snapshot/tripwires.rs b/src/hyperlight_host/src/sandbox/snapshot/tripwires.rs index f1b3b0b0a..46302ed05 100644 --- a/src/hyperlight_host/src/sandbox/snapshot/tripwires.rs +++ b/src/hyperlight_host/src/sandbox/snapshot/tripwires.rs @@ -16,7 +16,7 @@ use super::file::{ SNAPSHOT_ABI_VERSION, }; -const EXPECTED_ABI_VERSION: u32 = 5; +const EXPECTED_ABI_VERSION: u32 = 6; const EXPECTED_MT_CONFIG: &str = "application/vnd.hyperlight.snapshot.config.v3+json"; const EXPECTED_MT_SNAPSHOT: &str = "application/vnd.hyperlight.snapshot.memory.v1"; const EXPECTED_MT_TRANSPORT: &str = "application/vnd.hyperlight.snapshot.transport.v1"; diff --git a/src/hyperlight_host/src/sandbox/uninitialized_evolve.rs b/src/hyperlight_host/src/sandbox/uninitialized_evolve.rs index 0bd355474..5cfcddd6b 100644 --- a/src/hyperlight_host/src/sandbox/uninitialized_evolve.rs +++ b/src/hyperlight_host/src/sandbox/uninitialized_evolve.rs @@ -4,6 +4,7 @@ use rand::RngExt; use tracing::{Span, instrument}; use super::SandboxConfiguration; +use super::snapshot::NextAction; #[cfg(any(crashdump, gdb))] use super::uninitialized::SandboxRuntimeConfig; use crate::hypervisor::hyperlight_vm::{HyperlightVm, HyperlightVmError}; @@ -77,6 +78,7 @@ pub(super) fn initialize_sandbox(u_sbox: UninitializedSandbox) -> Result Result = sandbox.call("ReadRetainedGuestByteChunks", ()).unwrap(); + assert_eq!(actual.concat(), expected); + + let ret = sandbox.call::("ReleaseGuestByteChunks", ()).unwrap(); + assert_eq!(ret, retained); + + let after_release: Vec = sandbox.call("ReadRetainedGuestByteChunks", ()).unwrap(); + assert!(after_release.is_empty()); + + sandbox.restore(snapshot).unwrap(); + let actual: Vec = sandbox.call("ReadRetainedGuestByteChunks", ()).unwrap(); + assert_eq!(actual.concat(), expected); + + let ret = sandbox.call::("ReleaseGuestByteChunks", ()).unwrap(); + assert_eq!(ret, retained); + }); +} + +/// Both guest APIs keep host-reply buffers alive across calls and restore. +#[test] +fn retained_host_reply_bytes_survive_restore() { + common::with_all_guests(|path| { + let mut sandbox = SandboxBuilder::from_file(&path) + .host_function("HostEchoByteChunks", |value: Vec| value) + .build() + .unwrap(); + + let expected = vec![0xa5; 6 * 1024]; + let retained: i32 = sandbox + .call( + "RetainHostByteChunks", + vec![Bytes::copy_from_slice(&expected)], + ) + .unwrap(); + + assert_eq!(retained as usize, expected.len()); + let snapshot = sandbox.snapshot().unwrap(); + + let echo: String = sandbox.call("Echo", "hello".to_string()).unwrap(); + assert_eq!(echo, "hello"); + + let actual: Vec = sandbox.call("ReadRetainedHostByteChunks", ()).unwrap(); + assert_eq!(actual.concat(), expected, "guest: {path:?}"); + + let ret = sandbox.call::("ReleaseHostByteChunks", ()).unwrap(); + assert_eq!(ret, retained); + + let after_release: Vec = sandbox.call("ReadRetainedHostByteChunks", ()).unwrap(); + assert!(after_release.is_empty()); + + sandbox.restore(snapshot).unwrap(); + let actual: Vec = sandbox.call("ReadRetainedHostByteChunks", ()).unwrap(); + assert_eq!(actual.concat(), expected, "guest: {path:?}"); + + let ret = sandbox.call::("ReleaseHostByteChunks", ()).unwrap(); + assert_eq!(ret, retained); + }); +} + +/// Captured aliases survive scratch reuse and further snapshot generations. +#[test] +fn retained_byte_chunks_survive_pool_generation_reuse() { + let mut sandbox = SandboxBuilder::from_file(hyperlight_testing::simple_guest_as_pathbuf()) + .guest_log_level(tracing_core::LevelFilter::OFF) + .host_function("HostEchoByteChunks", |value: Vec| value) + .build() + .unwrap(); + + let expected = vec![0xa5; 8192]; + let actual: i32 = sandbox + .call( + "RetainGuestByteChunks", + vec![Bytes::copy_from_slice(&expected)], + ) + .unwrap(); + + assert_eq!(actual as usize, expected.len()); + + let actual: i32 = sandbox + .call( + "RetainHostByteChunks", + vec![Bytes::copy_from_slice(&expected)], + ) + .unwrap(); + + assert_eq!(actual as usize, expected.len()); + + let snapshot = sandbox.snapshot().unwrap(); + sandbox.restore(snapshot).unwrap(); + + let actual: i32 = sandbox + .call("ByteChunksLen", vec![Bytes::from(vec![0x5a; 12288])]) + .unwrap(); + + assert_eq!(actual, 12288); + + let actual: Vec = sandbox.call("ReadRetainedGuestByteChunks", ()).unwrap(); + assert_eq!(actual.concat(), expected); + + let actual: Vec = sandbox.call("ReadRetainedHostByteChunks", ()).unwrap(); + assert_eq!(actual.concat(), expected); + + // Capture recovered pools while their older retained aliases remain live. + let snapshot = sandbox.snapshot().unwrap(); + sandbox.restore(snapshot).unwrap(); + + let actual: i32 = sandbox + .call("ByteChunksLen", vec![Bytes::from(vec![0x5a; 12288])]) + .unwrap(); + + assert_eq!(actual, 12288); + + let actual: Vec = sandbox.call("ReadRetainedGuestByteChunks", ()).unwrap(); + assert_eq!(actual.concat(), expected); + + let actual: Vec = sandbox.call("ReadRetainedHostByteChunks", ()).unwrap(); + assert_eq!(actual.concat(), expected); + + let replacement = vec![0x3c; 6 * 4096]; + let actual: i32 = sandbox + .call( + "RetainGuestByteChunks", + vec![Bytes::copy_from_slice(&replacement)], + ) + .unwrap(); + + assert_eq!(actual as usize, replacement.len()); + + let actual: Vec = sandbox.call("ReadRetainedGuestByteChunks", ()).unwrap(); + assert_eq!(actual.concat(), replacement); + assert_eq!( + sandbox.call::("ReleaseGuestByteChunks", ()).unwrap(), + 6 * 4096 + ); + assert_eq!( + sandbox.call::("ReleaseHostByteChunks", ()).unwrap(), + 8192 + ); +} + /// Test that validates interrupt behavior with random kill timing under concurrent load /// Uses a pool of 100 sandboxes, 100 threads, and 500 iterations per thread. /// Randomly decides to kill some calls at random times during execution. @@ -1740,7 +1895,10 @@ fn page_fault_handler() { /// This validates that the exception handling path does not require heap allocations. #[test] fn fill_heap_and_cause_exception() { - with_rust_sandbox(|mut sandbox| { + // The heap must run out before scratch backing its CoW pages. + let configure = |builder: SandboxBuilder| builder.heap_size(40 * 1024); + + with_rust_sandbox_from(configure, |mut sandbox| { let result = sandbox.call::<()>("FillHeapAndCauseException", ()); // The call should fail with an exception error since there's no handler installed diff --git a/src/hyperlight_host/tests/sandbox_host_tests.rs b/src/hyperlight_host/tests/sandbox_host_tests.rs index 949b8335f..6421fa5b3 100644 --- a/src/hyperlight_host/tests/sandbox_host_tests.rs +++ b/src/hyperlight_host/tests/sandbox_host_tests.rs @@ -37,15 +37,19 @@ fn pass_byte_array() { fn fragmented_control_round_trip_releases_buffers() { // The control body exceeds the four inline segment slots. let input = "x".repeat(5 * SandboxConfiguration::DEFAULT_H2G_BUFFER_SIZE); + with_all_guests(|path| { let mut sbox = SandboxBuilder::from_file(path) .heap_size(256 * 1024) + .scratch_size(512 * 1024) .build() .unwrap(); - let output: String = sbox.call("Echo", input.clone()).unwrap(); - assert_eq!(output, input); - // Snapshot preparation rejects retained transport buffers. + for _ in 0..3 { + let output: String = sbox.call("Echo", input.clone()).unwrap(); + assert_eq!(output, input); + } + sbox.snapshot().unwrap(); }); } @@ -612,6 +616,48 @@ fn h2g_capacity_failure_does_not_poison_sandbox() { }); } +#[test] +fn retained_sub_page_h2g_slot_keeps_neighbors_available() { + let mut cfg = SandboxConfiguration::default(); + cfg.set_h2g_buffer_size(256); + cfg.set_h2g_pool_pages(1); + + with_rust_uninit_sandbox_cfg(cfg, |sandbox| { + let mut sandbox = sandbox.evolve().unwrap(); + let expected = vec![0x5a; 16]; + + // Every H2G slot shares the retained slot's page. + let retained: i32 = sandbox + .call( + "RetainGuestByteChunks", + vec![Bytes::copy_from_slice(&expected)], + ) + .unwrap(); + assert_eq!(retained as usize, expected.len()); + + let snapshot = sandbox.snapshot().unwrap(); + let echo: String = sandbox.call("Echo", "source".to_string()).unwrap(); + assert_eq!(echo, "source"); + + let released: i32 = sandbox.call("ReleaseGuestByteChunks", ()).unwrap(); + assert_eq!(released, retained); + + for _ in 0..2 { + sandbox.restore(snapshot.clone()).unwrap(); + + // The host writes this request beside the retained slot before guest entry. + let echo: String = sandbox.call("Echo", "restored".to_string()).unwrap(); + assert_eq!(echo, "restored"); + + let actual: Vec = sandbox.call("ReadRetainedGuestByteChunks", ()).unwrap(); + assert_eq!(actual.concat(), expected); + + let released: i32 = sandbox.call("ReleaseGuestByteChunks", ()).unwrap(); + assert_eq!(released, retained); + } + }); +} + fn assert_g2h_reply_capacity_failure_is_recoverable(queue_size: usize, pool_pages: usize) { let mut cfg = SandboxConfiguration::default(); cfg.set_g2h_buffer_size(4096); @@ -707,6 +753,43 @@ fn g2h_reply_capacity_retained_buffers_are_recoverable() { }); } +#[test] +fn retained_g2h_reply_on_every_page_keeps_neighbors_available() { + let mut cfg = SandboxConfiguration::default(); + cfg.set_g2h_buffer_size(256); + cfg.set_g2h_pool_pages(2); + + with_rust_uninit_sandbox_cfg(cfg, |mut sandbox| { + sandbox.set_max_guest_log_level(tracing_core::LevelFilter::OFF); + + // The retained reply touches both pool pages. + sandbox + .register("HostEchoByteChunks", |_: Vec| { + vec![Bytes::from(vec![0xa5; 6000])] + }) + .unwrap(); + sandbox.register("HostNoOp", || {}).unwrap(); + let mut sandbox = sandbox.evolve().unwrap(); + + let retained: i32 = sandbox + .call("RetainHostByteChunks", Vec::::new()) + .unwrap(); + assert_eq!(retained, 6000); + + let snapshot = sandbox.snapshot().unwrap(); + + for restore in [false, true] { + if restore { + sandbox.restore(snapshot.clone()).unwrap(); + } + + sandbox.call::<()>("RoundTripHostNoOp", ()).unwrap(); + let released: i32 = sandbox.call("ReleaseHostByteChunks", ()).unwrap(); + assert_eq!(released, retained); + } + }); +} + #[test] fn g2h_reply_capacity_uses_available_upper_buffers() { let mut cfg = SandboxConfiguration::default(); diff --git a/src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs b/src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs index ade8e5559..164a276c8 100644 --- a/src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs +++ b/src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs @@ -8,7 +8,7 @@ //! publish. See `docs/snapshot-versioning.md`. /// Goldens version, a `vMAJOR.MINOR` string. -pub(crate) const GOLDENS_VERSION: &str = "v5.0"; +pub(crate) const GOLDENS_VERSION: &str = "v6.0"; /// Old majors kept loadable through a compatibility path, verified /// alongside `GOLDENS_VERSION`. A backwards-compatible break (Option 2) diff --git a/src/tests/c_guests/c_simpleguest/main.c b/src/tests/c_guests/c_simpleguest/main.c index 318ace812..5cf74e6e3 100644 --- a/src/tests/c_guests/c_simpleguest/main.c +++ b/src/tests/c_guests/c_simpleguest/main.c @@ -366,6 +366,56 @@ hl_ReturnValue *round_trip_host_byte_chunks(const hl_FunctionCall *params) { return result; } +// Host return handles own their chunks until hl_free_byte_chunks. +static hl_ByteChunks *retained_host_chunks = NULL; + +static int32_t byte_chunks_len(const hl_ByteChunks *value) { + uintptr_t len = 0; + + if (value != NULL) { + for (uintptr_t i = 0; i < value->count; i++) { + len += value->chunks[i].len; + } + } + + assert(len <= INT32_MAX); + return (int32_t)len; +} + +hl_ReturnValue *retain_host_byte_chunks(const hl_FunctionCall *params) { + const hl_FunctionCall host_call = { + .function_name = "HostEchoByteChunks", + .parameters = params->parameters, + .parameters_len = 1, + .return_type = hl_ReturnType_ByteChunks, + }; + hl_call_host_function(&host_call); + + hl_ByteChunks *output = hl_get_host_return_value_as_ByteChunks(); + assert(output != NULL); + hl_free_byte_chunks(retained_host_chunks); + retained_host_chunks = output; + return hl_result_from_Int(byte_chunks_len(output)); +} + +hl_ReturnValue *read_retained_host_byte_chunks(const hl_FunctionCall *params) { + (void)params; + + if (retained_host_chunks == NULL) { + return hl_result_from_ByteChunks((hl_ByteChunks){0}); + } + + return hl_result_from_ByteChunks(*retained_host_chunks); +} + +hl_ReturnValue *release_host_byte_chunks(const hl_FunctionCall *params) { + (void)params; + int32_t len = byte_chunks_len(retained_host_chunks); + hl_free_byte_chunks(retained_host_chunks); + retained_host_chunks = NULL; + return hl_result_from_Int(len); +} + hl_ReturnValue *return_null(const hl_FunctionCall *params) { (void)params; return NULL; @@ -483,6 +533,9 @@ void hyperlight_main(void) // so we use hl_register_function_definition directly hl_register_function_definition("24K_in_8K_out", twenty_four_k_in_eight_k_out, 1, (hl_ParameterType[]){hl_ParameterType_VecBytes}, hl_ReturnType_VecBytes); hl_register_function_definition("RoundTripHostByteChunks", round_trip_host_byte_chunks, 1, (hl_ParameterType[]){hl_ParameterType_ByteChunks}, hl_ReturnType_ByteChunks); + hl_register_function_definition("RetainHostByteChunks", retain_host_byte_chunks, 1, (hl_ParameterType[]){hl_ParameterType_ByteChunks}, hl_ReturnType_Int); + hl_register_function_definition("ReadRetainedHostByteChunks", read_retained_host_byte_chunks, 0, (hl_ParameterType[]){0}, hl_ReturnType_ByteChunks); + hl_register_function_definition("ReleaseHostByteChunks", release_host_byte_chunks, 0, (hl_ParameterType[]){0}, hl_ReturnType_Int); hl_register_function_definition("ReturnNull", return_null, 0, (hl_ParameterType[]){0}, hl_ReturnType_Void); hl_register_function_definition("ReturnNullWithError", return_null_with_error, 0, (hl_ParameterType[]){0}, hl_ReturnType_Void); hl_register_function_definition("ReturnValueWithError", return_value_with_error, 0, (hl_ParameterType[]){0}, hl_ReturnType_VecBytes); diff --git a/src/tests/rust_guests/Cargo.lock b/src/tests/rust_guests/Cargo.lock index 1712a115e..108f75b9d 100644 --- a/src/tests/rust_guests/Cargo.lock +++ b/src/tests/rust_guests/Cargo.lock @@ -287,6 +287,7 @@ dependencies = [ "flatbuffers", "hyperlight-common", "hyperlight-guest-tracing", + "itertools", "serde_json", "tracing", ] @@ -921,4 +922,4 @@ dependencies = [ name = "zmij" version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" \ No newline at end of file +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/src/tests/rust_guests/simpleguest/src/main.rs b/src/tests/rust_guests/simpleguest/src/main.rs index f24ec4529..3753b276e 100644 --- a/src/tests/rust_guests/simpleguest/src/main.rs +++ b/src/tests/rust_guests/simpleguest/src/main.rs @@ -443,6 +443,20 @@ fn echo_guest_byte_chunks(data: Vec) -> Vec { data } +/// Read each segment's endpoints and release its owner before returning. +#[guest_function("ByteChunksLen")] +fn byte_chunks_len(data: Vec) -> i32 { + let mut len = 0; + + for chunk in data { + black_box(chunk.first().copied()); + black_box(chunk.last().copied()); + len += chunk.len(); + } + + len as i32 +} + static mut RETAINED_GUEST_CHUNKS: Option> = None; static mut RETAINED_HOST_CHUNKS: Option> = None; @@ -454,6 +468,16 @@ fn retain_guest_byte_chunks(data: Vec) -> i32 { len as i32 } +/// Return the retained bytes without releasing their buffers. +#[guest_function("ReadRetainedGuestByteChunks")] +fn read_retained_guest_byte_chunks() -> Vec { + // SAFETY: the guest is single threaded, so the static has no concurrent access. + #[allow(static_mut_refs)] + unsafe { + RETAINED_GUEST_CHUNKS.clone().unwrap_or_default() + } +} + #[guest_function("ReleaseGuestByteChunks")] fn release_guest_byte_chunks() -> i32 { // SAFETY: the guest is single threaded, so the static has no concurrent access. @@ -474,6 +498,16 @@ fn retain_host_byte_chunks(data: Vec) -> Result { Ok(len as i32) } +/// Return the retained bytes without releasing their buffers. +#[guest_function("ReadRetainedHostByteChunks")] +fn read_retained_host_byte_chunks() -> Vec { + // SAFETY: the guest is single threaded, so the static has no concurrent access. + #[allow(static_mut_refs)] + unsafe { + RETAINED_HOST_CHUNKS.clone().unwrap_or_default() + } +} + #[guest_function("ReleaseHostByteChunks")] fn release_host_byte_chunks() -> i32 { // SAFETY: the guest is single threaded, so the static has no concurrent access.