ADR 0024: Runtime Off-chain Logic (ROFL)
Component
Oasis Core, Oasis SDK
Changelog
- 2024-02-26: Notifications
- 2023-11-27: Initial draft
Status
Proposed
Context
Sometimes we may want the runtime compute nodes to run additional off-chain logic that communicates with the on-chain state securely (e.g. ensuring that the off-chain logic is being run by the same node operator, is properly attested when running in a TEE, etc.).
The off-chain logic may then perform non-deterministic and potentially expensive things (like remote HTTPS requests or complex local computation) and securely interact with the on-chain logic via transactions.
The main use case driving this proposal is support for running attested light client committees that read and verify information from other chains, then make this information available to Oasis runtimes with no additional trust assumptions.
Decision
While similar functionality can be implemented entirely independently on the application layer (and such solutions already exist), this proposal attempts to reuse the same security and attestation infrastructure that is already available for on-chain parts of the runtimes, specifically:
-
Compute nodes and runtime binary distribution and execution can stay the same as it has been for existing node operators that run the runtimes. Handling of the off-chain logic part should be done transparently if the runtime provides it.
-
Existing attestation, consensus and freshness proof flows can be leveraged for ensuring that the off-chain logic is running in a secure environment.
One important consideration is also whether to have the off-chain logic part of the same runtime binary or have it as a completely separate binary running in its own process. This proposal decides on the latter to ensure that the off-chain TCB is completely separate from the on-chain TCB. Given that the logic running off-chain can be much more complex and can interact with untrusted external services, ensuring this separation is important as a defense-in-depth measure.
The proposed architecture extends the composition of the runtime so that it now contains the following components:
-
Runtime On-chain Logic (RONL) is what has existed as the sole runtime component before this proposal. It contains the logic (and TCB) that is responsible for executing the deterministic on-chain logic of the runtime.
-
Runtime Off-chain Logic (ROFL) is an optional runtime component that may run in parallel with RONL and is part of its own TCB. It also uses the same general runtime framework and RHP, but instead of implementing the on-chain batch scheduling, execution and query methods, it only implements specific notification hooks that can trigger arbitrary actions.
Both RONL and ROFL are managed as independent runtimes by the Oasis Node as host, using the existing runtime host architecture. Failure of ROFL does not affect RONL which can proceed to run as usual.
Attestation
An assumption made in this proposal is that both RONL and ROFL components are developed and built together, by the same entity, and are part of the same release. This means that we can simplify attestation by making RONL being able to attest ROFL by being aware of its exact identity.
The idea is that during the release build process, ROFL is built first, its signer-independent identity (e.g. MRENCLAVE) is measured and included during compilation of RONL. The signer-dependent part of identity (e.g. MRSIGNER) is assumed to be the same for both and can be read from trusted CPU state (since it may not be available during the build process due to offline signing).
Alternatively, one can imagine a proposal where the ROFL identity is backed by some sort of on-chain governance process defined in the RONL component. Defining such a mechanism is outside the scope of this proposal.
The process for ROFL attestation proceeds as follows:
-
Remote Attestation. The normal runtime attestation flow is initiated by the host. As a result of this flow, the
node.CapabilityTEE
structure is generated which includes the remote attestation quote and additional data. -
Node Endorsement. The host verifies the
node.CapabilityTEE
structure and if deemed correct, it signs it using the node's identity key and the following domain separation context:oasis-core/node: endorse TEE capability
The signature is stored in a new structure
EndorsedCapabilityTEE
which is defined as follows:type EndorsedCapabilityTEE struct {
// CapabilityTEE is the TEE capability structure to be endorsed.
CapabilityTEE CapabilityTEE `json:"capability_tee"`
// NodeEndorsement is the node endorsement signature.
NodeEndorsement signature.Signature `json:"node_endorsement"`
} -
Updating Node-Endorsed CapabilityTEE in ROFL. The
EndorsedCapabilityTEE
is sent to ROFL to be stored and available for establishing secure EnclaveRPC sessions. -
RONL Verification. When establishing a new session with RONL, the endorsed TEE capability is presented during session establishment. RONL verifies the quote, ensures the enclave identity is one of the known identities set at compile-time and verifies the node endorsement against the locally known node identity (both RONL and ROFL must be from the same node).
If all the checks pass, a secure EnclaveRPC session is established.
This flow needs to be repeated whenever RAK changes for any reason and also periodically to ensure freshness (consistent with the quote policy configured for the runtime in the consensus layer).
Updates to the ORC Manifest
The ORC manifest is extended with a field that can specify extra components which currently include ROFL binaries in a similar way as we already support regular runtime binaries (e.g. specifying the executable and SGX metadata).
The manifest is updated as follows:
// Manifest is a deserialized runtime bundle manifest.
type Manifest struct {
// ... existing fields omitted ...
// Components are the additional runtime components.
Components []*Component `json:"components,omitempty"`
}
// ComponentKind is the kind of a component.
type ComponentKind string
const (
// ComponentInvalid is an invalid component.
ComponentInvalid ComponentKind = ""
// ComponentRONL is the on-chain logic component.
ComponentRONL ComponentKind = "ronl"
// ComponentROFL is the off-chain logic component.
ComponentROFL ComponentKind = "rofl"
)
// Component is a runtime component.
type Component struct {
// Kind is the component kind.
Kind ComponentKind `json:"kind"`
// Name is the name of the component that can be used to filter components
// when multiple are provided by a runtime.
Name string `json:"name,omitempty"`
// Executable is the name of the runtime ELF executable file.
Executable string `json:"executable"`
// SGX is the SGX specific manifest metadata if any.
SGX *SGXMetadata `json:"sgx,omitempty"`
}
The top-level executable
and sgx
fields are supported for backwards
compatibility and implicitly define a new Component
of kind ComponentRONL
.
Updates to the Runtime Host Protocol
This proposal includes some non-breaking updates to the Runtime Host Protocol in order to support the ROFL component, as follows:
-
Consensus Block Notification. No updates are required to facilitate notifications about consensus layer blocks as this is already handled as part of the existing RHP flow. The only change is that for ROFL, these notifications invoke a hook that can be implemented by the runtime.
-
Runtime Transaction Submission. A new method
HostSubmitTx
is introduced which allows ROFL to submit transactions to the runtime. It works by queueing the transaction in the transaction pool (local queue) for later scheduling.type HostSubmitTxRequest struct {
// RuntimeID is the identifier of the target runtime.
RuntimeID common.Namespace `json:"runtime_id"`
// Data is the raw transaction data.
Data []byte `json:"data"`
// Wait specifies whether the call should wait until the transaction is
// included in a block.
Wait bool `json:"wait,omitempty"`
// Prove specifies whether the response should include a proof of
// transaction being included in a block.
Prove bool `json:"prove,omitempty"`
} -
Notify Registration. A new method
HostRegisterNotify
is introduced which allows ROFL to register to be notified by the host when specific events occur. Note that delivery of these notifications is best effort as a dishonest host may withold notification delivery or generate spurious notifications.Registering for notifications overwrites any previous configuration.
type HostRegisterNotifyRequest struct {
// RuntimeBlock subscribes to runtime block notifications.
RuntimeBlock bool `json:"runtime_block,omitempty"`
// RuntimeEvent subscribes to runtime event emission notifications.
RuntimeEvent *struct {
// Tags specifies which event tags to subscribe to.
Tags [][]byte `json:"tags,omitempty"`
} `json:"runtime_event,omitempty"`
} -
Notification Delivery. A new method
RuntimeNotify
is introduced which allows the host to deliver event notifications based on previously registered notifiers.type RuntimeNotifyRequest struct {
// RuntimeBlock notifies about a new runtime block.
RuntimeBlock *roothash.AnnotatedBlock `json:"runtime_block,omitempty"`
// RuntimeEvent notifies about a specific runtime event being emitted.
RuntimeEvent *struct {
// Block is the block header of the block that emitted the event.
Block *roothash.AnnotatedBlock `json:"block"`
// Tags are the matching tags that were emitted.
Tags [][]byte `json:"tags"`
} `json:"runtime_event,omitempty"`
} -
RONL-ROFL Communication. The existing EnclaveRPC is reused to facilitate the communication between the two components if/when needed. For this purpose the endpoint identifier
ronl
is made available in the ROFL host method handler to address the RONL component. -
Updating Node-Endorsed CapabilityTEE in ROFL. A new method
RuntimeCapabilityTEEUpdateEndorsementRequest
is introduced which allows the node to refresh theEndorsedCapabilityTEE
for ROFL.type RuntimeCapabilityTEEUpdateEndorsementRequest struct {
// EndorsedCapabilityTEE is an endorsed TEE capability.
EndorsedCapabilityTEE node.EndorsedCapabilityTEE `json:"ect"`
}
Updates to EnclaveRPC RAK Binding
Version 2 of the RAKBinding
structure is introduced for establishment of
EnclaveRPC sessions, as follows:
pub enum RAKBinding {
// ... previous versions omitted ...
/// V2 format which supports endorsed CapabilityTEE structures.
#[cbor(rename = 2)]
V2 {
ect: EndorsedCapabilityTEE,
binding: Signature,
},
}
Additionally, the relevant EnclaveRPC session implementation is updated to facilitate thew new authentication mechanism via endorsed TEE capabilities and the session demultiplexer is updated to support authentication policies on incoming connections.
Updates to the Runtime Host Sandbox
This proposal updates the runtime host sandbox to support optionally allowing external network requests. These are then allowed only for the ROFL component (if any is available for a runtime).
The following modifications are required:
-
When setting up the Bubblewrap sandbox,
--share-net
is passed to share the network namespace with the sandboxed runtime. All other namespaces are still unshared. -
The runtime loader is modified to accept an additional argument
--allow-network
which then changes the usercall extension to pass through any address passed in theconnect_stream
handler.
Configuration
ROFL may require additional configuration which it may do through one of several ways:
-
On-chain Configuration. Configuration for the ROFL component may be stored in on-chain state. ROFL would then query the current configuration and apply it locally.
-
Local Per-Node Configuration. In case some per-node configuration is required (e.g. to allow the node operator to override a default), the existing runtime local configuration mechanism can be used where configuration is provided as part of the RHP handshake. All configuration for ROFL should be contained under the
rofl
configuration key.
Untrusted Local Storage
ROFL may utilize the existing untrusted node-local storage to store things like
sealed data local to the node. This store is shared between RONL and ROFL, but
all ROFL keys are transparently prefixed by rofl.
on the host such that only
RONL can see (but not necessarily read) ROFL's keys but not vice versa.
Updates to the Oasis SDK
A convenient way to develop ROFL modules alongside the on-chain support functionality should be implemented in the Oasis SDK, including a convenient way for ROFL to submit runtime transactions in a way that can be verified on-chain as coming from a specific node/runtime instance.
Consequences
Positive
-
Oasis runtimes can easily be extended with arbitrary off-chain logic that can securely interact with on-chain functionality.
-
Node operators do not need to perform much additional configuration in order to support the new off-chain logic.
Negative
- Additional complexity is introduced to the Runtime Host Protocol and to the node binary.