Non-"dag-cbor" IPLD

[Stebalien]

Background

Currently, we have quite a few places that simply assume that all IPLD data is dag-cbor.

In the sdk, send assumes dag-cbor:

• ref-fvm/sdk/src/send.rs

  Line 29 in f7b51ba

  sys::ipld::block_create(DAG_CBOR, params.as_ptr(), params.len() as u32)?

• ref-fvm/sdk/src/send.rs

  Line 38 in f7b51ba

  return_codec: _, // assume cbor for now.

In the builtin actors, the "trampoline" assumes dag-cbor:

• https://github.com/filecoin-project/builtin-actors/blob/91ff9fa5c057359174424eb59b34965b10d19e90/runtime/src/runtime/fvm.rs#L556
• https://github.com/filecoin-project/builtin-actors/blob/91ff9fa5c057359174424eb59b34965b10d19e90/runtime/src/runtime/fvm.rs#L577

On-chain, the Message and Receipt types assume dag-cbor (simply because they don't allow us to specify anything else):

• ref-fvm/shared/src/message.rs

  Line 23 in f7b51ba

  pub params: RawBytes,

• ref-fvm/shared/src/receipt.rs

  Line 13 in f7b51ba

  pub return_data: RawBytes,

However:

• For FEVM, we'd like to support "raw" IPLD blocks because FEVM calls use untyped byte arrays.
• In M2.2, wasm actors will be raw blocks as well (most likely, at least).
• We should probably add support for bare "cbor" (not "dag-cbor") for message parameters as message parameters generally aren't allowed to link to anything.
• The system cares about the codec:
  • It needs to know the codec so it can perform reachability analysis.
  • Actors themselves need to know how to decode their parameters. Right now they can just assume "cbor", but we'd like to allow raw as well to avoid having to wrap every object in a bit of CBOR.

Changes

We have three categories of changes:

• On-chain messages/receipts.
• FVM SDK/APIs.
• "Embedded" blocks.

Messages/Receipts

Currently, on-chain messages don't have a field to specify the codec. So the simplest solution here is to just say that all on-chain message parameters must be "cbor" (not "dag-cbor", just "cbor", because message parameters can't link to anything).

However, receipts aren't so simple: if an actor happens to be invoked from off-chain, we don't want that invocation to fail because the target actor returns a "raw" response. Luckily, we're already changing the receipt structure, so including an additional codec along with the return value shouldn't be a huge issue.

Options:

• change the message format to include a codec (allowing "cbor" and "raw").
• change the receipt format to include a codec.
• change the receipt format to link to the returned value (supports both codecs and larger values).
  • Automatically use "inline" CIDs?

SDKs/APIs

We primarily need to replace most uses of RawBytes with some form of IpldBlock (filecoin-project/builtin-actors#758) abstraction. I.e., something that actually caries the codec.

To do this, we'll need to modify:

• The sdk's send (both params and return).
• The runtime's tranpoline (again, both params and return).

Really, actors shouldn't care much about the codec and should support both raw and cbor implicitly.

See the issue linked above for details.

Embedded

Finally, we have some cases where we embed some message/block in another message/block.

• Cron callbacks.
• Init parameters
• Multisig messages.

We currently do this by just encoding it to bytes and embedding it in a byte array with RawBytes (which assumes that the bytes are actually CBOR). However:

• This doesn't allow for, e.g., "raw" blocks.
• This approach will affect reachability analysis as the system won't be able to see that these embedded messages have embedded links.

Options:

• Replace embedded RawBytes with CIDs wherever possible.
  • Allow them to be inlined?
• Require dag-cbor, but store it as an inline "value" instead of as bytes?

The CID approach may have an additional hashing cost in some cases, but, IMO, it's still the best option. I spent some time thinking about potentially supporting "runtime" CIDs for this case (filecoin-project/FIPs#482) but the cost of hashing is so low, it's probably just not worth it.

Alternatives

Honestly, the best alternative is... require CBOR everywhere. This is an option, it's just a bit unfortunate as all raw byte arrays (wasm bytecode, evm bytecode, evm parameters/returns, etc.) would need to be wrapped in a bit of CBOR. But if this problem becomes too gnarly... this is what we'll stick with for M2.1.

A slightly softer approach would be to say that all parameters and return values must be CBOR, but we can still put other types of blocks. This would let bytecode stay as "raw" (and would make it easier for us to allow raw parameters/return values in the future).

[anorth]

For FEVM, we'd like to support "raw" IPLD blocks because FEVM calls use untyped byte arrays

I assume this is mainly about parameter and result blocks for internal sends. My understanding at present is that top-level messages will remain as-is, with the EVM message serialised inside the params. For that case, the CBOR byte array implied by RawBytes in the Message seems ok (ditto Receipt). Requiring Message to be [dag-]CBOR is already the spec, and is still ok as far as I see.

I don't yet see sufficient motivation to change receipt return data to be anything other than CBOR either. Are we just talking about the 1-byte type tag identifying a byte array, in case an actor wants to return some opaque bytes?

We should probably add support for bare "cbor" (not "dag-cbor") for message parameters as message parameters generally aren't allowed to link to anything.

I'm not quite sure what this means in practice. Certainly actors should be able to accept CIDs as members of a parameters structure, without any implication of that linking to something accessible on-chain. Does non-dag CBOR mean CBOR but prohibiting the CID field tag (so actors will work around it by using a byte array instead, and casting back to a CID on chain)?

because message parameters can't link to anything

Why not? As in, why not allow CIDs in parameters that resolve to something that's not in chain state? What goes wrong? I think we should revisit this before expanding restrictions here. It would be nice just to allow CIDs and have other systems just deal with the fact that they may not be resolvable, but I could be convinced otherwise. I think there's already a hack around this somewhere in the piece CIDs referenced in deal proposals.

I realise a potential complication is that it's difficult to express a query (e.g. IPLD selector) that includes exactly the chain and doesn't attempt to follow links from params, but is that a knockout blow? What are the other issues?

[the vm] needs to know the codec so it can perform reachability analysis

I understand this for blocks saved in state. Can you convince me it applies to params too? It might have something to do with passing references to state (as capabilities) between actors? Is the plan for this written up (I've forgotten details)?

We currently [embed blocks] by just encoding it to bytes and embedding it in a byte array with RawBytes (which assumes that the bytes are actually CBOR).

I know there's a comment on RawBytes that mentions CBOR, and it has helper methods etc, but as I user I just think of this as the thing that wraps a byte array and prepends the "byte-array" CBOR tag when my opaque bytes are embedded inside a CBOR structure. It doesn't constrain what I put in those bytes. I consider it up to the receiver to know how to decode the bytes (and it can use the RawBytes helper methods if it happens to be CBOR). This knowledge is embedded in, e.g. the built-in actors trampoline. So AFAICT it does support raw blocks, just not with convenient helper methods.

I don't see the problem for reachability analysis yet. CIDs that are encoded inside byte arrays embedded in another block won't be considered reachable. Passing a block-visibility capability would require a non-wrapped CID.

The change I would advocate here is changing RawBytes to mean just a wrapper around bytes, without implying they are encoded in any specific way. There might additionally be some call for a structure that carries both bytes and the self-describing encoding spec (an inline CID or an IpldBlock type), but many cases won't need that.

SDKs/APIs
Mostly 👍 , but I don't agree with "actors shouldn't care much about the codec and should support both raw and cbor implicitly". I think actors should be very concrete about the data they are expecting, including its encoding. Frameworks like the built-in Runtime may well abstract it away from actual actor implementations, but that's essentially developer discretion.

[Stebalien]

I assume this is mainly about parameter and result blocks for internal sends.

Also for state. We already used "raw" blocks for actor code, and plan to use it for FEVM byte code.

I assume this is mainly about parameter and result blocks for internal sends. My understanding at present is that top-level messages will remain as-is, with the EVM message serialised inside the params. For that case, the CBOR byte array implied by RawBytes in the Message seems ok (ditto Receipt). Requiring Message to be [dag-]CBOR is already the spec, and is still ok as far as I see.

The message is CBOR. We could leave the message parameters as CBOR, but the receipt is tricky for the reasons mentioned above. The issue is that internal sends and external sends go through the same interface, so an external send might result in a non-CBOR return.

The system could auto-convert from raw to CBOR, but I'd really rather not touch that if at all possible.

We should probably add support for bare "cbor" (not "dag-cbor") for message parameters as message parameters generally aren't allowed to link to anything.

I'm not quite sure what this means in practice. Certainly actors should be able to accept CIDs as members of a parameters structure, without any implication of that linking to something accessible on-chain. Does non-dag CBOR mean CBOR but prohibiting the CID field tag (so actors will work around it by using a byte array instead, and casting back to a CID on chain)?

All dag-cbor is valid CBOR, it's just that the system wouldn't interpret and/or care about any CIDs present. At the end of the day, this wouldn't really mean anything in practice, it's just a clean way to distinguish between "CIDs in this object are reachable" and "CIDs in these parameters are not".

In general is that we assume that all CIDs in parameters, state, and return-values are "reachable". This lets us pass sub-states between actors, and link them into the actor state without worrying about anything.

Why not? As in, why not allow CIDs in parameters that resolve to something that's not in chain state? What goes wrong? I think we should revisit this before expanding restrictions here. It would be nice just to allow CIDs and have other systems just deal with the fact that they may not be resolvable, but I could be convinced otherwise. I think there's already a hack around this somewhere in the piece CIDs referenced in deal proposals.

We'll, we're not currently checking anything. But the thought was to either:

1. Current plan: Whitelist different CID "types" (not great, but works).
2. Future maybe: Use the "cbor" codec of the "dag-cbor" for those blocks?

The problem is deterministically deciding between something being "resolvable" and something that just happens to exist in the client's blockstore. There are other solutions (like keeping reachable sets in state) but saying "if you link to it, it's reachable" is by far the simplest.

Note: we could have some form of "tainting'. I.e., we could say that all CIDs in state must be reachable, but CIDs in parameters aren't assumed to be reachable unless they came from state. But IMO, that doesn't buy us much and is complicated.

[the vm] needs to know the codec so it can perform reachability analysis

I understand this for blocks saved in state. Can you convince me it applies to params too? It might have something to do with passing references to state (as capabilities) between actors? Is the plan for this written up (I've forgotten details)?

Yes, it has to do with passing state. Basically, I'd like to be able to pass an entire IPLD tree from state from actor A to actor B, and have actor B read through that tree (potentially linking it into their own state).

In terms of write-ups, there's https://www.notion.so/pl-strflt/FVM-Design-Space-PUBLIC-ce9771fe18ea461ebb406716607d5790#fcd0718f66b7481fb020691f338e8356, but a lot of the details have been punted to M2.2.

The change I would advocate here is changing RawBytes to mean just a wrapper around bytes, without implying they are encoded in any specific way. There might additionally be some call for a structure that carries both bytes and the self-describing encoding spec (an inline CID or an IpldBlock type), but many cases won't need that.

So, the issue here is that I'd like to be able to pass an entire state-tree into an actor in construction. To do that, I need the system to understand that I'm passing said tree so it can reason about what CIDs should be reachable by the target actor and what CIDs shouldn't.

The only form of reachability that wouldn't have this requirement is global reachability. I.e., if we consider a CID anywhere in the state-tree reachable by any actor. However, that:

1. Requires very careful accounting/garbage collection.
2. Has some nasty "spooky-action-at-a-distance" side-effects.

[anorth]

All dag-cbor is valid CBOR, it's just that the system wouldn't interpret and/or care about any CIDs present. At the end of the day, this wouldn't really mean anything in practice, it's just a clean way to distinguish between "CIDs in this object are reachable" and "CIDs in these parameters are not".
In general is that we assume that all CIDs in parameters, state, and return-values are "reachable". This lets us pass sub-states between actors, and link them into the actor state without worrying about anything.

So I think we fundamentally need to support both reachable and non-reachable here, with explicit representation of intent. Perhaps one can be the default for CID and the other explicitly worked-around (e.g. raw byte array that the receiver has to interpret), but we should have clear patterns for both.

But we still need to be able to pass raw bytes, too.