Welcome pre-existing content addressed data via supporting large blocks

We want to support the data that is already content address, but we can’t get there without lifting the block limit. The real issue we want to solve is https://discuss.ipfs.tech/t/ipld-and-ipfs-a-pitch-for-the-future/15095

What is this?

Popular IPFS implementations like Kubo and Helia support the wide array of data that is already content addressed by developing sufficient specs and tooling to be able to support incrementally verifiable large blocks. The utility of IPFS moves closer new userbases, with content-addressable data from Docker containers to Blockchains state-trees now being viable for distribution with IPFS protocols without requiring any additional or duplicative processing.

Specific examples:

  • It is possible for Iroh to have meaningful interop with major existing IPFS tooling (e.g. kubo, helia, boost).
    • Even if the integration isn’t completed, making meaningful progress such that there are no non-implementation specific blockers is great progress.
  • Large raw blocks can operate as files just like small raw blocks
  • It is possible to support IPLD DAGs with large blocks even if those blocks are not raw
    • Note: this support is less critical than the others given that once we have raw blocks people can build their own data processing tooling.

Why is this a good idea?

What do we need to do?

  • Specification for storing and transporting incrementally verifiable large blocks in CAR files
    • Unlocks incrementally sending large raw blocks (i.e. files) with the Trustless HTTP Gateway
    • Unlocks indexing (and serving) of trustless CARs by Boost (for blocks smaller than a single piece)
  • boxo and helia based implementations for being able to fetch data (including large blocks) without exposing out-of-memory errors or attack vectors
    • Ideally this is supportable within kubo (and bifrost-gateway). While supporting those binaries is not strictly necessary, users who were hoping for iroh to be compatible with kubo or ipfs.io will be unhappy without this.