🔁

Specifications for end-to-end HTTP retrieval and deserialization of popular content-addressed data

What is this?

A minimal set of specifications we need to have to enable radical interop thanks to HTTP. This is the golden path informed by and benefitting projects like IPFS Chromium, or any light integration of content-addressed data that does not want to deal with details of P2P (including mobile).

What do we need to do?

For IPFS to be a real open protocol, we need to have useful (implementable) specifications for doing data retrieval using nothing more than HTTP.

This means making specs.ipfs.tech a go-to resource for IPFS implementers and developers who are integrating IPFS within their HTTP ecosystem.

Why this is a good idea?

The main value of having a well specified golden path for end-to-end HTTP retrieval is to reduce integration cost and cognitive overhead / coordination complexity.

  • Various parts of the stack can be hidden behind an HTTP-based black box and delegated to different software, teams, or even third-party companies.
  • Specifications remove ambiguity, make it clear what is mandatory, optional, or unspecified.
    • It’s really important to remove guesswork and explicitly say when an area is unspecified (rather than just leaving it implied by the absence of text in the specification).
  • Creates clear divergence and convergence points

We have a historical evidence of specifications-first approach working, and creating long-lasting impact across ecosystem. For example:

  • CIDv2 idea was discussed in the public, and not rushed, and eded up getting dropped
  • HTTP Gateway specifications became the basis for both “HTTP transport “ and ipfs:// in Brave/Chromium, and is the only alternative to bitswap. Is it perfect? no. But it exists, including conformance tests, unlike other hypothetical alternatives we’ve discussed for over a year.
  • IPNS Specifications not only helped IPFS Chromium project, but we got useful contribution back, where specification detail did not match reality, and now they do.

Instances where this would have been helpful

  • IETF and other standard bodies
    • Lack of end-to-end specifications makes conversations difficult or simply impossible.
    • The fact that different libraries have different behaviors, and we have no source of truth on what is mandatory, what is optional, and what is a bug, makes it impossible for non-PL employees to reverse-engineer things and come up with bug-free specification
  • IPFS Chromium
    • Ability to fetch content over HTTP is a hard requirement for this project – we are reusing HTTP client present in the chromium codebase.
    • If IPFS nodes listened on HTTP next to bitswap, this project could add ability to discover new HTTP peers that are known to have CID would enable semi-p2p without including full libp2p, and retrieve the data using the same HTTP API as from public gateways.
  • Project Rhea
    • Having clear specification and compliance tests allowed multiple teams to align.
    • Saturn/Lassie want to implement a generic support for HTTP peers, and prioritize them over bitswap/graphsync.
  • Boost
    • Exposing HTTP interface for retrieval is often preferable by SPs.
    • The lack of specifications around HTTP transport creates a surface for bugs and confusion when it comes to implementation details, even for PL team members and our own internal projects (example PR).
  • dagHouse
    • Bitswap costs produced by Project Rhea/Saturn are greatly reduced thanks to HTTP transport.
    • If other IPFS peers were able to use HTTP instead of bitswap, the cost of operating the service would go down even further.

What’s out of scope current (and what we’d take on here with more time)?

Specifications for building true P2P implementation compatible with Kubo and Helia

What would success look like?

What would “good impact” look like?

  • We have end-to-end specifications for retrieving data from IPFS
  • Compliance with the majority of IPFS specifications can be tested with tools that speak HTTP
  • IPFS Implementations appear without any involvement from IPFS Shipyard.

What measurements do we need to get in place before doing the work so we can measure the changes?

  • Monitor User-Agents on ipfs.io, dweb.link, trustless-gateway.link, and ipfs-delegated.dev.
  • Monitor use of ipfs/gateway-conformance and specific deployment method (prebuilt binary, Docker image, git repository, or related Github Action).