Content Routing

Confidence Level:Brainstorm

Content Routing is the mechanism by which parties looking for content in the network identify parties providing that content

Dependencies

Name	Kind	APIs
Chain	optional	default
Reputational Index	optional	default

Roadmap

V0: Existing or 3 months

At the moment there are several in progress proposals and prototypes for content routing:

• ChainSafe built a prototype of content-routing based on GossipSub
• Pegasys has proposed a DHT based content routing system
• ResNetLab is currently researching major changes to IPFS's DHT

All of the current prototypes diverge quite a bit from the API interface described below.

V0.5: 6 months

The simplest path towards building a functioning prototype for content-routing for the retrieval market is to build a global index of content (i.e. a gateway). There are some very clear directions we can go to assemble such an index: - we can examine the Filecoin chain to look at current deals - we can encourage miners to use the ProvideCID/Provide DAG API calls to post CIDs for data they are storing - we can create retrieval deals for payload CIDs discovered on the Filecoin chain, then add other CIDs contained in the retrieved piece to the index

V1: 1 Year

If we can build a functioning index/gateway, as true "retrieval miners" come online, they can also use the existing APIs continue to augment the index above.

V2: Future

An index will likely hit scaling limits fairly quickly. In a year, hopefully ResNetLab's content routing efforts will produce clear next steps for more scalable solutions

Preliminary API

This API is purely speculative, based on the likely needs for people seeking content they can retrieve quickly

TIP

ResNetLab is working on a new content routing system. The new interface will be very general -- allowing storage of fairly arbitrary data on composoble routing platforms (DHT, index, etc). When their work done, we will either switch to use their interface directly or build the interface described below on top of it.

package contentrouting

import (
	"context"

	"github.com/filecoin-project/retrieval-market-spec/docs/components/exchange"
	"github.com/ipfs/go-cid"
	"github.com/ipld/go-ipld-prime"
	"github.com/libp2p/go-libp2p-core/peer"
)

type ExchangeOption struct {
	// Name of exchange protocol
	ProtocolName exchange.ExchangeProtocolName
	// Offer parameters
	Parameters ipld.Node
}

// GeographicRegion content is located in
// TODO: what should this look like?
type GeographicRegion ipld.Node

// ContentProvider is someone who is providing the content requested
// on the given exchange and parameters
type ContentProvider struct {
	Peer         peer.ID
	ProtocolName exchange.ExchangeProtocolName
	Parameters   ipld.Node
	Region       GeographicRegion
}

// ContentRoutingAPI provides mechanisms to find providers of content
type ContentRoutingAPI interface {

	// Search for peers who are able to provide the given cid
	//
	// When count is 0, this method will return an unbounded number of
	// results.
	FindProvidersCID(ctx context.Context,
		cid cid.Cid,
		desiredRegion GeographicRegion,
		acceptedExchanges []exchange.ExchangeProtocolName,
		count int) (<-chan ContentProvider, error)
}

// DAGContentRoutingAPI provides mechanisms to find providers of content by
// CID+Selector
type DAGContentRoutingAPI interface {
	ContentRoutingAPI
	// Search for peers who are able to provide the given DAG, as expressed
	// by CID + Selector
	//
	// When count is 0, this method will return an unbounded number of
	// results.
	FindProvidersDAG(ctx context.Context,
		cid cid.Cid,
		selector ipld.Node,
		desiredRegion GeographicRegion,
		acceptedExchanges []exchange.ExchangeProtocolName,
		count int) (<-chan ContentProvider, error)
}

// ContentProvidingAPI offers tools for providers to annouce their content to
// the network
type ContentProvidingAPI interface {
	// ProvideCID is called by a provider to announce that they are offering
	// the given content to a network, on the given exchange protocols
	// and parameters
	ProvideCID(ctx context.Context,
		cid cid.Cid,
		regions []GeographicRegion,
		exchangeOptions []ExchangeOption) error
}

type PrioritizedCID struct {
	Priority uint64
	Cid      cid.Cid
}

// BatchConcentProvidingAPI offers tools for providers to announce several CIDs at once,
// including identifying those that are most likely semantically important
type BatchConcentProvidingAPI interface {
	ContentProvidingAPI

	// ProvideCIDs is called by a provider to announce that they are offering
	// a set of CIDS to the network
	ProvideCIDs(ctx context.Context,
		prioritizedCIDs []PrioritizedCID,
		regions []GeographicRegion,
		exchangeOptions []ExchangeOption) error
}

// DAGContentProvidingAPI offers tools for providers to annouce their content to
// the network by CID+Selector
type DAGContentProvidingAPI interface {
	ContentProvidingAPI

	// ProvideDAG is called by a provider to announce that they are offering
	// a DAG, expressed by a CID+Selector, to a network, on the given exchange
	//  protocols and parameters
	ProvideDAG(ctx context.Context,
		cid cid.Cid,
		selector ipld.Node,
		regions []GeographicRegion,
		exchangeOptions []ExchangeOption) error
}