Filecoin Retreival Market

# Detailed Roadmap

This outlines everything we aim to build for each milestone in the roadmap

TIP

For an overview of what each milestone means, see Roadmap Overview

# V0 Milestone: Existing Or Finished In 3 Months

Roles

Content Provider

There is no current mechanism for a content provider to participate in the retrieval market, other than to use a storage miner and hope that they serve retrievals quickly.

Marketplace Provider

There is today no MarketPlace provider in the Filecoin Retrieval Market. The immediate task in front of us to enable a MarketPlace is to build the most critical indexes: first the Reputational Index (for reputation) and second the initial Content Routing index to help Filecoin users find content.

Payment Provider

There is currently no way for someone who wants to participate as a payment provider in the retrieval market to make money doings so. Instead, payments are run directly on client and provider nodes.

Retrieval Client

Currently, the only instance of a retrieval client is a Lotus Node, or a Lotus Lite Node combined with a Lotus Gateway. This makes for a very high bar to entry. Running a full Lotus Node is often impossible even on a powerful computer, and even running Lotus Lite you're cut off from much of the IPFS stack unless you also run PowerGate.

Retrieval Provider/Miner

Currently, the only way to be a Filecoin Retrieval Miner is to be a Filecoin Storage Miner, an incredibly high bar to entry that among other things doesn't incentivize doing the job of retrieval mining well (given storage rewards produce greater profit).

Components

Content Distribution

The storage API is arguably somewaht similar to this distribution API, except the providers make the offers and money is paid for retrieval, not storage.

Filecoin Chain

Lotus already contains two implementations of the above API:

  • A Lotus Full Node
  • The Lotus Gateway API, which wraps a JSON-RPC to the full node

Content Bid Index

There is no current content bid index for retrieval in Filecoin. Operation Dumbo Drop vaguely resembed a content bid index for Filecoin storage.

Data Transfer

Data Transfer has been implemented, but only supports a single Transport mechanism. Moreover, the current implementation has architectural issues, as documented in go-data-transfer#158

Content Routing

At the moment there are several in progress proposals and prototypes for content routing:

All of the current prototypes diverge quite a bit from the API interface described below.

Exchange

Retrieval Protocol V1 is the current implementation of Exchange in Filecoin. It operates through a system of incremental payments, with the data provider sending small chunks then asking for payment before sending more.

Payment Channel Manager

A Payment Channel Channel manager is currently implemented in Lotus and used for retrievals. It works, but it does have one significant pain point: as written, it always creates a blocking chain message to deposit funds in a channel at the beginning of a retrieval. This prevents performant retrieval

Reputational Index

Not having a reputational index has made retrieval in the current Filecoin network largely unusable. During the next three months, Protocol Labs will likely build a prototype reputation index in the form of a retrieval deal bot. The bot will make deals with storage miners and begin to asses the reliability of different miners.

Transport

Currently, a single Transport interface is imported for Data Transfer: Graphsync. The interface does not look exactly like the API described below, and is somewhat Graphsync specific.

Wallet

Lotus implements a LocalWallet based on a local Keystore, and a remote wallet that works on a JSON-RPC endpoint. It's not clear whether the remote wallet contains well fleshed out authentication controls

# V0.5 Milestone: Six Months

Roles

Marketplace Provider

Building out the IPFS Gateway to reliably locate and serve Filecoin content will lay the foundation for a robust MarketPlace Provider in the future. We have a number of tasks to get there:

  • identifying content in the network
  • benchmarking miners
  • unlocking these indexes not just on the IPFS Gateway but to regular Filecoin or IPFS clients.

Payment Provider

The Filecoin blockchain contains primitives to support more complex payment channel schemes like "hub and spoke". However, none of these primitives have ever fully been used on mainnet. Before any work towards more robust payment provider system begins in earnest, the first step is exercise the Filecoin chain primitives and determine what, if anything, needs to change to support complex systems of payment in the retrieval market. If changes are required, they would have to go through a FIP process and an actors upgrade, so it makes sense to start early.

Retrieval Client

Why's Estuary is a prototype instance of a retrieval client taken out of the Lotus stack. It will be interesting to see where this heads in the future

Components

Filecoin Chain

It's not clear whether the Lotus Gateway API works properly across an open network, and this may be important in a case where several parties in the retrieval market are not running a full lotus node.

Content Bid Index

As a starting point, a single global content bid index seems straightforward to build. PL could build one itself, or offer a dev grant to a third party.

However, the "content distribution" side of the retrieval market needs further research. We are competing with web 2 CDN providers, so we need to make sure our offerings are comparable.

Data Transfer

Team Ignite is implementing a second transport for data transfer as part of the F3 Project. As part of this, Team Ignite will need to improve Data Transfer to support multiple protocols, and determine how Data Transfer will choose between protocols.

Content Routing

The simplest path towards building a functioning prototype for content-routing for the retrieval market is to build a global index of content (i.e. a gateway). There are some very clear directions we can go to assemble such an index: - we can examine the Filecoin chain to look at current deals - we can encourage miners to use the ProvideCID/Provide DAG API calls to post CIDs for data they are storing - we can create retrieval deals for payload CIDs discovered on the Filecoin chain, then add other CIDs contained in the retrieved piece to the index

Exchange

The proposed API below represents a slightly more semantic and clean design than the current implementation. It may make sense to conform to the proposed API in preparation for adding other exchange protocols in the future.

Payment Channel Manager

The API specified below is designed to remove some of the ambiguity in the current payment channel manager, and to efficiently unlock paying for lots of retrievals ahead of time, so that they can proceed in batchs without being blocked by the chain. This fulfills the original purpose of a payment channel.

Reputational Index

Once we have a prototype index, we can make it available as an API as well as a web portal, and begin to integrate it into retrieval clients.

Transport

Team Ignite is implementing a second transport for data transfer as part of the F3 Project: STP, or simple transport protocol. To support both protocols, the Transport interface will need to be made fully protocol nuetral. The API below is a suggested revision to the interface.

# V1 Milestone: 1 Year

Roles

Content Provider

Building a successful MVP for content providing in the retrieval market is likely a year long project. All of the following much be in place for content providing to work:

  • We need to build first iterations of the Content Bid Index and Content Distribution components. Both of these components are proposed in this document as brainstorms. They will likely need revision along the way.
  • We need to deliver a 3 way exchange mechanism, where a miner sends data to a client, in exchange for the client verifying to the content provider they received the data, in exchange for the content provider sending payment to the miner. Making sure we do this in a way so that all parties receive fair exchange in a trustless environment may not be easy.
  • We need to find an appropriate piece of software in which to run the content providing system. It's possible go-IPFS is that system. Alternatively, it's possible the first customers of content providing are IPFS pinning serves that want to pay to insure the content people pay them to store is served quickly.

All of this makes for a lot of work for a single year. However, enabling content providers is key to the economics of a retrieval market CDN. People browsing the web expect to browse for free in most cases. To deliver a functioning retrieval market, we need to unlock payments for parties other than the client.

Marketplace Provider

The remaining task for the "MarketPlace MVP" is to build a Content Bids Index, and to provide a way for others to run other Marketplaces to begin to lay the foundations for decentalization.

Payment Provider

The value of more complex payment systems is that they scale better than a system where every time two parties transact they setup a direct channel. The moment when this scaling becomes a neccesity for the retrieval market is a bit hard to predict. In the immediate sense, more can be done to unblock the existing simple PaymentChannelManager. At the same time, if retrievals are ever to compete in the CDN space, we simply can't block on chain operations, ever, in the context of retrieval.

Retrieval Client

Running retrieval through go-ipfs might produce a smaller, more functional retrieval client. To do this we would:

  • Put go-data-transfer in go-ipfs
  • Connect go-ipfs to index-style Content Routing for Filecoin
  • Implement clear rules for negotiation between IPFS transfer vs Exchange based paid retrieval. Figure out how if at all IPFS will interface with a Reputational Index to determine the most reputable miners to pull from.
  • To keep go-ipfs open and nuetral, we may want leave Filecoin specific exchange out of the go-ipfs binary. Instead, we could allow IPFS to communicate with any Exchange Client running locally but not in the same process (enabling potentially non-Filecoin paid retrievals)
  • The smallest version of this second process consists of an Exchange Client and a Wallet, talking to a remote Payment Provider, if such a provider exists.
  • A slightly larger version would run a Payment Channel Manager locally but talk to a remote Chain

Alterative 1: Myel is exploring support for retrieval in IPFS through a plugin that runs Data Transfer and a custom Exchange/Content Routing protocol to turn IPFS nodes into both Retrieval Clients and Retrieval Providers

Alternative 2: build a custom go IPFS/Filecoin retrieval client outside of go-ipfs. A single process IPFS/Filecoin node build from the group up might prove lighter and more nimble than trying to build on top of go-IPFS.

Retrieval Provider/Miner

Our immediate MVP goal for the retrieval miner is to enable miners to serve retrievals without running Lotus Storage Mining software. Similar to the client, go-ipfs might produce a smaller, more functional retrieval provider. To do this we could:

  • Put go-data-transfer in go-ipfs
  • Have the miner advertise its content to index-style Content Routing for Filecoin
  • Accept incoming requests and call out to an Exchange Provider to manage checkpointing and verification of payment
  • Run the payment channel manager locally in a Lotus Lite node or potentially talk to a remote Payment Provider, if such a provider exists.

Alterative 1: As with the Retrieval Client, Myel's work to enable retrieval through a plugin that runs Data Transfer and a custom Exchange/Content Routing protocol seems like an avenue worth exploring further

Components

Content Distribution

It may make sense for hosting and distribution to operate more like an automatic ledger -- content provider asks get automaticallys matched with miner pararms and content providers may not be allowed to reject deals that fall within the parameters of the ask.

Filecoin Chain

It's not clear we need anything else. It might make sense to extract the gateway API down to the above reduced package to remove any direct dependency on the lotus repo at some point.

Content Routing

If we can build a functioning index/gateway, as true "retrieval miners" come online, they can also use the existing APIs continue to augment the index above.

Exchange

In a more traditional CDN context, a content provider is often pays for bandwidth and resources on a servers so that a client is able to download data quickly. The client themselves often does not pay anything for this data. To support this kind of usage, we will likely have to augment the Exchange Protocol to support this kind of 3-way transaction. A client might in this scenario instead send a proof to the content provider that they have received the data causing the content provider to unlock payment to the miner.

Payment Channel Manager

It's fairly ineffecient to create a payment channel between every party that wants to transact on a network. A common technique for managing payment channels in a large scale crypto currency network is a "hub and spoke pattern". In this scenario, two parties who do not yet have a direct payment channel can go through an intermediary they've both already established a channel with. The mechanism to do this called is "Hash-Locked-Time-Codes", and the Filecoin blockchain has the machinery to do this. This is possibly something we need to implement to get payment channels to work at scale, even for an "MVP" retrieval market CDN. Both Bitcoin and Ethereum offer networks that have this functionality

Wallet

We may need to be able to delegate remote signing on a per-address basis. Lotus

# V2 Milestone: Future Directions

Roles

Content Provider

Likely the above set of work may bleed into 2022. Beyond that, the proposed design makes content providers somewhat passive -- they make bids, then wait for miners to offer. It probably makes sense to offer other kinds of negotiation, and more complex ways of putting together hosting packages.

Marketplace Provider

Centralized indexing is likely a strategy with a limited lifespan as the network grows. The next steps is to bring in the results of ResNetLabs research to scale Content Routing, and likely there will be similar efforts neccessary around Miner and Content Bid Indexes.

Payment Provider

The long term question in terms of payments is how the chain itself can incentivize retrieval. This is an semi-open research question, which it probably makes sense to devote time to sooner rather than later, even if actual development doesn't start for a year or two.

Retrieval Client

The biggest pool of potential clients for the Retrieval Market are web browsers. To get retrieval running in the browser however, we must first convert several parts of the Filecoin stack currently in Go to javascript (or Web Assembly):

  • The following components currently have no JS equivalent: Data Transfer, Exchange, and a Wallet.
  • We'd need to implement a Transport, but it's possible we could use STP, an in development version of Transport based on HTTP, that might be easier to implement in Javascript
  • Hopefully, by the time we built retrieval in the browser, the JS client could talk to Payment Providers and MarketPlace Providers to access the remaining components remotely

Retrieval Provider/Miner

The biggest long term risk for Retrieval Miners is making sure it's profitable to do only retrieval mining. This means we may need to research cryptoeconomic incentives for retrieval mining over the long term.

Components

Content Distribution

Ultimately this is a wide open research problem and hopefully during 2021 ResNetLab can look at potential apporaches.

Content Bid Index

Eventually, an efficient Retrieval Market would have several Content Bid Indexes, perhaps serving unique geographic regions.

Data Transfer

Data Transfer has not yet tacked the problem of very large transfer. This is something we will need to look at to enable usage of large static datasets in the retrieval market.

Content Routing

An index will likely hit scaling limits fairly quickly. In a year, hopefully ResNetLab's content routing efforts will produce clear next steps for more scalable solutions

Exchange

Fair exchange is a hard problem and out existing retrieval protocol is very much an "MVP". While it may be sufficient for some time, improved protocols might unlock superior performance, particularly if they become tied to crtypographic incentires.

Payment Channel Manager

There are several further innovations in the world of payment channel mechanisms (https://finalitylabs.io/static/media/SetPaymentChannels.8a29d449.pdf, https://eprint.iacr.org/2017/635.pdf). Third parties looking to innovate in this space can offer custom implementations of the payment channel manager as a way of becoming service providers for fast payments

Reputational Index

A single point of failure index is insufficent to provide reputation for the network for the long term. Over time, third parties can create competetive indexes.

Last Updated: 4/9/2021, 11:27:11 PM