Protocol handler API

[Gozala]

Primary goal of this milestone is to enable Firefox addon to provide custom network protocol implementation. This will not enable add-on to provide full IPFS, Dat, SSB implementations (as that would for example require exposing UDP socket API to add-ons) but it would enable add-on to add associated protocol handler and possibly delegate actual networking to local node via native messaging API or a through a public gateway. Unlike current support of decentralized network protocols this will not redirect request but rather load them under specified URL and have content origin be set to scheme://${hostname}.

Option 1

Allow addon to register ServiceWorker against custom network protocol, allowing Firefox to delegate all network requests to the dedicated ServiceWorker run in the separate background thread.

Option 2

There is already established Electron API used by Beaker Browser and IPFS extension for Brave browser to do exactly that. Compatible Firefox addon API could be exposed to enable that in Firefox. Firefox could leverage existing components like nsIProtocolHandler, nsIStandardURL, nsIURL to expose such API.

🐞 1271553

API Draft

Current work-in progress implementation has a following API exposed under browser.protocol namespace

interface ProtocolAPI = {
  registerProtocol(scheme:string, handler:Request => Response):Promise<void>;
}

interface Request = {
  url:string;
}

interface Response = {
  contentCharset?: string; // default: "utf-8" 
  contentLength?: number; // default: -1 (Unknown)
  contentType?: string; // default: "text/html"
  content: AsyncIterator<ArrayBuffer>;
}

Status

• Wire background script with nsIProtocolHandler
• Fix Malformed URL error reports in the browser console.
• Handle unload / unregister properly.
• Fix reported unhandled exceptions in browser console.
• Fix tab progress spinner
• Wire back-pressure / cancelling mechanism.
• Allow protocol to provide content in secure context. 🚫🐞
• Address "connection is not secure" dropdown
• Support for response status codes
• Support for redirects
• Support request headers & request methods
• Support response headers

[Gozala]

There is some relevant discussion in regards to going with Service Worker like API at https://github.com/Gozala/libdweb/issues/8#issuecomment-390437124

[Gozala]

Made some progress on this issue last week. As things stand now above described API is available to the background script if protocol permission is requested. Current implementation still does not handle back-pressure or cancelling as in if consumer attempts to pause or cancel underlying channel, but I'd work towards getting this wired up as well.

Only other thing that I'm considering is going with ProtocolWorker API due to reasons described at https://github.com/Gozala/libdweb/issues/8#issuecomment-390437124. Other the performance it's also just a lot simpler to manage multiple messaging channels with worker than it is to do it with add-on's background script.

[Gozala]

I observe uncaught exceptions being reported in the browser console, but can't seem to figure out where it's coming from

[Gozala]

It seems that for whatever reason loader spinner for new tab into which content from custom protocol handler is loaded never finishes spinning. Oddly enough it works fine for the first tab. Maybe it's related to #issuecomment-390724031 ?

[Gozala]

In separate conversation @olizilla asked if we could support CSP headers in custom protocols ? At least that is a mechanics in Electron.

Support for HTTP headers in non-http protocol seems odd on one hand, but on the other hand it's reasonable request. According to the [nsIChannel] (https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsIChannel) page on MDN:

After a request has been completed, the channel is still valid for accessing protocol-specific results. For example, QueryInterfacing to nsIHttpChannel allows response headers to be retrieved for the corresponding http transaction.

I wonder if implementing nsIHttpChannel interface from our custom channels would allow us to do that.

@olizilla I also as I mentioned in that conversation there are some protocol flags that could be used to control policies to a much lesser degree.

[lidel]

@Gozala a somehow related question: are these flags taken in consideration when browser decides if loaded resource is a Secure Context? Asking because localhost is marked as a Secure Context (even on HTTP without encryption), and I see flags like URI_IS_LOCAL_RESOURCE, URI_IS_LOCAL_FILE, which could be kinda true for things like a locally running IPFS node.

[Gozala]

@Gozala a somehow related question: are these flags taken in consideration when browser decides if loaded resource is a Secure Context?

I honestly don't know, but that is what I would expect.

Turns out protocolFlags do not affect isSecureContext, I tried using same flags as file protocol

Ci.nsIProtocolHandler.URI_IS_LOCAL_FILE |
Ci.nsIProtocolHandler.URI_IS_LOCAL_RESOURCE |
Ci.nsIProtocolHandler.URI_STD |
Ci.nsIProtocolHandler.URI_NOAUTH

But loaded page still reports self.isSecureContext as false.

Asking because localhost is marked as a Secure Context (even on HTTP without encryption), and I seen flags like URI_IS_LOCAL_RESOURCE, URI_IS_LOCAL_FILE, which is kinda true for things like a locally running IPFS node.

In case of URI_IS_LOCAL_RESOURCE my guess it's what used for resource:/// URLs that are firefox bundled content and as description says it "would not hit the network" which I don't think is true of IPFS content. It also note imply secure context it resource:/// urls inherit that from the loader.

As of URI_IS_LOCAL_FILE I'm confused by the description of this flag. But I also think it's pretty much for file:/// URLs.

At the moment protocol handler uses URI_STD | URI_INHERITS_SECURITY_CONTEXT which I assumed was enough for all intended purposes, but it turns out self.isSecureContextisfalse` wihch is not great.

[Gozala]

In regards to isSecureContext I did some research and according to the nsIContentSecurityManager.idl :

/**
   * Implementation of
   * https://w3c.github.io/webappsec-secure-contexts/#is-origin-trustworthy
   *
   * The value returned by this method feeds into the the Secure Context
   * algorithm that determins the value of Window.isSecureContext and
   * WorkerGlobalScope.isSecureContext.
   *
   * This method returns false instead of throwing upon errors.
   */
boolean isOriginPotentiallyTrustworthy(in nsIPrincipal aPrincipal);

Then looking at the current implementation of isOriginPotentiallyTrustworthy it does not seem like any protocol, channel or URI flags affect the outcome, it's a hard-coded list of protocol schemas that are granted "secure context".

😞

[Gozala]

Turns out there is a bug 1328695 that says "isOriginPotentiallyTrustworthy should consider URI Flags" and as far as I can tell from the attached changes it adds URI_SAFE_TO_LOAD_IN_SECURE_CONTEXT flag that is then used in isOriginPotentiallyTrustworthy implementation instead of hard-coded protocol schema list. Which is exactly what we'd need to enable isSecureContext in the custom protocol implementations.

😌 😀

[Gozala]

Infinite loading issue is finally resolved!

[Gozala]

Unfortunately I discovered yet another issue this time it's a clam that connection is not secure:

And I have also verified it's not related to the isSecureContext as that even with system principal same error seems to appear. I am suspecting it's related to securityInfo field of nsIChannel.