Refactor library so that it can work with any HTTP client crate

[evanlinjin]

The Problem

Right now rust-esplora-client is designed to only work with HTTP client crates that are baked in (i.e. minreq for blocking, reqwest for async). However, these HTTP client crates may not be preferred by the caller. The caller may also wish to use a different version of the HTTP client crate.

Proposed Solution

Represent each endpoint call as a distinct type. Introduce a trait that is to be implemented for each endpoint call type. This trait will contain the template of how to form the HTTP request and parse the HTTP response body and status code.

pub trait HttpRequest {
    type Output: for<'a> serde::de::Deserialize<'a>;

    fn request_url_path(&self) -> String;
    fn request_method(&self) -> &str;
    fn request_body(&self) -> Vec<u8>;
    fn parse_response<E>(status: i32, body: &[u8]) -> Result<Self::Output, ResponseError<E>>;
}

pub enum ResponseError<E> {
    Json(serde_json::Error),
    HttpStatus { status: i32, body: Vec<u8> },
    Client(E),
}

Examples of endpoint types...

pub struct GetTx {
    pub txid: Txid,
}

impl HttpRequest for GetTx {
    type Output = Option<Transaction>;

    fn request_url_path(&self) -> String {
        format!("/tx/{}/raw", self.txid)
    }

    fn request_method(&self) -> &str {
        "GET"
    }

    fn request_body(&self) -> Vec<u8> {
        Vec::with_capacity(0)
    }

    fn parse_response<E>(status: i32, body: &[u8]) -> Result<Self::Output, ResponseError<E>> {
        match status {
            200 => Ok(serde_json::from_slice(body).map_err(ResponseError::Json)?),
            404 => Ok(None),
            error_status => Err(ResponseError::HttpStatus {
                status: error_status,
                body: body.to_vec(),
            }),
        }
    }
}

pub struct PostTx {
    pub tx: Transaction,
}

impl HttpRequest for PostTx {
    type Output = ();

    fn request_url_path(&self) -> String {
        "/tx".to_string()
    }

    fn request_method(&self) -> &str {
        "POST"
    }

    fn request_body(&self) -> Vec<u8> {
        bitcoin::consensus::encode::serialize(&self.tx)
            .to_lower_hex_string()
            .as_bytes()
            .to_vec()
    }

    fn parse_response<E>(status: i32, body: &[u8]) -> Result<Self::Output, ResponseError<E>> {
        match status {
            200 => Ok(()),
            error_status => Err(ResponseError::HttpStatus {
                status: error_status,
                body: body.to_vec(),
            }),
        }
    }
}

Example for calling any endpoint with minreq:

pub fn call_with_minreq<R: HttpRequest>(
    url_base: &str,
    request: R,
) -> Result<R::Output, ResponseError<minreq::Error>> {
    let req = match request.request_method() {
        "GET" => minreq::get(format!("{}{}", url_base, request.request_url_path())),
        "POST" => minreq::post(format!("{}{}", url_base, request.request_url_path())),
        unhandled_request_method => {
            panic!("unexpected request method: {}", unhandled_request_method)
        }
    }
    .with_body(request.request_body());

    let resp = req.send().map_err(ResponseError::Client)?;
    R::parse_response(resp.status_code, resp.as_bytes())
}

Example for calling any endpoint with reqwest:

pub async fn call_with_reqwest<R: HttpRequest>(
    client: Client,
    url_base: &str,
    request: R,
) -> Result<R::Output, ResponseError<reqwest::Error>> {
    let method =
        reqwest::Method::from_str(request.request_method()).expect("request method must be valid");
    let resp = client
        .request(
            method,
            &format!("{}{}", url_base, request.request_url_path()),
        )
        .body(request.request_body().to_vec())
        .send()
        .await
        .map_err(ResponseError::Client)?;
    let status = resp.status().as_u16() as i32;
    let body = resp.bytes().await.map_err(ResponseError::Client)?.to_vec();
    R::parse_response(status, &body)
}

In Conclusion

As can be seen, the code needed for a HTTP client crate to work with the trait is minimal.

[evanlinjin]

master...evanlinjin:rust-esplora-client:any_http_client_lib

[oleonardolima]

Indeed it's a problem, and I've been facing this while I work on adding Tor support with arti-client on bitcoindevkit/bdk-cli#149.

Thanks for the suggestion! I've been working on this one, but with a different approach, here: 9cbc387

It mainly focuses on having a trait (e.g trait AsyncEsploraClient) that have the methods: get_response, get_opt_response, get_response_json, and so on ...

AFAICT it would require to anyone that's implementing a new client just to implement this trait for the client, as each client could (e.g reqwest) or not have (e.g hyper) useful methods for parsing, as long as they're parse to the correct types.

I would say it's on a higher level, specifically for the methods+parsing we expect. It wouldn't require for us and users have to implement a trait for each method on esplora API 🤔.

I'm wondering which one of both approaches would require minimal changes or be better on the long run. I'll tidy up my PR and also take a deeper look on your suggestion.

[evanlinjin]

I think it's okay to break things completely since it's just a client library. Aiming for "minimal changes" shouldn't be one of our requirements imo.

---

I would say it's on a higher level, specifically for the methods+parsing we expect. It wouldn't require for us and users have to implement a trait for each method on esplora API 🤔.

Correction: It's us (the devs of rust-esplora-client) that has to create a type for each endpoint and impl a trait for each endpoint type. The users of this library definitely do not have to do this. Users just create a single function for a given HTTP client type (if it's not provided). This single function works with all endpoints.

---

Abstracting out the transport (IO) means we need two traits, one for blocking and one for async. Abstracting out the request formation and response handling means we only need one trait (since this will not change regardless of the transport).

Requiring users to impl a trait for an HTTP client type means they need to wrap the HTTP client type. Since one cannot impl an external trait for an external type.

---

However, having each endpoint as a method on a type is a nicer API. It makes all endpoints easier to find than separate types for each endpoint.

[tnull]

I think it's okay to break things completely since it's just a client library. Aiming for "minimal changes" shouldn't be one of our requirements imo.

In this regard, I want to mention that we're indeed leaning on specific error codes and reqwest error variants to discern the reasons why a specific call failed. If the updated API would still allow to do this, I agree that API breaks are not the end of the world and I'm happy to switch. However, it would be an issue for us, e.g., if the error types would get much less expressive due to the need of finding the least common denominator (in fact, we'd highly appreciate more expressive error types to allow discerning different HTTP 400 errors which is currently only possibly by matching on the error message, i.e., error-prone).

[oleonardolima]

Abstracting out the transport (IO) means we need two traits, one for blocking and one for async. Abstracting out the request formation and response handling means we only need one trait (since this will not change regardless of the transport).

Requiring users to impl a trait for an HTTP client type means they need to wrap the HTTP client type. Since one cannot impl an external trait for an external type.

Cool! I've thinking a bit about both approaches. I will move and explore more towards your proposed one.

As minimal changes are not a problem, it's better to make it generic over the request/response instead of IO. It becomes simple enough for the user, and more future-proof instead of the one I was working on.

--

In this regard, I want to mention that we're indeed leaning on specific error codes and reqwest error variants to discern the reasons why a specific call failed. If the updated API would still allow to do this, I agree that API breaks are not the end of the world and I'm happy to switch. However, it would be an issue for us, e.g., if the error types would get much less expressive due to the need of finding the least common denominator (in fact, we'd highly appreciate more expressive error types to allow discerning different HTTP 400 errors which is currently only possibly by matching on the error message, i.e., error-prone).

Good! Thanks for raising awareness on error handling, I'll have that in mind while working at it.

[evanlinjin]

I think there is no point in having multiple request methods on HttpRequest. We can either introduce a type alias or a struct with public fields. I'm leaning towards struct with public fields (i.e. named RequestData) since it's easier to work with for the user.

[LLFourn]

Why would anyone care about what HTTP client is being used under the hood? I mean as long as it does HTTP correctly.

iirc the reason we had two different HTTP clients in the first place was that @tcharding needed something that compiled to SGX. reqwest didn't but ureq did. Then somehow we got minireq (can't remember why).

I think we should just choose an HTTP client and stick with it. We should use the same for blocking and non-blocking or even consider removing the non-blocking version.

[evanlinjin]

Why would anyone care about what HTTP client is being used under the hood? I mean as long as it does HTTP correctly.

I can think of a couple examples where this is useful.

1. If someone wants to access Esplora via some specialized network (i.e. TOR with something like arti-client).
2. If someone is serving Esplora on a webserver with some other HTTP services and is already accessing the services with another HTTP client library and they would like to reuse the TCP connection.
3. A combination of the aforementioned points.

I think we should just choose an HTTP client and stick with it. We should use the same for blocking and non-blocking or even consider removing the non-blocking version.

I agree with this. However, making this library easily extensible would be useful imo.

[LLFourn]

If someone wants to access Esplora via some specialized network (i.e. TOR with something like arti-client).

The better way to design this is to have the option for the user to pass in the TCP connection (or whatever) to the HTTP client:

https://docs.rs/hyper/latest/hyper/client/conn/http1/fn.handshake.html

If someone is serving Esplora on a webserver with some other HTTP services and is already accessing the services with another HTTP client library and they would like to reuse the TCP connection.

This seems severely under motivated. I'm pretty sure we are making a bunch of parallel connections anyway. Unless we are using HTTP 2.0 now?

The question about whether HTTP 2.0 is important because if we are creating lots of parallel connections then we'd have to pass in a "connection builder" to do parallel requests. If it's HTTP 2.0 we can just reuse the same connection always.

I agree with this. However, making this library easily extensible would be useful imo.

Yes but not extensible with multiple HTTP implementations. There's one way to do it. Stick to one implementation.

[oleonardolima]

The better way to design this is to have the option for the user to pass in the TCP connection (or whatever) to the HTTP client:

https://docs.rs/hyper/latest/hyper/client/conn/http1/fn.handshake.html

I do agree, but AFAICT hyper is the only one that currently supports this approach/design of replaceable transport. It looks like ureq will also have a similar feature on v3, but it isn't out yet.

Yes but not extensible with multiple HTTP implementations. There's one way to do it. Stick to one implementation.

Still, we'd need to have a single implementation that does both blocking/non-blocking, right?

However, we could also be opinionated about that (e.g supporting only blocking), as long as the user has alternatives.

I think that by making it extensible, we can choose to support a single implementation (e.g. minreq), while still making it simple enough for the user who would like to use something different to implement a custom handler with its HTTP client of choice.