Testing with Reqwest

[dbettin]

When I run unit tests I would like reqwest to return setup data instead of making the actual request. So, Is there a way to stub out reqwest to return canned responses for unit testing?

[Arnavion]

https://docs.rs/crate/reqwest_mock/ perhaps? I haven't used it myself.

[leoschwarz]

Hi, I'm the author of that crate. Basically it defines the trait Client which provides more or less the same API (with a few omissions) as reqwest and has different implementors providing different behaviour, so you can (or more like have to) write your code generic over that trait.

As for structs implementing Client there currently is only DirectClient, which just makes a request and does nothing else, and ReplayClient which records responses to a file on the first request and later on "replays" them by providing the same response again. What you want sounds a bit like a StubClient which I haven't had the time yet to implement. However if you have ideas or suggestions feel free to contact me or open an issue at https://github.com/leoschwarz/reqwest_mock.

That said my two main reasons not to promote my crate yet is that having your whole client code be generic is potentially annoying and I want to add a useful type providing boxing optimized for the DirectClient case which is sort of intended for "production use" then.

[matthewkmayer]

I'm super keen on this. Is the best way to help to try https://github.com/leoschwarz/reqwest_mock and report any sharp edges?

[Screwtapello]

It's reasonable to have third-party crates implement different kinds of mocks, because different people will want to test their code in different ways. However, I feel like reqwest should provide traits that define the interface of Client, Request and Response since those things really are defined by reqwest, and it's a shame to make other crates copy-and-paste reqwest's API changes.

[pwoolcoc]

Right now reqwest_mock is not working for me because it no longer implements all of reqwests API. I'd been assuming I could slightly change the structure of my code by using a trait to specifically mock out the process of converting a Request to a Response. Then, instead of calling client.get().foo().bar().send()?, I'd use an implementation of that trait to replace the last .send() part, so it looks like http_sender.send(client.get().foo().bar()). This works great for the case where you actually want to send the request (http_sender.send just calls .send, basically), but it turns out you can't actually construct Response objects manually, so making a "mock" http_sender can't actually work :-/

So, it would be great if there was some kind of ResponseBuilder or something, to allow test code to construct Response objects.

[leoschwarz]

@pwoolcoc You are right that this is probably the best spot to conduct the mocking and not being able to create a response instance (and serializing/deserializing requests and responses for the replay client) has lead me to the current api duplication approach (I'm sorry but I don't have the resources to update it at this time).

I think by now everything minus the serialization is handled by the http crate (there is http::response::Builder), technically serialization could be implemented even from a different crate since there are accessors for all fields. Edit: I'm actually not sure of the current state of http crate integration with reqwest, I guess an ideal solution for this issue is blocked on that.

[pwoolcoc]

@leoschwarz no worries, I understand. are http::Response and reqwest::Response interchangable like that? I thought reqwest used it's own Request and Response types?

EDIT: just saw your edit, yea I'll have to look into it some more. I know reqwest uses the http crate for Headers, not sure what else though

[dhl]

I find mockito to be a fairly rustic solution so far.

It spins up a web server for the lifetime of the test, and you use the cfg flag to switch between the actual endpoint for production and the test server during test:

#[cfg(test)]
use mockito;

// ...

// The host to be used for non-test (production) compilation
#[cfg(not(test))]
let host = "http://example.com";

// The host to be used in test compilation
#[cfg(test)]
let host = &mockito::server_url();

let url = format!("{}/endpoint", host);
let text = reqwest::get(&url)?.text()?;

The advantage of this approach is that it works with any HTTP client library, and does not require any change to existing libraries.

[imbrn]

Hey, any updates on that?

I was thinking about creating a project that wraps reqwest with a fluent API for unit tests with mocking. This project would support any kind of library for making HTTP requests, but reqwest would be the default implementation.

It would be a kind of wrapper that behaves differently depending on the environment. So in test code, it could be a mocked struct which implements the exact same API of non-test code, but with additional methods for configuring HTTP calls behavior; something like this:

#[test]
fn my_test() {
	let mut wrapper = Wrapper::new(); // the name doesn't matter yet
    wrapper.mock()
		.get()
		.from_url("https://example.com")
        .to_status_code(200)
 		.done();

	wrapper.get("https://example.com");

	// then here we can access the wrapper object to get info about the requests made
}

So I was wondering if it'd be a better idea to implement that as a feature of the reqwest instead. 🤔

If you guys agree with me, I can try to do it inside this project. Or is it better to keep it in a separate project? What do you think?

[ta32]

i don't like the idea, of spinning up a webserver to implement unit tests. If this is required you may as well call it an integration test and test more broadly

what i did was just create a trait around a client that uses reqwest. Then made sure the client had minimal logic as its not unit tested. Then i can use mockall to services above it.

However, if the reqwest library creates a trait, then it would be easier to mock https://docs.rs/mockall/0.7.2/mockall/#external-traits and test at a lower level of my code without relying on integration tests.

[leoschwarz]

I've discontinued reqwest_mock, but since this issue is still open I would like to point to these potentially relevant crates for anyone who comes accross this issue in the future:

• https://github.com/chorusone/rvcr
• https://github.com/lukemathwalker/wiremock-rs
• https://github.com/beltram/stubr
• https://github.com/alexliesenfeld/httpmock
• https://github.com/lipanski/mockito (mentioned before in this conversation)

[lambda-fairy]

Note that there is impl From<http::Response> for reqwest::Response. So you can construct an http:Response and test with that.

https://github.com/seanmonstar/reqwest/blob/17c893ffc0d3832d61cb1c0cf278340b7e95557e/src/async_impl/response.rs#L431-L443

[matteosantama]

As a word of caution, the solution by @lambda-fairy works only for http = "0.2". It would be great if building responses was natively supported by reqwest.

[mohe2015]

I think it doesn't matter whether you need to build an http 0.2 response type or a custom reqwest response type though I assume reqwest uses that type internally anyways. Also there is an open PR that updates http to 1.0 so at that point it should interoperate with the whole http 1.0 ecosysten

[Dovchik]

Any news on that? It's a shame there is no trait to mock for unit testing

[mazong1123]

I made an example by leveraging injectorpp to mock reqwest. https://github.com/microsoft/injectorppforrust/blob/main/tests/reqwest.rs

It's not perfect as you need to construct a TcpStream to represent the fake data you want reqwest to return. However, it does provide a way to mock without setting up a mockserver.

In theory, I can add a wrapper on top of these so users don't need to construct a TcpStream manually. However, I'm just wondering how many people are actually interested in this approach.