plabayo
add `curl` module to rama-http-types
It's gonna be a relatively small module with 2 methods:
pub fn request_headers_to_curl_command(request: Impl IntoCurlHeadersCommand) -> Result<String, OpaqueError>pub async fn request_to_curl_command(request impl IntoCurlCommand) -> Result<(String, Request), OpaqueError>
(1) only deals with the headers, so not the body, while (2) deals with the actual body.
That means that (2) needs full access to the body. It doesn't however have to consume or destroy it, so it can return it. But it does make it potentially a more heavy lift process.
-
IntoCurlHeadersCommandwould be a sealed trait and can be implemented for now already for:- request
Parts - request
&Parts Request<B>for anyB&Request<B>for anyB
- request
-
IntoCurlCommand` would be a sealed trait and can be implemented for now already for:
Request<Body>(own it)
Where the latter Body needs to respect:
Body: rama::http::dep::http_body::Body<Data = bytes::Bytes, Error: Into<BoxError>>
+ Send
+ Sync
And that's about it.
The result should be a valid (one-line) curl command.
Example:
request_headers_to_curl_command(&Request::builder().uri("http://example.com").header("accept", "application/json").body(()).unwrap()).unwrap()
Would result in:
curl --http1.1 -H 'accept: application/json' http://example.com
Make sure to add sufficient test cases in that module.
As icing on the cake you can add a new property curl: <cmd> to the EchoService such that services such as https://echo.ramaproxy.org/ will have that info added to the http object:
{
"http": {
"curl": "curl --http1.1 -H 'accept: application/json' http://example.com",
...
}
...
}
Remarks:
- make sure to have also a look at https://docs.rs/rama/latest/rama/http/io/fn.write_http_request.html as it will show you how you can get the original header order and casing of your request, which is importnat
- pseudo header info you do not need as there is no way to pass this to curl
- tls can be ignored for now as well
hey @GlenDC ! found this project through This Week In Rust newsletter and found it so interesting, I've been wanting to learn more about networking :') would love to take a stab at this!
Hi @Taenerys thank you for your interest in the project. I'll assign the issue to you.
In case you have questions, need help, guidance or mentoring, do let me know. I'll gladly assist you. As such feel free to open a PR as soon as you want or need it, even if very early WIP. If it helps you it's alright.
How s this going @Taenerys ? Happy to help / guide where you can use me :) I hope you're doing well either way, and do see me as a team partner or colleague where you can and want.
thanks @GlenDC ! really appreciate -- your feedbacks really keep me going! manu apologies last week I have gotten a bit busy, but definitely will continue on this and keep you updated this week!
@Taenerys a friendly check in. Is this something you still want to pursuit or do you prefer to hand this over? Either way cool, but I do need to get this going for certain purposes. Doesn't have to be fast, but I do want to see progress on this. If you're stuck, want help, have questions, require mentoring, do let me know. I'm here for you.
All yours @maartendeprez . Let me know if you have questions, feedback or need help. Feel free to open a draft PR prior to it being finished in case it can help you in any way. I'm here for you. Thank you for putting in the time and effort for this one.
Hey, found this through the TWIR607! Do you still need help with this?
Hi @AydanPirani , welcome to the club. We keep issues like these open for the community as opportunities to (a) get you engaged into the framework and our community and (b) a learning opportunity. In return @soundofspace and I are here to mentor and guide you. So in case you are missing details, have questions, remarks, things you wish to discuss. Let us know.
I'll assign it to you in the meanwhile. If you do decide now or later to not want to pursuit it do let us know and we'll un-assign you without an issue. We understand choices can change and life can happen.
Only thing i would say try to keep it, certainly in the beginning, at 1 issue at a time. So do let me know if you prefer to start with this first, or with the JWA one (#509). So I can assign you to the one you first want to pick up @AydanPirani .
@GlenDC Hey there, thanks for the warm welcome! I definitely prefer this one, let's go with this for now :)
I'll take a look at getting set up tomorrow, and I'll see what I can get done.
@GlenDC Hey there, thanks for the warm welcome! I definitely prefer this one, let's go with this for now :)
I'll take a look at getting set up tomorrow, and I'll see what I can get done.
Great. I'll plan you in for this one in that case. Do feel free to reach out whenever, be it via here, a draft PR or discord. We are here for you.
Hey @GlenDC , just got a chance to look at this! Quick question, do you have a driver program to run/test code changes? If not, I can make one in examples/curl_command.rs, let me know what's preferred.
run/test code changes
What specifically do you want to run test?
- You should be able to unit test all cases easily directly in the module
- Once ready and tested you can also add it as a property in the http body from the echo service output. That's a service also used to drive https://echo.ramaproxy.org (you can run it yourself running 'just rama echo'
But 2 is just a user. 1 is where you test it and confirm. Which are just unit tests.
Gotcha, will take a look.
@GlenDC When I run cargo test on the base repo, I'm getting the following error - do you know why?
/Users/aydan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/aws-lc-sys-0.30.0/aws-lc/include/openssl/base.h:61:10: fatal error: 'stdlib.h' file not found
Gotcha, will take a look.
@GlenDC When I run
cargo teston the base repo, I'm getting the following error - do you know why?/Users/aydan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/aws-lc-sys-0.30.0/aws-lc/include/openssl/base.h:61:10: fatal error: 'stdlib.h' file not found
What are your system details? I assume macos based on yoru path? But what architecture and os version?
That said I rarely ever run cargo test myself, especially not like that. If I use it I do it with specific package (-p) and features (--features) defined.
- usually I run
just qaorjust testto run all my tests, this takes a while though, so in rare cases where I need to quickly iterate I can fall back tocargo test -p <package> --features <feature list required> - some ide's also support running a specific test or module of tests from within IDE, which specifies the
-pand--featuresautomatically for you
But running a bare cargo test will not be too useful on rama as no features will be enabled, so no code will be active.
--
In context of this issue, if you want to quickly run all possible tests for the relevant package you would run cargo test -p rama-http-types
But what architecture and os version?
Running OS 26.0 (perhaps this could be the issue), LLVM version 20.1.8
That said I rarely ever run cargo test myself, especially not like that. If I use it I do it with specific package (-p) and features (--features) defined.
Gotcha, thanks for the pointer! Installed just but getting the same error with just qa (as expected)
In context of this issue, if you want to quickly run all possible tests for the relevant package you would run cargo test -p rama-http-types
Just ran the latter command (cargo test -p rama-http-types), and it seems to be working fine now. Will ignore and pray it doesn't come up again layer ;)
OS 26.0
OS 26??
Will ignore and pray it doesn't come up again later ;)
rama-http-types doesn't use tls or crypto, so should be ok for this task
OS 26??
Yeah,
rama-http-types doesn't use tls or crypto, so should be ok for this task
Ack! Also, for the task: I'm looking into creating a curl folder under rama-http-types/src and adding the main functions there, is that a solid start?
Also, for the task: I'm looking into creating a curl folder under rama-http-types/src and adding the main functions there, is that a solid start?
Yeah I think that's fine. Especially for now. We can always move it around in future if needed. But seems to me like an excellent location for now.
What is OS 26 @AydanPirani ?
Apologies, forgot to finish response in earlier thread. On dev beta of the new OS, they changed numbering to go from 18->26.
https://www.apple.com/os/macos/
Ah ok wild, I am still on 15 😓 Not sure if it's related, but possible. Feel free to fix it if you know what the issue is. But for this task you can indeed ignore it.
Hey there @GlenDC ! Are you sure the error is being caused by TLS/crypto? I created the following minimal example program, and it seems to be throwing the same error:
use rama::http::Request;
#[tokio::main]
async fn main() {
let req = Request::builder().uri("http://example.com").header("accept", "application/json").body(()).unwrap();
println!("request: {:?}", req);
}
I was expecting to see at least some minimal output, but get the same fatal error: 'stdlib.h message - am I missing something here ?
(Referenced in my fork of Rama)
Your example enables a bunch of features (implicitly). Probably because of one of it.
FWIW you can remove that example. Small features like this require no big end user examples.
You'll already have plenty of unit tests and it will also be demonstrated in the echo service.
Your example enables a bunch of features
Got it. Is that as a result of the example itself, or as a result of the code? I tried to copy code from the initial issue, so it seems that the error will get thrown regardless within unit tests. Will try to get a resolution to the issue, and let you know once I (hopefully) identify a potential fix.
FWIW you can remove that example
Will do, thanks!
I would think that still means you have certain features enabled? Unless you need a 'cargo clean'.
Eg if you limit it to '-p rama-http-types' I would be surprised that you get such errors.
@soundofspace might be able to chip in as well.
@AydanPirani Looks like aws lc sys is having problems building. Normally it has prebuild binaries but I'm guessing since you are on a dev beta version you have to build from scratch.
This issue seems to throw the same error, and maybe it's possible to fix your setup in the same way
@AydanPirani Looks like aws lc sys is having problems building. Normally it has prebuild binaries but I'm guessing since you are on a dev beta version you have to build from scratch.
This issue seems to throw the same error, and maybe it's possible to fix your setup in the same way
Hey, @GlenDC @soundofspace thanks for the help! I got the error resolved (needed to uninstall brew clang and use xcode clang), now working on the implementation of the actual issue itself. Should have something in by early tomorrow, will ping once it's ready :)
Also, for some reason I don't see myself as assigned to the issue - can you please fix?
@GlenDC @soundofspace Hey, quick question:
IntoCurlHeadersCommand would be a sealed trait and can be implemented for now already for: request Parts request &Parts Request<B> for any B &Request<B> for any B
Why do we need separate implementations for all of these? Can we not have a generic implementation for any Request, or do we want to be able to pass in Parts as well?
@GlenDC @soundofspace Hey, quick question:
IntoCurlHeadersCommand would be a sealed trait and can be implemented for now already for: request Parts request &Parts Request<B> for any B &Request<B> for any B
Why do we need separate implementations for all of these? Can we not have a generic implementation for any
Request, or do we want to be able to pass inPartsas well?
@soundofspace didn't you implement something like this recently to expose these properties for any of these (eg methods)?
@AydanPirani you don't want the http conversion into curl command parts to be duplicated. You simply want to be able to accept input from which you can get headers, method, version and extensions. But that's just a matter of using some kind of getters.