cargo-to-recipe.sh: new tool

[Crestwave]

I made this for personal use, so some things like the -d option might not make much sense for the end user; just let me know if you want to change or remove any of these. The lines that are commented out are due to #175.

[korli]

Please document the required tools (find, sha256sum, sed, tar).

[Crestwave]

Please document the required tools (find, sha256sum, sed, tar).

Ah, I thought that they were supposed to be in every Haiku system. I've added a check for them now.

[Crestwave]

By the way, would it be preferred to put the crates in an array or list on the recipe and evaluate the dependencies' SOURCE_URIs and SOURCE_DIRs at runtime? Or is the current format fine?

And if it is [fine], would it be preferred to list the dependencies' SOURCE_DIRs for submitted recipes even if it takes up quite a bit of space with the current format (I've always omitted them so far and used $sourceDir$i/* when copying the directory to the vendor)?

[pulkomandy]

Hello @nielx , since you are our Rust expert, do you have any input on this?

[nielx]

I do not have any particular feedback on this tool or whether it should be included in haikuporter. It makes sense to me though, to help automate packaging software that is built by cargo and rust, if the current dogma on how it is packaged persists.

My suggestion for validating would be to try this out on the existing rust recipes (and maybe add some more), run it, and see if the output is acceptable. It is not my call to decide whether the output should match the human-built recipes, or whether there can just be a disclaimer like # this section was autogenerated by cargo-to-recipe.sh

Having said that, I still do not understand why this particular method of packaging binaries built with cargo is preferred, but I will pick that fight if I ever submit a recipe for something. :-P

[Crestwave]

I do not have any particular feedback on this tool or whether it should be included in haikuporter. It makes sense to me though, to help automate packaging software that is built by cargo and rust, if the current dogma on how it is packaged persists.

My suggestion for validating would be to try this out on the existing rust recipes (and maybe add some more), run it, and see if the output is acceptable. It is not my call to decide whether the output should match the human-built recipes, or whether there can just be a disclaimer like # this section was autogenerated by cargo-to-recipe.sh

Having said that, I still do not understand why this particular method of packaging binaries built with cargo is preferred, but I will pick that fight if I ever submit a recipe for something. :-P

To clarify, I have been using this from the start; every single cargo recipe in HaikuPorts has been created by this tool and edited as appropriate. It is not feasible to create these recipes manually as a package can have over a hundred cargo dependencies (bat) and HaikuPorts' standards requires each to have their checksum and URL specified.

That said, the dependencies can be parsed by a simple script (https://github.com/haikuports/haikuporter/pull/176/commits/27ca90c7fea3225ce49cafcf8a3684e7307f3fff); this is just a fun project where I tried to automate as much as possible. Thus, I don't mind if this doesn't get merged, but do let me know of your requirements; if you prefer the simple script then I can cook one up and just keep this branch as an unofficial but more convenient version.

Here is a recipe created from scratch with cargo-to-recipe.sh -c rg https://github.com/BurntSushi/ripgrep/archive/13.0.0.tar.gz sys-apps/ripgrep: http://ix.io/3vHV Diff with https://github.com/haikuports/haikuports/blob/master/sys-apps/ripgrep/ripgrep-13.0.0.recipe: http://ix.io/3vHX

The same test with cargo-to-recipe.sh https://github.com/sharkdp/diskus/archive/v0.6.0.tar.gz sys-apps/diskus:

2c2,5
< DESCRIPTION=""
---
> DESCRIPTION="diskus is a very simple program that computes the total size of \
> the current directory. It is a parallelized version of 'du -sh'. On the \
> author's 8-core laptop, it is about ten times faster than 'du' for a cold disk \
> cache and more than three times faster on a warm disk cache."
4c7
< COPYRIGHT=""
---
> COPYRIGHT="diskus developers 2018-2019"
144c147
< ARCHITECTURES="!x86_gcc2 ?x86 ?x86_64"
---
> ARCHITECTURES="!x86_gcc2 ?x86 x86_64"
147c150
< SECONDARY_ARCHITECTURES="?x86"
---
> SECONDARY_ARCHITECTURES="x86"
201c204
< 	install -m 755 -d "$commandBinDir" "$docDir"
---
> 	install -m 755 -d "$commandBinDir" "$docDir" "$manDir/man1"
203a207,208
> 	install -m 644 doc/diskus.1 "$manDir/man1"
> 

I also use this to update packages, as most of the dependency info has to be replaced. I recently bumped ripgrep (https://github.com/haikuports/haikuports/commit/1a181e802731c4ac6f7bf67daf38d1eceb01970d), which changed it so much that Git considered it a rewrite. But the workflow was simply:

1. Update the package as normal (renaming the recipe and updating ripgrep's checksum)
2. cargo-to-recipe.sh -b sys-apps/ripgrep
3. Update the seq invocation to reflect the current number of SOURCE_URI's (this can be automated as well)

Also, I agree that more recipes using Rust would be great! However, a lot of them have huge feature-sets and depend on packages that need to be ported to Haiku. For example, bat's latest version does not even work because it now depends on sys-info for bug reporting. Incompatibility issues can and have been solved before but it can be a lot of work outside of creating the recipe, which is why we only have around 5 cargo programs packaged right now.

[nielx]

Thank you for all the information. I hope it is possible to divide the recipe file up into two sections, one that we expect a recipe-writer to manually change, and the other section that can be relied on to be always properly hand-generated. We might be able to accomplish that.

My suggestions/thoughts:

• Do we want to keep a (manual) list of crates that can build against native libraries? I am thinking about libz-sys, curl-sys, openssl, openssh, libnghttp2, xslt and probably others. This will automate most if not all of the build dependencies and runtime dependencies.
• Can we define a variable in the hand-craft section called CARGO_FEATURES (or something like that) to specify a list of features you need at the build stage? This way the recipe writer will not manually have to edit the BUILD() function.
• Can we investigate whether we can use cargo install to perform the minimum install? if that works, we can then see if the default generated install can call something like INSTALL_EXTRAS() to do things like installing man pages. This will be in the hand-crafted section.

The recipe file would then look something like this:

# This file was generated by cargo-to-recipe.sh with the command {...}
# This section is maintained by the recipe writer
SUMMARY="A search tool that combines the usability of ag with the speed of grep"
COPYRIGHT="2015 Andrew Gallant"
ARCHITECTURES="!x86_gcc2 x86 x86_64"
SECONDARY_ARCHITECTURES="x86"
CARGO_FEATURES="pcre2"

INSTALL_EXTRAS()
{
	install -m 755 -d "$commandBinDir" "$docDir" "$manDir"/man1 \
		"$dataDir"/bash-completions/completions \
		"$dataDir"/zsh/site-functions

	install -m 644 README.md "$docDir"
	install -m 644 target/release/build/ripgrep-*/out/rg.1 "$manDir"/man1

	install -m 644 target/release/build/ripgrep-*/out/rg.bash \
		"$dataDir"/bash-completions/completions/rg
	install -m 644 complete/_rg "$dataDir"/zsh/site-functions
}

# The section below was generated by the cargo-to-recipe.sh tool
...

As a final caveat: I am not part of the HaikuPorts core group, so before getting into this, we may want to get a blessing from those who are whether or not this script will be accepted.

[waddlesplash]

As far as I'm concerned, those who know more about Rust than those in the "core group" should be the ones suggesting packing policy here. So far what I have seen at least makes basic sense so far as I know, so by all means continue.

[pulkomandy]

I'm going to merge this, because I tried to write a recipe for rust based software manually yesterday, and gave up after adding a hundred source URIs for all needed crates manually. I think we can start with this version of the script, and iterate on it as needed if we find problems with it. There is no need to keep it in a merge request forever.

[augiedoggie]

Is there a formal policy for rust recipes? I would really prefer to use cargo with the --locked option to do the downloading instead of listing every crate in the recipe.

[pulkomandy]

There isn't one yet. If you know something about Rust and how its packaging works, you can start writing a wiki page, and, I guess, make changes to this script, or delete it if it turns out there is a much simpler way and this is not needed.

I have no idea how any of this works, but we definitely had to do something about this 6 year old merge request, and in the last 6 years it seems no one suggested anything else.

[augiedoggie]

We're already using both styles and so I was wondering if one is preferred. For example rav1e and tree-sitter both use cargo to do the downloading while things like ripgrep list all of the crates in the recipe and use haikuporter to download them.

[pulkomandy]

OK, I didn't notice the other style. It seems fine as well, but it results in the downloads not being checksummed by us (does Cargo do enough checks on them?) and also not mirrorable by the normal ports-mirrors system (which I think needs them to be in SRC_URI).

I don't know if this is really important, in other cases we have done quite some work to remove network access from the build phase.

Maybe we can add a "fetch" step which can be customized? The default would be to fetch anything that's in SRC_URIs, but haikuporter could also call the FETCH() {} function from the recipe, and then we can run "cargo fetch" there and maybe store the results in the ports-mirror as well? I'm not very familiar with how th ports mirror works, so maybe this doesn't make sense.

[Begasus]

In case as cargo-c: Downloaded 330 crates (70.2 MB) in 49.54s (largest was openssl-src at 9.6 MB)

Manually listing them up in a recipe is overkill (IIRC only Gentoo has some kind of process like that), the Cargo.lock holds the crates used and lists up the checksums, I'm just not sure (no sign in the build process) on how/if they are checked against?

Tried this script on the existing cargo-c recipe, but seems it change the SOURCE_URI_2 making it unusable.