[Perference] using iterator/channel refactor loop

[cathaysia]

Using a loop to store too much information at once can cause memory problems. It's better to refactor it using iterators or channels.

[spikespaz]

Will edit to explain when at work

Here is a state machine I wrote for this

https://docs.rs/awaur/latest/awaur/paginator/index.html https://docs.rs/awaur/latest/awaur/endpoints/index.html

Example 1:

API: https://github.com/EssentialMC/curseforge-rs/blob/master/src/official/endpoints.rs#L157-L181 Impl: https://github.com/EssentialMC/curseforge-rs/blob/master/src/official/request/pagination.rs

Example 2:

API: https://github.com/EssentialMC/modrinth-rs/blob/master/src/endpoints.rs#L35-L53 Impl: https://github.com/EssentialMC/modrinth-rs/blob/master/src/request/pagination.rs

[cathaysia]

I have been thinking about the reconstruction of this project some time ago. Yesterday I thought that the entire project is essentially a crawler. I should refer to scrapy's architecture and divide the project into crawlers and middleware. But since you've written a separate project for this, let's do it your way first. Let me first take a look at the openvsx interface.

[spikespaz]

Well, I never got a chance to explain that today. But I'm glad you at least got to see it.

If you think we can provide a polished API to VSCM and OpenVSX, I would like to do that, and publish the two crates.

My crate awaur should provide decent utility for abstracting some of the pagination logic and response construction and deconstruction boilerplate. If you think it's a good fit, and a good idea to create two independent crates for the REST APIs (I understand that VSCM is undocumented) I think we should do so and I will become an active contributor/maintainer of multiple related crates.

Using awaur in broad scenarios such as this would allow me to move more repetitive parts into the crate, and to polish it for use further down the road. I have plans on using it for AMO's Firefox Extensions API as well. You also don't have to move away from tokio and reqwest, there is currently only support for isahc but it is intended to become agnostic. Collaboration would get much help with that.

Would like to hear your thoughts.

[cathaysia]

Yes, I plan to split vscode and openvsx's api into two separate crates. I have observed that openvsx has a swager definition file and am currently trying to use the tool to generate the client code. If possible, I also plan to generate vscode's api through tools.

But for now I plan to work on this repository because the idea is not mature enough yet

[spikespaz]

What tools have you looked at for generating these wrappers?

[cathaysia]

the swagger schema of openvsx is here: https://open-vsx.org/v3/api-docs/default

here are some resources maybe helpful:

• https://github.com/swagger-api/swagger-codegen
• https://github.com/OpenAPITools/openapi-generator
• https://open-vsx.org/swagger-ui/index.html

here are the steps:

. download openvsx schema file. . java -jar swagger-codegen-cli.jar generate -i openvsx.json -l openapi-yaml . openapi-generator-cli generate -i openapi.json -g rust --skip-validate-spec

You can also try it if you are interested. The generated code cannot be compiled yet. I need to further determine which step is the problem.

[spikespaz]

Well, VSCode Marketplace seems like a good task for me?

[cathaysia]

maybe we can write vscode marketplace api by openapi?

[spikespaz]

Honestly, I indeed prefer handwritten and carefully defined. But not in JSON. I'm only interested in making sure we have enough of this proprietary API to accomplish the task, quickly and asynchronously, to query, fetch, and download extensions.

[spikespaz]

Also I don't see a link to download a .jar anywhere: https://github.com/swagger-api/swagger-codegen

[cathaysia]

just run nix-shell -p openapi-generator-cli , swagger-codegen is useless, please use openapi-generator-cli.

and the wsagger-codegen download method is:

wget https://repo1.maven.org/maven2/io/swagger/codegen/v3/swagger-codegen-cli/3.0.50/swagger-codegen-cli-3.0.50.jar -O swagger-codegen-cli.jar

[cathaysia]

Honestly, I indeed prefer handwritten and carefully defined. But not in JSON. I'm only interested in making sure we have enough of this proprietary API to accomplish the task, quickly and asynchronously, to query, fetch, and download extensions.

Whether it is handwritten or automatically generated, as long as the code can complete the task correctly and reliably, I think it is feasible. The way code is generated requires further evaluation before a direction can be determined