feat: Add queue name prefix setting

[dax]

When using Apalis in tests, it is useful to isolate queues used by tests to avoid mixing jobs between them when they are executed concurrently. To create this isolation, this PR introduces a queue_name_prefix setting to prefix queue names when some value is set.

[geofmureithi]

What is broken specifically? J::NAME allows someone to provide a namespace.

[geofmureithi]

Also, I don't see anything related to tests.

[dax]

Sorry, my description was not clear enough. For tests (in my app, not in the lib), I need to make the queue name dynamic, each test must have a unique queue name to avoid mixing jobs between messages. Example:

• test1 starts worker1 and creates job1
• test2 starts worker2 and creates job2

I would like to avoid worker1 to handle job2 or worker2 to handle job1 when tests are executed concurrently (In my case, each test/worker pair has a dedicated database schema to work with). With a static (ie. at compile time) queue name, there is no way to assign jobs to dedicated workers. This PR introduces a namespace that can be added per test (ie. at runtime, not compile time).

I hope this helps 🙏

[geofmureithi]

I think you are looking for --test-threads=1 https://github.com/geofmureithi/apalis/blob/05cbe06253b7aad7f0b8b037027f3adf798a42c3/.github/workflows/redis.yaml#L33

[dax]

@geofmureithi Of course, I could limit the test thread count to 1 but this defeats the purpose of having parallel test execution and thus slowing their execution down 🤷

[geofmureithi]

Isn't it better to adjust your tests to fit the code, instead of adding more code just to fit your test requirements? That part was done intentionally, its the approach used for all other backends and the recommended way to prevent tests from interacting with each other according to the docs. Tests will possibly be slower for a few seconds. Also it might be wise to let tests emulate the actual production environment. Another solution might be to create different structs for each test (quick macro may be handy) with a setup and teardown. If you can share some code, it might help see the specific pain point.

[dax]

Here are the tests. They call an API endpoint that creates jobs asynchronously handled here.

A unique API, worker and DB (ie. database name) are created for each test here. I just wanted to extend the setup to have a unique queue name too.

[geofmureithi]

I see your point. Could you consider annotating the parts you have added with #[cfg(test)] and #[doc(hidden)]? It might also require the tests to be updated and removing --test-threads=1 from the tests to avoid any breaking changes in the future.

[dax]

Could you consider annotating the parts you have added with #[cfg(test)] and #[doc(hidden)]?

I'm not 💯 sure but if I add #[cfg(test)] to my changes here, would I still be able to use this API from my own project tests? (ie. is the cfg(test) also apply to my project dependencies?) 🤔

[geofmureithi]

Hmm you may be right. I know this is a tricky thing, but could we put a pin on this then I can look at a solution for the end of the week. I think API parity is important it may need to be applied on other backends.

[dax]

I have made a test with #[cfg(test)] and indeed, it does not work for dependencies. I tried another thing by declaring a new feature test in apalis-redis and it does the trick.

Let me know what you prefer or if you have another idea. Nothing urgent 🙏

[geofmureithi]

Yeah it would not work. This issue is related to https://github.com/geofmureithi/apalis/discussions/191 Let me look into it over the weekend.

[geofmureithi]

To easily achieve this, I think we should just drop the traits Job and Message. I am considering adding a namespace field on Worker.

[geofmureithi]

Closed because of #338