Quartz.NET 4.0 Roadmap

[lahma]

This is an umbrella issue gathering things that could be done in 4.0 timeframe.

• [ ] Refactor persistence API to be more document database friendly, https://github.com/quartznet/quartznet/issues/814 probably contributes to this
• [x] Use ValueTask instead of Task, RamJobStore doesn't need heavy machinery
• [x] Use Microsoft.Extensions.Logging.Abstractions instead of liblog
• [x] Add HTTP API for scheduler, maybe via Quartz.AspNetCore package, existing issue #511
• [x] Use the new TimeProvider abstraction #2079
• [x] Remove binary serialization support #2324
• [x] Remove remoting support #2447
• [x] Remove full framework support #2226

[Sicos1977]

A replacement for .net remoting like a rest api would also be a nice thing

[lahma]

Thanks @Sicos1977 , added https://github.com/quartznet/quartznet/issues/511 to the list

[Pinox]

An option to load persistent data from a cloud provider like Azure Table Storage (No SQL) would be an awesome low cost solution for running quartz.

Currently I have quartz persistent data running on a separate SQL DB and it's such a waste of money especially the fact that I'm not running many jobs on the DB. Table storage where you only pay for the data that you use will make much more sense for many people as I think most Quartz db's are probably very small in size. Table storage is cheap, fast and accessible from anywhere with near infinite scaling. A 1GB solution on Azure Table Storage will cost you $0.13 per GB

https://azure.microsoft.com/en-gb/services/storage/tables/

[lahma]

@Pinox what I've understood the row key is used for fast lookups in table storage. Quartz needs to check next fire time which is changing value for triggers and if you have thousands of triggers might become a problem. Azure table storage might not scale well in this case?

[Pinox]

@lahma I know there is a limit for transactions (20 000 per second) and entities (2000 per second.) per partition. So yes if you do more than thousands per second it will not scale well on Azure Table Storage but you still have general NoSQL DB's as option for such scenario's.

I just think if Quartz was able to work on NoSQL then for smaller type workloads it will be a perfect fit for a NoSQL solution such as Azure Table Storage and if you really need massive record update per second then you can always throw it in bigger NoSQL DB like CosmoDB / MongoDB and then you are can use the most appropriate hardware for your task that is persistent and very cost effective.

SQL Server now supports JSON format , so you can also run a "NoSQL database" on SQL Server without the relational schema that is usually the case.

https://docs.microsoft.com/en-us/sql/relational-databases/json/store-json-documents-in-sql-tables?view=sql-server-ver15

[ghd258]

MYSQL 8.0 now supports JSON format , so you can also run a "NoSQL database" on MYSQL 8.0 without the relational schema that is usually the case.

[DaveLaa]

It would be great to have a sepparation of functionality into specific libraries. There could be a Quartz.Net Core dll without external dependencies and specific dlls für the different extensions. This would reduce the necessity to add libraries to a project that doesn't use them.

[lahma]

There are some libraries that are ubiquitous, like logging tracing etc so I would consider them as the necessary evil. Especially with .NET Core / NET5 it's quite normal that libraries that need to plug-in into larger constructs have these. DI system is a bit on the fence, but building a bridge is also an option.

[Pinox]

FYI - new timer API approved by MS.

https://github.com/dotnet/runtime/issues/31525

[cd21h]

Please add job execution pipeline to allow applying cross-cutting concerns to job handlers.

Something similar to ASP.NET Core middleware https://docs.microsoft.com/en-us/aspnet/core/fundamentals/middleware/write?view=aspnetcore-5.0

[Vandersteen]

Middleware pipeline would be a welcome addition for us. I know there are listeners, but they do not allow to 'wrap' things like: https://github.com/quartznet/quartznet/discussions/1224

[lahma]

Middleware pipeline would be a welcome addition for us. I know there are listeners, but they do not allow to 'wrap' things like: https://github.com/quartznet/quartznet/discussions/1224

I think a PR would be most welcome. We can start breaking things in main as current version can be branched to 3.x maintenance.

[Doomer3D]

Please replace Json.NET with System.Text.Json for .NET 5+

[mkstephenson]

I would also second the replacement of Json.NET if it is possible

[lahma]

If someone wants to contribute a STJ implementation the same way there's a separate project for NSJ integration, that would be nice. We could offer it in 3.x timeline and make it default in 4.0.

[mkstephenson]

I'll have a look at it if I have time, can't promise anything at this stage though.

[tonyqus]

Please replace Json.NET with System.Text.Json for .NET 5+

I don't think System.Text is mature enough so far. JSON.NET is still much better than System.Text. If you have tried JSON Path Selector feature, you will understand what I mean. Don't replace a library just because the other one is newer.

[lahma]

Please replace Json.NET with System.Text.Json for .NET 5+

I don't think System.Text is mature enough so far. JSON.NET is still much better than System.Text. If you have tried JSON Path Selector feature, you will understand what I mean. Don't replace a library just because the other one is newer.

I think we can support both, JSON.NET shouldn't just disappear to ensure existing users keeping the old desired behavior. I can imagine there would be regressions with just changing to STJ, hard to predict things that can go wrong.

[tonyqus]

Please consider replacing MS DI with AutoFac. We did see a few unexpected issues happen in MS DI and Scoped lifecycle are not reliable at all. I believe it's kind of design issue of MS DI.

[tonyqus]

I'm reading some comparison between Quartz.NET and Hangfire. The major complaint is about how Jobs are created. Hangfire supports any method while Quartz.NET only supports IJob based tasks.

For example,

BackgroundJob.Enqueue(
    () => Console.WriteLine("Fire-and-forget!"))

I haven't dive deep into the code. Do you think it's possible to support any method with automatic IJob task creation?

[tonyqus]

Another major complaint is about automatic creation of database schema. Since Quartz.NET use sql script instead of code-first ORM (such as Entity Framework), the initialization setup experience is not as good as Hangfire.

Do you think we can add Entity framework based database schema creation feature? If you agree adding this to 4.0, I'm willing to work on that.

[lahma]

Please consider replacing MS DI with AutoFac. We did see a few unexpected issues happen in MS DI and Scoped lifecycle are not reliable at all. I believe it's kind of design issue of MS DI.

Or Lamar, or DryIoc.. the list goes on. There's already an Autofac integration that you can use.

I haven't dive deep into the code. Do you think it's possible to support any method with automatic IJob task creation?

I don't personally like the Hangfire way of things when we start to talk about persisted jobs. Clean and simple for in-memory jobs but complexity starts to hit with things like () => someService.Method(counter++), how to handle all context and captures, makes things harder to test too IMO.

Do you think we can add Entity framework based database schema creation feature? If you agree adding this to 4.0, I'm willing to work on that.

I don't think we need EF to just handle schema. Because schema doesn't change that often I think it would be enough to detect via switch whether intitial schema creation is allowed and the schema is missing. Maybe add support for detecting database schema version and if it's different from what Quartz requires, give a startup error.

[Cricle]

Will support nosql like MongoDB, LiteDB store?

#1019 I try but there is no suitable one

[lahma]

Will support nosql like MongoDB, LiteDB store?

#1019 I try but there is no suitable one

The core project don't have resources to support such specialized one, you can always improve the existing ones though.

[Cricle]

The core project don't have resources to support such specialized one, you can always improve the existing ones though.

I fork https://github.com/glucaci/mongodb-quartz-net and make it like #1480 in PR https://github.com/glucaci/mongodb-quartz-net/pull/38

But I don't think the PR status change will happen recently. So I publish Nuget package .

Thanks for everyone's efforts.

[Misiu]

1. Add support for Redis. There is a project that adds Redis and has a PR that allows support for Quartz 3 (https://github.com/icyice80/QuartzRedisJobStore/pull/4) but official support would be awesome. Especially that the project has some limitations - https://github.com/RedisLabs/redis-quartz#limitations

2. Add documentation and sample on how to configure and use clustering - I have 3 instances of .NET 6 REST API behind Traffic, so I'd like to be able to easily configure clustering in Quartz.

3. Simplify how we register and set up Job Stores. Take a look at https://github.com/glucaci/mongodb-quartz-net#basic-usage, you must set properties (NameValueCollection). Ideally we should be able to do the same using fluent API.

[Cricle]

1. Add support for Redis. There is a project that adds Redis and has a PR that allows support for Quartz 3 (Ported to Quartz.net 3 icyice80/QuartzRedisJobStore#4) but official support would be awesome. Especially that the project has some limitations - https://github.com/RedisLabs/redis-quartz#limitations
2. Add documentation and sample on how to configure and use clustering - I have 3 instances of .NET 6 REST API behind Traffic, so I'd like to be able to easily configure clustering in Quartz.
3. Simplify how we register and set up Job Stores. Take a look at https://github.com/glucaci/mongodb-quartz-net#basic-usage, you must set properties (NameValueCollection). Ideally we should be able to do the same using fluent API.

I think 3. can be a PR(Although the project has been passively updated), if it display or I solved it, I will merge in temp Nuget package

[s3YwCf2ZbfJG4SHAfjQMAjtsf]

I don't think we need EF to just handle schema. Because schema doesn't change that often I think it would be enough to detect via switch whether intitial schema creation is allowed and the schema is missing. Maybe add support for detecting database schema version and if it's different from what Quartz requires, give a startup error.

Hangfire provides additional nugets to handle schemas with NHibernate. Yesterday, I started with Quartz.Net because of the issues I was having with Hangfire and Oracle and .Net 6. After a few hours, it was great that the schema script was available and saved me a great deal of time. Almost all of the requirements have been handled in just a few hours. I need to go back and handle load balancing with the AdoJobStore.

[michaeljon]

Has anyone done any work getting System.Text.Json as a supported serializer? I'd be willing to pick up and help. We have some really hairy JSON structures and I would not want to have to go back in and write Json.Net serializers for them.

[mkstephenson]

I started to look at it and got some stuff down, however I got stuck around some of the structures (and figuring out some of the tests), then some other stuff came up and I didn't get round to finishing it. I just checked the repo now and there appears to be some stuff surrounding System.Text.Json? However I don't know how complete it is.