Add support for date & time handling

[LaurenzV]

Add support for date & time handling in Typst.

Todo list

• [x] Improve error messages for parse/format failures by accessing the errors structs from the time crate.

Discussion points

• [ ] Decide whether to use icu::datetime or time as the underlying data structure.

Before finishing

• [ ] Add documentation & examples.
• [ ] Add tests.

[laurmaedje]

I haven't considered this carefully but I tend towards a combined date & time type. Also probably rather not as a built-in Value, but as a Dynamic one.

[LaurenzV]

Alright, I will combine them to a single datetime.

I'll see if I can manage to change it to dynamic, could you maybe briefly explain what the advantages of this would be (or what your reasoning is)? Just for my understanding. 😄

[laurmaedje]

Multiple things:

• It doesn't feel primitive enough to be one of the "built-in" core values
• By being dynamic it can be part of the library instead of the compiler, which is good to keep the compiler small
• If it's in a dynamic it doesn't matter whether it has a big memory footprint or is expensive to clone as its inside of an Arc anyway. Things that are directly in a Value shouldn't exceed 24 bytes so that Value stays at 32 bytes max.

[LaurenzV]

Sorry to bother again, but do you have any thoughts on the other issues I mentioned as well (e.g. whether there should be a new module, which "language" for formatting datetimes to choose and whether it would be okay to introduce a "now" (which would be very problematic considering that all functions should be pure and always return the same result))?

If you currently don't have the capacity to think about this I totally get it of course, just let me know! In that case I will just go with what I think makes most sense, and in the end you can decide whether it's okay or not. :)

[laurmaedje]

Decide whether it makes sense to add a datetime module

If these are all methods, then we don't need a Typst module, right? since it won't be part of the global namespace.

Decide whether to use the date patterns provided by the time

I also think the first one is far more readable.

Figure out how to best support localization

You might want to check out icu::datetime. We want to start using that framework for more stuff in the future.

Add method for getting the current date/time.

I think just date via today() is enough. Getting it should probably be a method on the World, then incremental compilation will automatically pick it up if it changes and recompile the necessary parts. No refresh button needed. (Btw, PDF export doesn't start from scratch, it also uses the cache.)

Decide in which way timezones should be suppported

No idea at the moment. What are the options?

[LaurenzV]

If these are all methods, then we don't need a Typst module, right? since it won't be part of the global namespace.

If we just have the datetime method, then no. But what if we also want to add something like today and parse_date? Where would you put that?

I also think the first one is far more readable.

Agreed, I really like it! And it should support most of the formats that are commonly used.

You might want to check out icu::datetime. We want to start using that framework for more stuff in the future.

I'll check it out! But I remember you saying in another thread that icu4x wouldn't work well with your web app (because of lacking WASM support)? So are you sure this would work in this case?

I think just date via today() is enough. Getting it should probably be a method on the World, then incremental compilation will automatically pick it up if it changes and recompile the necessary parts. No refresh button needed. (Btw, PDF export doesn't start from scratch, it also uses the cache.)

Didn't know this, I'll try it!

No idea at the moment. What are the options?

For the time crate, it should be pretty simple. There basically are two datetime types, the first one is PrimitiveDateTime which is timezone-agnostic and basically just consists of a Date and a Time, and then there is OffsetDateTime, where you can define an offset hour, minute and second in addition, so it's pretty straightforward. I haven't looked yet at how it works in the icu::datetime crate, but I imagine that it's going to be similar.

[laurmaedje]

If we just have the datetime method, then no. But what if we also want to add something like today and parse_date? Where would you put that?

I would be okay with today being global. Parsing could also just be datetime taking a string as an alternative input?

I've also thought about having some way to scope definitions to types, so that we can have datetime.parse and float.parse. But for that we would first need proper support for types at all! (We also could scope definitions to functions, but feels more like a hack.)

But I remember you saying in another thread that icu4x wouldn't work well with your web app

That was about icu not icu4x. The former has existed for a long time whereas the latter is brand new, written in Rust and it supports all kinds of lazy data loading, which would be nice for the web app. Icu4x didn't exist when we started Typst.

[LaurenzV]

I've also thought about having some way to scope definitions to types, so that we can have datetime.parse and float.parse.

That would definitely be nice! But yeah, I think I will disregard date parsing now and focus on constructing dates in general, date parsing can always be added as an additional feature later on.

Other than that, I think the rest is more clear now, I will let you know once I think it's ready for a review.

[LaurenzV]

I took a closer look at the icu::datetime crate, I think it would be a viable option, but as far as I've seen there I've seen, there is a big tradeoff:

• What's nice about it is definitely the localization. As you mentioned, it's also possible to lazily load the translations for certain languages, which would be very nice for the web app and could probably be implemented similarly to how it's done with fonts currently. In essence, when formatting a date, you have the choice between choosing a short, medium, long and full version. For example, for German the result would be:

short: 15.10.20
medium: 15.10.2020
long: 15. Oktober 2020
full: Donnerstag, 15. Oktober 2020

• And also, since you mentioned that you plan on using this framework more in the future, it would obviously be nice if we could stick to this one dependency instead of having to add other dependencies for specific functionality.

However, there are two issues:

• We won't really be able to add support for date arithmetic the way we have it right now (i.e. the ability to add a duration to a datetime to get a new datetime) because this doesn't seem to be supported by the crate. Of course, we could use both time and icu::datetime and convert between them as necessary, but I think this could get messy very quick.
• And more importantly: It doesn't seem possible to format a date using a custom pattern (i.e. by defining the positions of yyyy, mm etc. manually). While icu::datetime uses such patterns internally to define how the dates should be formatted in each language, they don't seem to expose any methods where I can format it by myself (I think something like this function is what I would need, but it isn't publicly exposed (probably for a good reason)). So I think this is a big downside, because I do imagine that there are a lot of people who want to customize their date formats in a different way then proposed by the standard.

But I leave it to you to decide which one you deem more important. 😄

On another note, you mentioned that you want to have only a datetime object instead of having date and time as well. I get why, but I still think this might not be the best option in our case. For example, if we implement the today method (which should only return today's date), when converting it into a datetime object we would still have to store a time along with it, and when calling the display method on it would display it like "yyyy-mm-dd hh:mm:ss" (since it makes sense to show the hour, minute and second for a datetime object by default), which is not ideal.

So my suggestion would be that I only implement a date object for now which doesn't store any time with it, because honestly, I think much more people need ways of dealing with dates than with dates + times, and if it turns out that date works well and people still seem to want a way of dealing with times, we can extend this in a different PR.

What do you think?

[laurmaedje]

Dealing only with dates for now is a good suggestion in my opinion. My only real reason for doing everything together is because date + time + datetime is just too much API surface in my opinion. I will look into ICU more.

[LaurenzV]

Dealing only with dates for now is a good suggestion in my opinion. My only real reason for doing everything together is because date + time + datetime is just too much API surface in my opinion.

Good, I will change this then.

I will look into ICU more.

Here is a good starting point for that, found this to be much more useful than the docs.rs documentation.

[LaurenzV]

I have a small question, I'm currently trying to write the documentation for the date type and I want to give a list of the components/modifiers that can be used to display a date. However, it seems like this causes a conflict because lists in Markdown are also used to generate the method signature (see my changes to types.md).

What would you suggest me to do in this case?

[laurmaedje]

What would you suggest me to do in this case?

This is the one time in your life where it comes to the rescue that Markdown has multiple supported syntaxes for lists. You can use * as a marker for your lists. :)

[alerque]

Dealing only with dates for now is a good suggestion in my opinion. My only real reason for doing everything together is because date + time + datetime is just too much API surface in my opinion. I will look into ICU more.

I do suggest involving a datetime type and steering clear of having separate date & time ones. If there isn't one library that covers what you need, the direction of bolting on a math operations to a sound datetime type is generally easier than bolting on good date & time localization, formatting, and so on to a math functions library.

Also leveraging ICU to not reinvent the wheel here will serve well. You're going to want to work out your locales handling such that variations fall back to sensible bases anyway for other i18n uses such as Fluent in #216.

[LaurenzV]

I do suggest involving a datetime type and steering clear of having separate date & time ones.

I thought about this a bit, I think one way we could go about this is to only provide a single datetime type to the user, but internally use an enum that can be either Date Time or DateTime depending on how the user initialized the datetime, which would allow us to define specific formats for each of them. So for example, if a user creates the date by writing #datetime(year: 2015, month: 6, day: 20), it will be stored as a Date internally so that when displaying it it uses the YYYY-MM-DD format by default, while if the user also provides an hour/minute, it will be stored as a DateTime instead.

The only disadvantage is that it would limit how different datetime objects can be added/subtracted from each other (since you can't really subtract a Time from a Date, but I think as long as this is explained clearly in the documentation it shouldn't be too big of an issue. Thoughts?

[LaurenzV]

@laurmaedje Unless you have some major issues with the current API design, I think this PR would be ready for a review now (no rush though). :)

The only thing that's still missing is making today part of the datetime function scope, which I will do once #1032 is merged.

Let me know what you think!

[stepbrobd]

I was just Googling how to add current date to typst and found this open PR, nice work!

[PgBiel]

So @LaurenzV , after doing some reading here, I must say, this is looking impressive; good job! Just got a few comments/questions:

1. I wonder if it'd be viable to store timezone in dates, instead of having all of them be naive? It would certainly make many date operations easier. Then one could use some sort of .is_naive() method which just checks if the .offset property is none or a number. Overall, it would be nice to have some first-class support for timezones. (Edit: also methods to convert between timezones, and to UTC)
2. Related to (but not necessarily a consequence of) the above, I think today() should have an option like offset: X to get the current date at a certain timezone (e.g. today(offset: -3) for today at UTC-3). Chrono makes this easy by using FixedOffset as the timezone type.
3. It would be cool to have something like display(iso8601: true) to default to displaying the datetime with the ISO 8601 format.
4. I think it'd be nice to have a weekday() function (that's something I'd certainly use), which should return e.g. an integer between 0 and 6 according to some convention (e.g. 0 = sunday) to indicate the current weekday.
5. Please add a newline at the end of the test file it hurts my eyes AAAAAAA (lol)

Aaand that's it! Hope those ideas aren't too much? :p

[LaurenzV]

I wonder if it'd be viable to store timezone in dates, instead of having all of them be naive? It would certainly make many date operations easier. Then one could use some sort of .is_naive() method which just checks if the .offset property is none or a number. Overall, it would be nice to have some first-class support for timezones. (Edit: also methods to convert between timezones, and to UTC)

Hmm, at least considering the way we currently have it (having only one exposed datetime function which handles everything from date, time to datetime), I don't think this makes much sense, as a timezone would only make sense for the datetime, but not for date and time alone. But if we do want to add it, it should be possible to add it afterwards without breaking much of the current API, so I'm thinking maybe this could be explored as part of a different PR? I'm just not sure whether this would be a very necessary feature for now, I think 95% of the use cases are already covered by being able to get the current date and displaying it in a custom format.

Related to (but not necessarily a consequence of) the above, I think today() should have an option like offset: X to get the current date at a certain timezone (e.g. today(offset: -3) for today at UTC-3). Chrono makes this easy by using FixedOffset as the timezone type.

Yeah I think this is a good idea, I think I will change that! But still, for now I would limit it to getting the system date in the provided offset, but internally I would still store it as a plain date without any additional information (since again, just a date with only time zone information doesn't make that much sense in my opinion.

It would be cool to have something like display(iso8601: true) to default to displaying the datetime with the ISO 8601 format.

Isn't the current default already in the ISO format (except for the fact that we don't have milliseconds, which could easily be added, just wasn't sure whether it's really necessary for now)?

I think it'd be nice to have a weekday() function (that's something I'd certainly use), which should return e.g. an integer between 0 and 6 according to some convention (e.g. 0 = sunday) to indicate the current weekday.

I can add that! You actually can already display the weekday, I just didn't add a function to expose it in the Rust code.

Please add a newline at the end of the test file it hurts my eyes AAAAAAA (lol)

Will change. :p

Thanks for the feedback, much appreciated!

[PgBiel]

Hmm, at least considering the way we currently have it (having only one exposed datetime function which handles everything from date, time to datetime), I don't think this makes much sense, as a timezone would only make sense for the datetime, but not for date and time alone.

That's a fair point; I hadn't considered that. Indeed, this would require some rather structural changes.

But if we do want to add it, it should be possible to add it afterwards without breaking much of the current API, so I'm thinking maybe this could be explored as part of a different PR? I'm just not sure whether this would be a very necessary feature for now, I think 95% of the use cases are already covered by being able to get the current date and displaying it in a custom format.

Yeah, that's 100% fair. I agree this PR should cover the majority of use-cases.

But still, for now I would limit it to getting the system date in the provided offset, but internally I would still store it as a plain date without any additional information (since again, just a date with only time zone information doesn't make that much sense in my opinion.

Of course.

Isn't the current default already in the ISO format ()?

I meant with the T syntax (instead of space), as that's sometimes required. This isn't a strict requirement though, as one can use custom formats for this; just thought it's so common that it could warrant some special option for it too, maybe?

I can add that! You actually can already display the weekday, I just didn't add a function to expose it in the Rust code.

Yeah, I think this would be very useful programmatically :+1:

Will change. :p

:tada: :rocket: :+1:

Thanks for the feedback, much appreciated!

You're welcome! Thanks for the PR.

[LaurenzV]

Should all be done now. :D

[PgBiel]

Nice, good job! Just to say, I got a little curious; is weekday "a number or none" because invalid dates can be specified? If so, would it be worthwhile to add checks to datetimes? (Or maybe just add checks when timezones are eventually added?)

[LaurenzV]

Nope, the reason is that if I define something like datetime(hour: 14, minute: 40, second: 20), it will be stored as a time object internally, which obviously doesn't have an associated weekday. That's also the reason why all of the other fields can return none as well.

[PgBiel]

Perfect! Thanks for the explanation! :+1:

[LaurenzV]

today is now part of the function scope of datetime! However, I didn't know how the documentation for methods in function scopes should look like, so I haven't updated that part yet.

[laurmaedje]

I didn't know how the documentation for methods in function scopes should look like

I haven't updated the docs generation system yet.

[LaurenzV]

Okay, I hope I fixed everything that needs to be fixed. :D

[laurmaedje]

I think we're pretty much there. I pushed a small tidying commit and made some final remarks. (Sorry for bothering you with so much review comments.)

[PgBiel]

Hey, last minute thing haha, but I forgot to mention this earlier... would it be possible to have a Unix timestamp method as well? (Maybe with a customizable epoch? But that's not 100% necessary)

Edit: just saying this because I thought it'd be quick, but a follow-up PR could cover this as well if needed

[LaurenzV]

Hey, last minute thing haha, but I forgot to mention this earlier... would it be possible to have a Unix timestamp method as well? (Maybe with a customizable epoch? But that's not 100% necessary)

I wouldn't have anything against it, but I also think that's something that we can easily add in another PR.

Another thing I noticed, maybe it would be nice if the display method could take a function callback as well instead of just a string? This would allow people to create their own format and temporarily overcome the issue with missing localization, e.g. you could do

#let date = datetime.today().display(date => {
  let monthMap = {
    1: "Jänner",
    2: "Februar",
    3: "März",
    etc...
  }

  let translated-month = monthMap.at(date.month)
})

And do the same for everything else and then just construct the displayed datetime based on that. But again, I think we can also leave that for another PR. 😄 Not sure how urgently @laurmaedje wants to merge this.