malli
malli copied to clipboard
metosin
Better support for dates
Maybe :date-time and :local-date mapping to Java8 Time & Google dates? Or should we lean on libs like tick?
Agree. Looking at JSON Schema dates and java.time classes, maybe we should have these implemented:
:date->java.util.Date,:date-timein JSON Schema:date-time->java.time.Instant,:date-timein JSON Schema:local-time->java.time.LocalTime,:timein JSON Schema:local-date->java.time.LocalDate,:local-datein JSON Schema
not sure what the js classes would be, maybe Date for :date and the corresponding versions of goog.date for the rest?
Just linking this here for consideration: https://github.com/henryw374/cljc.java-time
I've done no research, just wanted to make sure it was known to exist for when this is implemented.
There is a discussion about tick, cljc-java-time and the future of js dates in clojureverse. It seems that there isn't a lightweight clj/s option available right now. Suggestion: malli will start with own malli.time ns, which has just the needed mappings to make java8 dates available that work with js too. For js, it would be all js/Dates for now.
Is this a bad idea?
suggestion
| malli | java | javascript | json schma | example string | info |
|---|---|---|---|---|---|
| :date | java.util.Date | js/Date | "date-time" | "2013-07-16T19:20:51Z" | java/clojure legacy |
| :instant | java.time.Instant | js/Date | "date-time" | "2013-07-16T19:20:51Z" | UTC, system-time |
| :date-time | java.time.OffsetDateTime | js/Date | "date-time" | "2013-07-16T19:23:51+03:00" | with offset, user-time |
| :local-time | java.time.LocalTime | js/Date | "time" | "09:15" | kids go to school |
| :local-date | java.time.LocalDate | js/Date | "date" | "2013-07-16" | wedding date |
did you @rschmukler have something done for this in your projects already? coudn't find any repo but have a feeling you did.
Personal opinion: When I've needed to do anything more interesting than show a date picker in cljs I've reached for cljc.java-time (or tick). It makes sense to not pull in something "heavy" by default - but is there a way to make using e.g. cljc.java-time opt-in?
For the cases where one actually has time related logic it is very likely one will be pulling in something like cljc-java-time - it's just too much logic around time to deal with ad-hoc logic on top of plain js/Date. It'd be a shame to end up needing to implement and use ones own malli schemas (e.g. :tick-date, :tick-instant, or :cjt-local-time, :cjt-date-time).
Perhaps if one needs to opt-in for "any" time related ns then a "plain" time implementation can exist alongside one or more (in separate ns) that rely on something more heavy weight? They'd all have the same json-schema and so forth, but the cljs side would utilize different code.
@ikitommi I personally use tick in almost all of my projects. It's worth noting that tick uses the cljc.java-time primitives.
I wonder if, as part of dropping foreign libraries, cljc.java-time could also library-ify its primitive types. This would potentially allow for a minimal dependency that malli could use, while leaving heavy lifting to other libraries. Perhaps @henryw374 could weigh in on the feasibility of this, or whether its too many layers of indirection to be useful.
I haven't yet done anything this full featured in my projects, but I'd be happy to contribute once we land on some of the design questions. One thing that makes this all difficult is allowing for the control of the serialized representation, and the implicit need for unifying format strings across clj and cljs. This is especially useful when writing schemas / transformers for JSON APIs that you might not control. Ideally we could have a :time/format property that allowed specifying the serialized representation. Consider the following:
[:map
[:created-at {:time/format "YYYY-MM-DD"} :time/local-date]]
The vast majority of my work is in clj, with only a little cljs.
I also use tick in almost all my projects.
I have a strong need for java.time.ZonedDate.Time, and I also use java.time.LocalDate, and when transforming between Postgres, I need java.sql.Date and java.sql.Timestamp, although I cannot imagine needing either of those two in cljs.
Currently I provide my own support for this stuff, like this:
(defn zoned-date-time?
[x]
(instance? java.time.ZonedDateTime x))
(defn local-date?
[x]
(instance? java.time.LocalDate x))
(defn sql-date?
[x]
(instance? java.sql.Date x))
And:
(def registry
(merge (m/predicate-schemas)
(m/class-schemas)
(m/comparator-schemas)
(m/base-schemas)
{:zoned-date-time (m/fn-schema :zoned-date-time #'zoned-date-time?)
:local-date (m/fn-schema :local-date #'local-date?)}))
As long as I can keep adding support for time-date formats I need that are not going to be bundled into Malli, then I can understand/appreciate not wanting to drag in large dependencies like tick, especially on the cljs side....
A few points on all this:
-
As others have said, there aren't tick or cljc.java-time entities - those clj/s libs are just sugar fns that operate on java.time and js-joda entities. So the question is whether malli will support java.time, js-joda & etc. I have a work in-progress mod of js-joda that is DCE friendly to some extent, in that it is pretty much all or nothing.... so IOW not something you could really call lightweight.
-
Maybe
malli.timecould define some protocols for whatever date ops it needs & those could be extended to whatever entities ppl are using, java.time, joda, js/Date, js-joda, string?? ... & Temporal (https://github.com/tc39/proposal-temporal which I expect everyone to be using in the not too distant future). IOW user chooses and pays just the baggage cost which is right for them. -
Formatting/parsing is Locale-specific - hence why js-joda has a range of locale specific add-ons so to reduce bundle size - https://www.npmjs.com/package/@js-joda/locale. It's hard to imagine anything 'lightweight' offering locale-aware parsing/formatting. Hopefully Temporal will include this though - they say they are considering it https://github.com/tc39/proposal-temporal/issues/796.
Just a question, is there something i can use ATM other than [:fn ...] schemas?
Here is some code from an internal application. I might flesh this out some more and roll it into teknql/wing which is my opinionated extension to the standard library. This could also be useful if you want to implement / extend your own. Currently I rely on tick:
(defn time-transformer
"Custom malli transformer for working with times"
[]
(mt/transformer
{:encoders
{'inst? {:compile
(fn [schema _]
(let [{:keys [json/type
json/duration]
:or {type :int
duration :millis}} (m/properties schema)
epoch (t/epoch)
inst->int (let [duration-f (case duration
:seconds t/seconds
:millis t/millis)]
#(duration-f
(t/between epoch %)))]
(case type
:string {:enter str}
:string-int {:enter (comp str inst->int)}
:int {:enter inst->int})))}
:date {:compile
(fn [schema _]
(let [{:keys [json/format]
:or {format "yyyy-MM-dd"}} (m/properties schema)
formatter (t/formatter format)]
{:enter #(if (t/date? %)
(t/format formatter %)
%)}))}}}
{:decoders
{:date {:compile
(fn [schema _]
(let [{:keys [json/format]
:or {format "yyyy-MM-dd"}} (m/properties schema)
formatter (t/formatter format)
str->date #?(:clj
#(java.time.LocalDate/from (.parse formatter %))
:cljs
#(cljc.java-time.local-date/parse % formatter))]
{:enter #(if (string? %)
(str->date %)
%)}))}
'inst? {:compile
(fn [schema _]
(let [{:keys [json/type
json/duration]
:or {type :int
duration :millis}} (m/properties schema)
epoch (t/epoch)]
(case type
:string {:enter t/instant}
:string-int {:enter #(t/>> epoch
(t/new-duration #?(:clj (read-string %)
:cljs (js/parseInt %)) duration))}
:int {:enter (fn [val]
(t/>> epoch
(t/new-duration val duration)))})))}}}))
Similarly, my registry looks like this:
(def custom-registry
{:date (m/-simple-schema {:pred t/date? :type :date})})
(m/encode [:date {:json/format "MM/dd/yyyy"}] #time/date "2020-01-05 {:registry custom-registry} (time-transformer))
=> "01/05/2020"
;; Copy pasted some tests, let me know if you would like further explaination
(let [val #time/instant "2020-11-19T22:06:33.177592119Z"]
(are [schema expected] (let [encoded (m/encode schema val (time-transformer)))
decoded (m/decode schema encoded (time-transformer))]
(is (= expected encoded))
(is (= (t/truncate val :seconds)
(t/truncate decoded :seconds))))
[inst?] 1605823593177
[inst? {:json/duration :seconds}] 1605823593
[inst? {:json/type :string-int}] "1605823593177"
[inst? {:json/type :string}] "2020-11-19T22:06:33.177592119Z"))
fyi update on the js platform date API 'Temporal'
- Temporal API has reached ‘stage 3’ - meaning it’s out for implementors
- I’ve been thinking about a Clojure take on java.time+Temporal here: https://github.com/henryw374/tempo
Thanks 😊
On Tue, 23 Mar 2021, 19:38 Tommi Reiman, @.***> wrote:
@henryw374 https://github.com/henryw374 looking forward to tempo, like the rationale.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/metosin/malli/issues/49#issuecomment-805179638, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXNSDHGGHXM7OQGONFQUUDTFDU3HANCNFSM4IPWFBZA .
Took a swing at an implementation without knowing this discussion existed, here is what I originally came up with:
(ns malli.time
(:require
[clojure.string :as str]
[malli.core :as m])
(:import
(java.time Duration LocalDate OffsetDateTime OffsetTime)))
(defn ->parser
[f]
(fn [x]
(if (string? x)
(try
(f x)
(catch Exception _ x))
x)))
(defn -string->duration [x]
(Duration/parse x))
(defn -string->offset-date-time [x]
(OffsetDateTime/parse x))
(defn -string->local-date [x]
(LocalDate/parse x))
(defn -string->offset-time [x]
(OffsetTime/parse x))
(def default-date-types
{:date {:class LocalDate :parser -string->local-date}
:time {:class OffsetTime :parser -string->offset-time}
:date-time {:class OffsetDateTime :parser -string->offset-date-time}
:duration {:class Duration :parser -string->duration}})
(defn class-base-name
[^Class c]
(last (str/split (.getName c) #"\.")))
(defn date-schema
([type]
(when-let [props (get default-date-types type)]
(let [{klass :class parser :parser} props
pred #(instance? klass %)
safe-parser (->parser parser)
message (str "Should be " (name type) " or " (class-base-name klass))]
(m/-simple-schema
{:type type
:type-properties
{:error/fn (fn [{:keys [value]} _]
(if (string? value)
(try
(parser value)
(catch Exception e (ex-message e)))
message))
:decode/json {:enter safe-parser}
:json-schema/type (name type)}
:pred pred})))))
As Henry pointed out, Duration's implementation is not compliant with the ISO
There are two competing drives here, one is supporting the strict standard subset of formats required for JSON schema, the other is better support for time. They're competing because a good support for time formats will probably allow things not supported by JSON Schema.
Do you think the mapping to Java classes here is correct with regards to JSON schema? worst case scenario we can have schema for stuff like :date-time and :iso/date-time.
Relating to your suggestion for class mapping, I'd rather not coerce anything by default to java.util.Date.
Regarding schema names, should we align more towards JSON schema or towards the implementing java classes?
@henryw374 any news on tempo / temporal? would like to merge in some initial impl for common dates for both clj/cljs. Or then just the CLJ version first (#545)
no news - still stage 3 & still lots of development activity.
I've no idea how long it'll take either. A prediction was made by one of the main devs that it would be in browsers by end of 2021 ... apparently not.
thanks for the quick reply! I would really like malli to get the time schema names correct from day1, even if it's just the JVM / java.time version. Is the spec frozen enough that we would know the good set of names & relevant Time classes? what would the set of classes & names be?
e.g. #545 has now:
(def default-date-types
{:local-date {:class LocalDate :parser -string->local-date :json-schema/type :date}
:offset-time {:class OffsetTime :parser -string->offset-time :json-schema/type :time}
:offset-date-time {:class OffsetDateTime :parser -string->offset-date-time :json-schema/type :date-time}
:duration {:class Duration :parser -string->duration}})
The names have been stable for well over a year now and cannot really change now since it's Stage 3 if I understand corrrectly. The table of contents at the top of https://tc39.es/proposal-temporal/docs/ shows the classes.
So it looks like the #545 things are the date-time types from https://json-schema.org/understanding-json-schema/reference/string.html#dates-and-times but with the Java.time names. Looking at those in turn:
- LocalDate. In Temporal this is called PlainDate.
- OffsetTime. In Temporal there is just PlainTime. There is no equivalent entity of time+offset.
- OffsetDateTime. In Temporal there is only the ZonedDateTime class for the
point in time, in a placeidea. Temporal Instant would parse the RFC3339 date-time format e.g.'1995-12-07T03:24:30+02:00'but would lose the offset ofc. Temporal ZonedDateTime would actually not even parse that format, because it requires a zone name e.g.'1995-12-07T03:24:30+02:00[+02:00]' - Duration - here I think Temporal's Duration is a natural fit, but java.time Duration isn't. Java.time Duration will not parse date-based parts e.g.
P3Ybc it has split those bits off into java.time.Period.
So.. all in all, not a straightforward match up :(
As a note: my current favorite solution for Cljs, when proper timezone support is required, is using js-joda directly. The API is mostly the same as java.time so cljc code doesn't need much reader conditionals.
If I understand correctly, supporting js-joda classes directly, would also support cljc.java-time, without enforcing that users must use js-joda through cljc.java-time.
Cljs implementation should be pluggable so we can provide versions for both js-joda, perhaps goog.date, and eventually Temporal API.
using jsjoda directly would be compatible with cljc.java-time, but FYI cljc.java-time depends on https://github.com/henryw374/cljs.java-time - this library just does a few things to make jsjoda more usable from cljs: it adds Clojurescript's equivalence, hash and comparison protocols to jsjoda objects and provides externs.
Hello. I see the recent release has support for java.time on the jvm :). I have begun an impl for js-joda here.
Some points to note:
- This assumes that other date-time implementations will live in the malli lib and have obvious ns-names like the js-joda one now has. Users then just require one of these that they're going to use.
- I have mostly copy/pasted over from existing java.time impl, changing the (Static/method call) syntax to the Dot notation '(. Static method call)` - doing this there is plenty of scope for factoring out the common bits bc this syntax works for both js-joda and java.time.
- I have not put js-joda in a deps.cljs dependency (ie made js-joda npm a dependency of malli), bc I'm not sure all the npm tools (shadow, bundle etc) would leave it out of the build (if the client code base doesn't require an ns that includes the js-joda require)? IOW this would be a bring-your-own js-joda kind of thing, either via npm or cljsjs.
- I have run the tests in local shadow dev only (my own config). I expect the existing cljs test setup would need to include npm stuff (for example as in the kaocha-cljs2 setup here) - alternatively use a cljsjs js-joda but then the various shadow builds you have might not work.
- I'm not sure I will get round to finishing off this impl or adding js-date, Temporal etc in the near future - so if anybody else wants to finish it off please do.
- I've started with Duration at random. Everything else probably 'just works' if commented out. A few
^jstype hints might be needed where the js-joda methods are called
this would be a bring-your-own js-joda kind of thing, either via npm or cljsjs.
:+1: I think this is the correct approach for JS libs.
alternatively use a cljsjs js-joda but then the various shadow builds you have might not work.
I'd avoid using any cljsjs packages.
Closing this, we have now optional :time/*** schemas using java.time and js-joda. It's under malli.experimental.time for now. Looking forward to seeing moving forward https://github.com/tc39/proposal-temporal. Big thanks to @bsless and @dvingo for all the work on this!!