filter parameter for limited sets of calendar events

[jfacemyer]

I looked around a bit, but didn't see an option for changing output content, not giving the vigil information on Saturdays, or preferring a Saint's proper memorial over an optional BVM on Saturdays, and only give the BVM if there's nothing else? (e.g. output the highest ranking or a preferred type each day?)

I was thinking about digging in and looking at implementing this, possibly, but wanted to know if this exists or was desired or was in the works already.

[JohnRDOrazio]

Hi @jfacemyer , the main purpose of this API is to output the correct liturgical celebrations for each day in a given calendar year, either in the universal roman calendar or for a specific nation or even a specific diocese. If there is an optional memorial on any given day, it will be output alongside the weekday, because it's optional. So if for example a Mass app were to use this calendar, there would be a choice of using the liturgical texts for the optional memorial or for the weekday. So in cases like that, no there is no option to output only one result, because how would choose which result to output? They're both equally important. I decided to incorporate Vigils into the calendar because that is also an important liturgical calculation: when there are two solemnities next to each other, which one has precedence when it comes to vigil masses / first vespers / second vespers? This is again important for any Mass app or breviary app or diocesan liturgical calendar, because it will determine which liturgical texts should be used for first vespers / second vespers or which Mass should be celebrated. The Saturday memorial of the Blessed Virgin Mary should only appear where it is permitted. If you see it appearing where it is not permitted please let me know.

[jfacemyer]

Ok, I get that.

Does it mean you would not be interested in a parameter to filter output like this, if I wanted to work on that?

[JohnRDOrazio]

I believe that if anything this could be done in a client implementation. Changing it on the server output would in my opinion be confusing and would require creation of more parameters which would make the endpoint even more confusing. However it is not difficult to create filters in a client implementation. For example, to remove vigil masses, you can delete any items in the JSON object that contain the word "vigil" in the name of the celebration (if your client is using English). I suppose to make it even easier to do for any language, I could create a new property "isVigilMass" in the resulting JSON object to clearly identify Vigil Masses, so as to be able to target them and filter them out. In fact, celebrations that have Vigil Masses already have a property "hasVigilMass" set to true, for example:

"MotherGod":{"name":"Maria Ss.ma Madre di Dio","color":"white","type":"fixed","grade":6,"common":"","date":"1609459200","displaygrade":"","eventidx":44,"liturgicalyear":"ANNO B","hasVigilMass":true,"hasVesperI":true,"hasVesperII":true}

But being able to target the actual Vigil Mass event isn't quite so straightforward, without searching for the term "vigil" in the name of the celebration. That I can do.

[JohnRDOrazio]

I have added the isVigilMass property to better identify Vigil Masses:

• 9cd59f1c0b6588b947dc44b5320d0f3c5c78be81
• 644ca568102095e338750db7ec72b101a38d5685 It is now available for usage on the staging API endpoint, for example: https://johnromanodorazio.com/LiturgicalCalendar-staging/LitCalEngine.php?returntype=JSON

Now it is very easy to target Vigil Mass events and filter them out from the results. I hope this fixes for you.

[jfacemyer]

Ok, thanks - I'll look at a client implementation!

[HPaulson]

@JohnRDOrazio I was going to make a similar suggestion/offer to implement filters and found this old issue.

It makes sense for a mass readings or breviary app that filtering should be done client-side, but I'm wondering about the iCal use-case. For people who use the API to subscribe to events inside their calendar app, it might be useful to have an option to filter by type of feast or other filters. One might only want to subscribe for solemnities, for example. This makes a ton of sense because the weekdays, optional memorials, etc can be very bloating to a calendar already filled with work and personal events. So a user might still want to see what the non-feria days on the calendar are, but not the rest. This is my reason for desiring such a filter feature, at least. I wonder your thoughts on this.

[JohnRDOrazio]

My thoughts are, I can see what you're trying to get at, but I still think it would be simpler to adopt in a client application because there are so many different ways a filter could be applied, how could the API have a filter that would make everyone happy? What if one person would like to see all Feasts and Solemnities, another would like to see Feasts of the Lord and Solemnities, another would like to see only Holy Days of Obligation... Among the Feasts of the Lord, Sundays are also counted: what if someone wanted to see Feasts of the Lord but not Sundays of Ordinary Time? I can see your point, but I can also see how arbitrary the filters could be. So I can't possibly see how to apply a filter reliably in the API. If you have any suggestions on how to go about doing this in a way that everyone would be happy, I might consider it. But there needs to be some objective criteria to the filter, and not just an arbitrary "I'm seeing too many entries in my calendar for my liking".

[HPaulson]

You're right - there's a LOT of filter options that could be useful. For this reason, I think it makes sense for filtering like this to be very arbitrary. I made a demo in deno/typescript with the type of system I think would be best here. It is modeled off a similar system in Directus. I ended up making it fairly complete, so my use case is satisfied, and I'm happy to close this if you don't think it's worth implementing in the actual api.

The only example we mentioned above that I could not implement using the generic filter options is only including Holy Days of Obligation. This is because I couldn't find any data field that relates to this, which makes sense because it'll be different in each diocese. Perhaps this could be added, similar to is_vigil_mass, on the diocese calendars that do exist?

. . .

I realize this system makes filtering fairly technical, because it requires an understanding of the json data and each type. Perhaps in the examples above there are even some mistakes. This would make it hard to add "human readable" names to these filters, but at least it gives a developer the chance to filter for their needs (and attempt the human readable if desired).

[JohnRDOrazio]

Great work! I haven't in fact implemented a Holy Day of Obligation property yet, I'll try to fix that. How could we name such a property? In Latin it's "dies festus de praecepto", in Italian it's "festa di precetto", in English it's "Holy Day of Obligation", as stated in the Code of Canon Law in canon 1246:

• Latin https://www.vatican.va/archive/cod-iuris-canonici/latin/documents/cic_liberIV_la.html#:~:text=Can.%201246%20%E2%80%94%20%C2%A7%201,omnium%20denique%20Sanctorum.
• Italian https://www.vatican.va/archive/cod-iuris-canonici/ita/documents/cic_libroIV_1246-1248_it.html#%C2%A72:~:text=Can.%201246%20%2D%20%C2%A71,tutti%20i%20Santi.
• English https://www.vatican.va/archive/cod-iuris-canonici/eng/documents/cic_lib4-cann1244-1253_en.html#TITLE_II.:~:text=Can.%C2%A01246%20%C2%A71,and%20All%20Saints.

Seeing the other properties are defined in English, perhaps for consistency we should stick to English and name it: is_holy_day_of_obligation with a boolean value of true or false. And rather than attach the property to every single event, perhaps it should only be attached to events that have a rank of "Feast" and up. "Feast of the Lord" (one rank higher than "Feast") would almost be enough, except that I'm seeing on https://en.wikipedia.org/wiki/Holy_day_of_obligation that in the Dominican Republic, the Feast of Our Lady of Altagracia and the Feast of Our Lady of Mercy are also considered Holy Days of Obligation (though they're probably Solemnities anyways?) and in Germany Saint Stephen's Day is also considered a Holy Day of Obligation (not sure if it's also considered a Solemnity in Germany?). Just to be on the safe side, I'll attach the property to events ranking Feast and up.

I suppose it could be useful to be able to add "middleware" to the API, that would allow to add extra functionality such as the filter you have implemented. But I could probably use some help in working out a good architecture for allowing different kinds of middleware to be implemented... I suppose there could be a middleware folder, where any scripts found in the folder would be run automatically by the API. But then we would have to have an understanding of what kinds of middleware scripts would be placed there, and how and when they should be run...

[JohnRDOrazio]

Or perhaps any middleware would be implemented in client libraries for the API, rather than in the API itself...

[HPaulson]

This seems reasonable. Perhaps middleware/extentions could be supported in a way that is agnostic to language. Here's one idea, taking some inspiration from package.json scripts:

• Have an extensions folder, which has a sub-folder for each middleware/extention.
• Each subfolder must have a extension.json top-level file. This file will describe the behavior of the extention. Example:

{
"name": "MyExtention1",
"version": "0.0.1",
"LitCalVersion": "^4.3",
"author": "MyName",
"github": "https://github.com/project/name",
. . .
"build": "deno task build",
"start": "deno run main.ts"
}

Using a setup like this one, the build command would be run on API-start. On each api request, a temp file would be created after the api generates the relevant response for that request, so that each extention can successively take in and modify the file's data. Order would be determined by some config file perhaps, or alphabetically? This file could contain the original args sent to the api, and the current json/ical/xml... data being processed (See example below). On each request, after this file is created, the start command of each middleware is called, with a flag designating the filepath of the temp file. After the api runs each middleware start command, it sends the new data in the file as the response. Each middleware, therefore, is responsible for taking in the correct file from the command argument, editing as necesary, and exiting the process when finished.

The temp file could also be a json file, somehting like this (but with all data available):

{
"queryParams": "?locale=en_us&year=2024...",
"endpoint": "/calendar/nation/US",
. . .
"type": "<json | yml | xml | ics>",
"data": "<json,yml,xml,ics data string>"
}

This setup would make it so self-hosted instances of the API can add n number of extentions they see fit, and it is very customizable. An extention can ignore requests based on the data in the file, can be built in any language, and can do pretty much anything it wants with the data. Of course these extentions would not be supported by the main insance of the api (for security reasons, and because not everyone would want to use them), but that's entiely ok I think.

Note that the self hosted instance would need to self-install any required software beforehand, like deno in the above example. Perhaps this could be standardized further by using make files that are called.

Unsure if this is the best solution for extentions, but the first one that came to mind. Curious your thoughts.

--

Regarding holy day of obligation, your idea LGTM. That should probaly become a seperate issue, though.

[JohnRDOrazio]

Holy Days of Obligation are now available in v5.2 (currently reachable at https://litcal.johnromanodorazio.com/api/v5/ !

[JohnRDOrazio]

Current Directus documentation for filter queries: https://directus.io/docs/guides/connect/filter-rules

[JohnRDOrazio]

This example filter (taken from the Directus docs):

{
  "_and": [
    {
      "field": {
        "operator": "value"
      }
    },
    {
      "field": {
        "operator": "value"
      }
    }
  ]
}

should be able to be expressed as a URL query parameter something like this:

?filter[_and][0][field][operator]=value&filter[_and][1][field][operator]=value

[JohnRDOrazio]

The simpler form, without logical operators, can be translated like this:

{
  "title": {
    "_contains": "Directus"
  }
}

URL query parameter:

?filter[title][_contains]=Directus

[JohnRDOrazio]

Nested logical operators can quickly become quite complex:

{
  "_or": [
    {
      "_and": [
        { "status": { "_eq": "published" } },
        { "category": { "_eq": "news" } }
      ]
    },
    {
      "author": { "_eq": "john" }
    }
  ]
}

should be able to be represented as a URL query parameter like this:

?filter[_or][0][_and][0][status][_eq]=published
&filter[_or][0][_and][1][category][_eq]=news
&filter[_or][1][author][_eq]=john

[JohnRDOrazio]

A few URL parse tests on https://php-play.dev:

Simple example

<?php
$query = parse_url('http://www.example.com?filter[title][_contains]=Directus', PHP_URL_QUERY);
parse_str($query, $params);
echo '<pre>';
print_r($params);
echo '</pre>';

Result:

Array
(
    [filter] => Array
        (
            [title] => Array
                (
                    [_contains] => Directus
                )

        )

)

Complex example

$query = parse_url('http://www.example.com?filter[_or][0][_and][0][status][_eq]=published&filter[_or][0][_and][1][category][_eq]=news&filter[_or][1][author][_eq]=john', PHP_URL_QUERY);
parse_str($query, $params);
echo '<pre>';
print_r($params);
echo '</pre>';

Result:

Array
(
    [filter] => Array
        (
            [_or] => Array
                (
                    [0] => Array
                        (
                            [_and] => Array
                                (
                                    [0] => Array
                                        (
                                            [status] => Array
                                                (
                                                    [_eq] => published
                                                )

                                        )

                                    [1] => Array
                                        (
                                            [category] => Array
                                                (
                                                    [_eq] => news
                                                )

                                        )

                                )

                        )

                    [1] => Array
                        (
                            [author] => Array
                                (
                                    [_eq] => john
                                )

                        )

                )

        )

)

[JohnRDOrazio]

PHPStan will want to know the "shape" of the array, so let's think it through:

Filter

The filter parameter will contain a value that is an associative array, where the key can be either a field name or a logical operator

Field name

A field name will have a value that is an associative array with a single property, where the key is a comparison operator and the value is the value to compare against

Logical operator

A logical operator is always a collection, the items of which are associative arrays where the key can again be either a field name or a logical operator

Valid logical operators are:

• _and
• _or

Comparison operator

We will only use a subset of comparison operators compared to those available in Directus, eliminating those that only pertain to geometry or to relational mapping:

Operator	Description
_eq	Equals
_neq	Doesn’t equal
_lt	Less than
_lte	Less than or equal to
_gt	Greater than
_gte	Greater than or equal to
_in	Is one of
_nin	Is not one of
_null	Is null
_nnull	Isn’t null
_contains	Contains
_ncontains	Doesn’t contain
_icontains	Contains (case-insensitive)
_nicontains	Doesn’t contain (case-insensitive)
_starts_with	Starts with
_istarts_with	Starts with (case-insensitive)
_nstarts_with	Doesn’t start with
_nistarts_with	Doesn’t start with (case-insensitive)
_ends_with	Ends with
_iends_with	Ends with (case-insensitive)
_nends_with	Doesn’t end with
_niends_with	Doesn’t end with (case-insensitive)
_between	Is between two values (inclusive)
_nbetween	Is not between two values (inclusive)
_regex	Regular expression (escape backslashes) [in validations]

I think we can also eliminate the _empty and _nempty operators, since most comparisons will be against string, integer, boolean and null values.

All operators take a single value to compare against, except for _between and _nbetween which take an array of two values to compare against (a lower bound and an upper bound). N.B.

• The two array values must be in the correct order: [lowerBound, upperBound].
• _between is inclusive: values equal to the boundaries are included.

Example JSON using _between or _nbetween:

{
  "created_at": {
    "_between": ["2024-01-01", "2024-12-31"]
  }
}

Equivalent URL query parameter:

?filter[created_at][_between]=2024-01-01,2024-12-31

The _eq and _neq comparisons are not strictly typed for numeric values in Directus; perhaps we can do a strict comparison based on the expected type of the field, by explicitly casting the value to the expected type.

[JohnRDOrazio]

Perhaps it would also be useful to map which comparison operators can be used with which fields based on the field type, and the types that make sense for the comparison operator.

[JohnRDOrazio]

Here is the table again with the possible types for each comparison operator:

Operator	Description	Possible Types
_eq	Equals	string, integer, float, boolean, null
_neq	Doesn’t equal	string, integer, float, boolean, null
_lt	Less than	integer, float, string (lexical), date/time
_lte	Less than or equal to	integer, float, string (lexical), date/time
_gt	Greater than	integer, float, string (lexical), date/time
_gte	Greater than or equal to	integer, float, string (lexical), date/time
_in	Is one of	array of string, integer, float, boolean, null
_nin	Is not one of	array of string, integer, float, boolean, null
_null	Is null	null
_nnull	Isn’t null	any
_contains	Contains	string
_ncontains	Doesn’t contain	string
_icontains	Contains (case-insensitive)	string
_nicontains	Doesn’t contain (case-insensitive)	string
_starts_with	Starts with	string
_istarts_with	Starts with (case-insensitive)	string
_nstarts_with	Doesn’t start with	string
_nistarts_with	Doesn’t start with (case-insensitive)	string
_ends_with	Ends with	string
_iends_with	Ends with (case-insensitive)	string
_nends_with	Doesn’t end with	string
_niends_with	Doesn’t end with (case-insensitive)	string
_between	Is between two values (inclusive)	integer, float, string (lexical), date/time
_nbetween	Is not between two values (inclusive)	integer, float, string (lexical), date/time
_regex	Regular expression	string

[JohnRDOrazio]

Now let's list only fields that might make sense for filtering on the server, with their relative types, and the possible comparison operators:

Field	Type	Comparison operators
event_key	string	_eq, _neq, _contains, _ncontains, _starts_with, _nstarts_with, _ends_with, _nends_with, _in, _nin, _regex
grade	integer	_eq, _neq, _lt, _lte, _gt, _gte, _between, _nbetween
is_vigil_mass	boolean	_eq, _neq
holy_day_of_obligation	boolean	_eq, _neq

If there are any requests for filtering on other fields, we can always look into allowing further fields for filtering, but I believe these will be the most useful for now.

[JohnRDOrazio]

Now for the approach to applying the filters on a PHP collection, for example using array_filter: we want to do some recursive parsing to produce closures that return a boolean for each object.

For example:

function buildPredicate(array $filter): callable {
    // handle logical operators
    if (isset($filter['_and'])) {
        $predicates = array_map('buildPredicate', $filter['_and']);
        return fn($item) => array_reduce($predicates, fn($carry, $p) => $carry && $p($item), true);
    }
    if (isset($filter['_or'])) {
        $predicates = array_map('buildPredicate', $filter['_or']);
        return fn($item) => array_reduce($predicates, fn($carry, $p) => $carry || $p($item), false);
    }

    // implicit AND when multiple fields
    $predicates = [];
    foreach ($filter as $field => $conditions) {
        foreach ($conditions as $op => $value) {
            $predicates[] = makeComparison($field, $op, $value);
        }
    }

    return fn($item) => array_reduce($predicates, fn($carry, $p) => $carry && $p($item), true);
}

and a simple comparison builder:

function makeComparison(string $field, string $op, mixed $value): callable {
    return match ($op) {
        '_eq'  => fn($item) => $item->$field == $value,
        '_neq' => fn($item) => $item->$field != $value,
        '_lt'  => fn($item) => $item->$field <  $value,
        '_lte' => fn($item) => $item->$field <= $value,
        '_gt'  => fn($item) => $item->$field >  $value,
        '_gte' => fn($item) => $item->$field >= $value,
        '_in'  => fn($item) => in_array($item->$field, (array)$value, true),
        '_nin' => fn($item) => !in_array($item->$field, (array)$value, true),
        '_null' => fn($item) => $item->$field === null,
        '_nnull' => fn($item) => $item->$field !== null,
        '_between' => fn($item) => $item->$field >= $value[0] && $item->$field <= $value[1],
        '_nbetween' => fn($item) => $item->$field < $value[0] || $item->$field > $value[1],
        '_contains' => fn($item) => str_contains($item->$field ?? '', $value),
        '_ncontains' => fn($item) => !str_contains($item->$field ?? '', $value),
        default => throw new InvalidArgumentException("Unsupported operator: $op"),
    };
}

Then, given a filter like:

$filter = [
    "_or" => [
        [
            "status" => [ "_eq" => "published" ],
            "category" => [ "_eq" => "news" ]
        ],
        [ "author" => [ "_eq" => "john" ] ]
    ]
];

You can run:

$predicate = buildPredicate($filter);
$filtered = array_filter($collection, $predicate);

Filter shape	Example	Logical meaning
Top-level _and	{ "_and": [ { "field1": { "_eq": "A" } }, { "field2": { "_eq": "B" } } ] }	Must satisfy all conditions (AND)
Top-level _or	{ "_or": [ { "field1": { "_eq": "A" } }, { "field2": { "_eq": "B" } } ] }	Must satisfy at least one condition (OR)
Top-level field(s)	{ "field1": { "_eq": "A" }, "field2": { "_eq": "B" } }	Must satisfy all conditions (AND)
Nested logical group	{ "_or": [ { "_and": [ ... ] }, { "_and": [ ... ] } ] }	Combine groups of conditions using mixed logic (AND/OR)
Single field, single operator	{ "field1": { "_eq": "A" } }	Simple direct comparison
Single field, multiple operators	{ "field1": { "_gte": 10, "_lte": 20 } }	All comparisons on that field must be true (AND)

When multiple fields appear at the same level without a top-level _and or _or, they are implicitly combined with AND logic.

[JohnRDOrazio]

@HPaulson do these notes seem to make sense? Any observations?

[HPaulson]

@JohnRDOrazio Sorry for the delay on this!

should be able to be represented as a URL query parameter like this:

Directus allows two syntaxes for this, the "JSON string" syntax and "square brackets" syntax. Checkout this docs link to see some examples of both.

In the sdk they use this function to facilitate the JSON string syntax. Basically they're just taking the filter as a js object, stringifying it, and passing the resulting string as a query parameter (this makes it very easy to parse on the server with a simple JSON.parse).

But the "square brackets" syntax is the one I prefer, and the one you mention in the examples above. For simplicity and clarity, I recommend only accepting this syntax, since the JSON string syntax is not as readable and I'm not sure if the server-side parse simplicity will carry over to php.

Note, however, that if dealing with JSON is fairly simple in php, this syntax may actually be preferable. You had mentioned wanting to avoid codebase complexity, and having a simple JSON parse of the query param would be simpler than having multiple functions to parse the square brackets. This will diminish human-readability of the filters in my opinion, but I imagine most people will use a UI to generate these query params anyway, so this is maybe a worthwhile concession.

I have an implementation of the "square brackets" syntax (at least the basic filters, not logical groups) in my tool if it's helpful. I use this function to parse query params from "square brackets" syntax into a js object. Then I normalize them to avoid implementing logical groups (I force multiple _eq operators on a single field to be an OR group, so a filter like &filter[grade][_eq]=0&filter[grade][_eq]=3 would mean "A grade of 0 OR 3", whereas in Directus' system this would be an implicit AND group as your tables describe). Finally I apply the filters on the json data from the API.

Your tables and understanding of the system look mostly correct. I found one issue:

All operators take a single value to compare against, except for _between and _nbetween which take an array of two values to compare against (a lower bound and an upper bound).

This is incorrect. Both _in and _nin also take arrays:

"status": {
            "_in": ["published", "draft"]
          }

This will return if status is either published or draft, acting as a kind-of OR group. You later write that the possible type for these is "array of string, integer, float, boolean, null", so I assume you caught this already, but wanted to be sure.

If there are any requests for filtering on other fields, we can always look into allowing further fields for filtering, but I believe these will be the most useful for now.

I disagree strongly on this point. If you're going to implement a generic filtering scheme like this one, I don't think there's a reason to limit the fields that can be filtered, as additional fields should not mean additional implementation. It is perhaps reasonable to exclude some _operators if they are annoying to implement and not likely to get used often, but once an operator is implemented, it should be generic and usable on any field. Your above makeComparison is like this: it does not care about the field it is evaluating, it is just comparing the field to a passed value in accordance with the operator. I think the function should stay this way, and in general I think it's a bad idea to reduce possible use cases to those that are expected unless it's much extra work to maintain the fringe cases, which it shouldn't be here.

Hope this is helpful, let me know if you're looking for anything else.