Custom Filter

[jregnier]

Hello, I have a unique issue with dates and time zones. I have a UTC date in the index and the app allows you to apply any time zone so when I go to a search in the index I apply the time zone offset to match UTC but the issue is related to the daylight savings time since it depends on the date in the index.

It seems like I need to somehow take the date in the index and apply the timezone then do a search. I was thinking that maybe I could accomplish this with a custom filter by getting creating some kind of temporary "virtual" field that has the calculated DateTime with the correct timezone.

Basically, I am looking for a way to query on some field that has manipulated value.

Thanks in advanced

[rclabo]

The switch to daylight saving time does not affect UTC. So the UTC date /time you write into the index when under daylight savings time and when not under daylight savings time will be the same for a given time zone.

So your only issue is that when you perform a search you need to first convert the current date/time for that time zone into it's proper UTC date/time. And that process will inherently take the daylight savings time (or not) into consideration.

[jregnier]

Hello Ron, thanks for the quick reply, you are absolutely right! Except I was in a hurry when writing this and forgot the most important part 🤦 . It's not a DateTime but just a time range so I don't have a month to go off. The more I look into this the more I feel this can't be done since reading the value for each doc and converting it would be too expensive.

[rclabo]

(scratching head). I see. I think it's a logic problem more than a Lucene.NET problem and I think the logic problem would exist with any data store not just Lucene. It's hard to even reason what it means to deal with times without dates in the context of daylight savings time.

Daylight savings time is fundamentally associated with only certain months of the year. But if the time exists without a date then it exists without a month association which means that it's almost nonsensical to incorporate the concept of daylight savings time into the search. (shrug)

[NightOwl888]

Time zone support has been added in #551 and #580 which has been included since 4.8.0-beta00016.

It isn't clear from your question whether you are using DateTools to make the conversions, but there are built-in options for storing dates in the index:

1. As a number representing milliseconds since the Unix Epoch (Milliseconds since Jan 1, 1970 12:00:00 AM UTC.).
2. As a number representing the number or .NET Ticks.
3. As a string in the format yyyyMMddHHmmssSSS.

DateToolsalso allows you to reduce the resolution of the date in certain cases, for example, when the seconds or time component is not important for the search.

As for the time zone, it only really matters if you store the date as a string because numeric dates do not include time zone info, they are always stored as UTC. When you search, your query needs to be built with UTC dates. This can be done using the classic or flexible query parser.

The most common way of dealing with time zones in .NET is to create DateTime instances using DateTimeKind.Local or DateTimeKind.Utc, both of which can be converted to the other. However, if you don't specify what the date is, it will be created with DateTimeKind.Unspecified and will assumed to be local time whether or not you intended that.

https://docs.microsoft.com/en-us/dotnet/standard/datetime/converting-between-time-zones

Using the built-in logic assumes that DateTimeKind.Local is the time zone you are converting to, and it should work for most scenarios (I suspect web apps adjust it automatically based on the user's time zone, although I haven't tested that scenario). However, both DateTools and flexible query parser support setting the current time zone explicitly for scenarios where your computer's local time zone is set to one time zone and you want to search using a different time zone. Of course, using DateTools, the time zone info (and DST by extension) doesn't matter unless you are storing the dates in the index as a string.

If you store the dates in the index as numbers, the only place time zone matters is when you are building the query to search the index, in which case you will need to convert the date from local time zone to UTC before performing the search. And when you do that, .NET will take care of the DST part automatically as part of the time zone conversion from local time.

[jregnier]

Hey guys thanks for the replies. @rclabo you are absolutely right that this isn't a Lucene problem but I'm trying to solve it with Lucene. Right now we use a SQLite DB and we have a custom SQL function that applies the given timezone and then does the query against that. The reason for this is that in the DB there is a full date and time but the query only has a time for example give me everything between 9-5. It's also important to note that in the application you have the ability to specify any timezone so it's not tied to a machine or anything like that. The timezone is easy because it's the same for the entire year but DST is not, so it depends on the month in the Lucene index if DST has an offset

Thanks for the help, I know this is not a normal situation lol

[NightOwl888]

Closing, as this appears to be resolved.