simpler time formatting at the "Collector HTTP"

[daniwb]

Hello Devs

I'm struggeling to define an HTTP Site with some Formatting.

The Comment on the Code itself of collector_http is incorrect: Delta is a datetime.timedelta JSON string, ex: '{days=-1}'.

Should be {"days"=-1}

Also I've found that it could be much easier to format the URL on the following lines:

    def format_url(self, url: str, formatting) -> str:
        try:
            return self.parameters.http_url.format(time=Time(formatting))

You could solve it directly not using this Class Time

        if formatting:
            http_url = self.format_url(self.parameters.http_url, formatting)

by using:

        if formatting:
            http_url = self.parameters.http_url.format(datetime.now()+timedelta(**formatting))

(Source: https://github.com/certtools/intelmq/blob/08e86dfa16b67a2729101ee87052fd7d7c22a0c9/intelmq/bots/collectors/http/collector_http.py)

The URL then would be: "http://somesite.com/time.min={:%Y-%m-%d+%H:%M:%S}.000000Z"

And returning: "http://somesite.com/time.min=time.min=2020-11-15+18:00:53.000000Z"

Or can you point me to the right place where to find the correct configuration for http_url and http_url_formatting? I'm unable to figure out how it should be (My 2 hours of testing lead me to the upper solution)

[ghost]

cc contributor of the code @e3rd

[ghost]

Thanks for your input!

In order to support the boolean values for this parameter, an additional if-clause is needed:

if formatting is True:
    formatting = {}

Also your suggested code does not support the formatting (that's what the getitem method is for in the Time class`). In most cases, URLs need some specific time format, not the ISO format.

While I like the idea, it's does not cover all the use cases we currently support unfortunately.

The Comment on the Code itself of collector_http is incorrect: Delta is a datetime.timedelta JSON string, ex: '{days=-1}'.

Should be {"days"=-1}

Thanks, fixed in 88967537c

[daniwb]

I understood now, how it is working, the tests has an example: https://github.com/certtools/intelmq/blob/develop/intelmq/tests/bots/collectors/http/test_collector.py

http://localhost/{time[%Y]}.txt

so in my case: "http://somesite.com/time.min={time[%Y-%m-%d+%H:%M:%S]}.000000Z"

And yes my suggested solution can provide any Time format, as provided with the URL, the formating is in curly brackets: "http://somesite.com/time.min={:%Y-%m-%d+%H:%M:%S}.000000Z"

But anyhow it solved my problem, thanks a lot for your quick Help!

[ghost]

I understood now, how it is working, the tests has an example: https://github.com/certtools/intelmq/blob/develop/intelmq/tests/bots/collectors/http/test_collector.py

http://localhost/{time[%Y]}.txt

That's also the example mentioned in the docs

And yes my suggested solution can provide any Time format, as provided with the URL, the formating is in curly brackets: "http://somesite.com/time.min={:%Y-%m-%d+%H:%M:%S}.000000Z"

Oh nice, that's nifty! If we change that we'd either need to ensure backwards compatibility and/or automatic conversion.