csaf_aggregator --interim doing nothing

[sva-andreasb]

Hi,

i am trying to setup a csaf_aggregator but having trouble to get the interim mode working following the instructions found in https://github.com/gocsaf/csaf/blob/main/docs/csaf_aggregator.md. Full mode works fine, interim runs a few milliseconds seems to do nothing.

[certproxy@hostname CertGetter]$ HTTPS_PROXY='proxyurl:8080' ./csaf/csaf_aggregator -c ./config/aggregator.toml --interim
time=2025-03-04T07:44:17.900+01:00 level=INFO msg="Starting workers..." num=2
time=2025-03-04T07:44:17.900+01:00 level=INFO msg="Setting up worker" worker=2 "BSI WID CERT Bund"=https://wid.cert-bund.de/.well-known/csaf/provider-metadata.json
time=2025-03-04T07:44:17.901+01:00 level=INFO msg="Setting up worker" worker=2 OpenXchange=https://www.open-xchange.com/.well-known/csaf/provider-metadata.json

This is my config:

#insecure = true
#openpgp_private_key =
#openpgp_public_key =
#interim_years =
#passphrase =
#write_indices = false
#time_range =
verbose=true

# specification requires at least two providers (default),
# to override for testing, enable:
#allow_single_provider = true

[aggregator]
  # Set if this instance shall be a mirror (aka `aggregator`) or a `lister`.
  # This determines the default value for the entries in [[provider]].
  category = "aggregator"
  name = "CSAF Aggregator"
  contact_details = "some1 @ somewhere"
  issuing_authority = "This service is provided as it is. It is gratis for everybody."
  namespace = "https://testnamespace.example.org"

[[providers]]
  name = "BSI WID CERT Bund"
  domain = "https://wid.cert-bund.de/.well-known/csaf/provider-metadata.json"

[[providers]]
  name="OpenXchange"
  domain="https://www.open-xchange.com/.well-known/csaf/provider-metadata.json"

Version:

[certproxy@hostname CertGetter]$ ./csaf/csaf_aggregator --version
3.1.1

Is this a bug or am i doing something wrong?

Regards, Andreas

[koplas]

csaf_aggregator --interim only does something if a interims.csv file has some entries. This file only gets written when doing a full download. This happens when calling without --interim.

If there are no entries in the interims.csv or the file does not exist, the interim worker will not download any documents. This happens here: https://github.com/gocsaf/csaf/blob/0848143a0bbcd83cecf626be7d8379759121de53/cmd/csaf_aggregator/interim.go#L196-L205

The write happens here: https://github.com/gocsaf/csaf/blob/0848143a0bbcd83cecf626be7d8379759121de53/cmd/csaf_aggregator/indices.go#L380 The callchain is process() -> full() -> mirror -> mirrorInternal() -> writeIndices() -> writeInterims().

It needs to be checked if this behavior has a reason; if it does, it must be documented to avoid this confusion.

@sva-andreasb Can you check if your setup works, after executing the full mode once?

[sva-andreasb]

Hi @koplas,

i did a full mode before interim, as said this works fine and loads a bunch of files.

I tried --interim directly after it, it did nothing (like expected). I also tried --interim a few hours later and the next day, but it still did nothing.

I can't find an interims.csv anywhere,

[koplas]

Do the used mirrors contain even interim documents? My local dataset of the BSI Mirror does not contain a document with the interim status.

[koplas]

The idea of the --interim feature is to allow to download interim documents with a higher frequency. For example you can start a full download every 24 hours. On a full download it will keeps track which downloaded documents have the status interim. Now you can start the interim download every 1 hour; and it will download all documents that had the interim status, when doing the full download.

This allows to download documents that may have higher change rate, without doing a full download. If the CSAF provider has no interim documents, it will not download any documents. This is expected behavior.

Feel free to open the issue again, if you find a case where this feature does not work, despite the provider having interim documents. You can check the status of the document using jq .document.tracking.status.