scitags/packet marking support

[marian-babik]

Motivation

Following our discussion with @bari12 at the last ATLAS S&C week, we would like to request adding support for scitags (scitags.org) to Rucio. This is needed in order to be able to pass the experiments and activity identifiers to the underlying transfer systems (FTS) and possibly storages at a later stage, so that they can be used by the Research and Education Networks and Experiments network monitoring systems. More details on the motivation can be found in our web page at scitags.org and in our technical specification at https://docs.google.com/document/d/1x9JsZ7iTj44Ta06IHdkwpv5Q2u4U2QGLWnUeN2Zf5ts/edit?usp=sharing

Modification

Add representation of the experiment and activity fields and their corresponding ids according to the mapping available at http://api.scitags.org/ (json schema is available at https://www.scitags.org/schemas/v1.0.0/registry.schema.json). Add capability to pass both identifiers to the FTS via regular channel currently used for this purpose.

[bari12]

Hi @marian-babik Thanks for following up on this. I think this should be relatively easy on our side. The main difficulty I see is that each Rucio instance does not really know which experiment it serves, so we would have to add something to the configuration:

[packet-marking]
expname=X

We would probably only fetch the experiment dictionary from the API once in a while and cache it locally (Any input what lifetime would be acceptable here?) and then just add expId and activityId to the parameters when submitting to FTS.

[marian-babik]

I think once a week would be fine, we don't expect this to change very often, it is mostly static information. Optionally, there is a field called "modified" in the feed, which has a timestamp when the feed was last updated (experiments/activities were added/changed/removed), so you can easily check if update is needed.

[marian-babik]

Following a discussion with FTS (@mpatrascoiu), I understand they would prefer to receive the two parameters using the following format:

• Using scitag attribute with value expID <<6 | activityID where <<6 | are bitwise operators and expID and actID are the numeric codes that map experiments and activities to numbers as found in the SciTags registry API (http://api.scitags.org/)

The main motivation for this is the updated HTTP/TPC specification related to SciTag, which uses the same format, so no additional processing would be required by FTS.

[bari12]

@marian-babik @mpatrascoiu This is done from Rucio side, but we are waiting for the last information how to transmit the information to fts (as part of the job meta?)

[mpatrascoiu]

Hello,

On submitting to FTS, the scitag field will be a per-transfer field.

Example on how to submit:

1. From the command line

$ fts-rest-transfer-submit --scitag <scitag> -s <fts-endpoint> <src> <dst>

2. From the Python module

import fts3.rest.client.easy as fts3

fts3.new_transfer(.., scitag=<scitag>, ..)

3. Via the REST API (JSON format)

{
  "files": [
    {
      "sources": [ "<src>" ],
      "destinations": [ "<dst>" ],
      "scitag": 65  # must be integer type
    }
  ],
  "params": {
    "key": "value"
  }
}

[rcarpa]

Thank you, Mihai. I already prepared the final change to set the correct file attribute. https://github.com/rucio/rucio/compare/master...rcarpa:rucio:patch-0-fix_fts_scitags_argument?expand=1

Is there already a released version of FTS which accepts this flag? For now I cannot merge the change into rucio because the test fails: our current test FTS instance doesn't yet support scitag. As a result, it doesn't return this field on the /jobs/transfer_id endpoint. Will wait for a version of FTS which returns it on the jobs endpoint.

[mpatrascoiu]

Hello Radu,

Yes, this was released in FTS v3.12.11 + FTS-REST Server v3.12.4. Currently, these packages are in the FTS RC repository and installed on the FTS3-Pilot instance (if you need an instance to submit against).

If you're interested in having this in the Rucio CI/CD pipeline, you'd have to get the packages from the FTS RC repository (link). Don't forget you also need schema v8.2.0 (fts-diff-8.2.0.sql)