DIRAC icon indicating copy to clipboard operation
DIRAC copied to clipboard

More configuration flexibility in InputDataPolicy

Open panta-123 opened this issue 2 years ago • 1 comments

Some additional flexibility will be good to add.

  1. Choose InputDataResolution policy according to jobType specified in CS. -> Lets say i want only User job to use InputDataByProtocol and other job to use DownloadInputData

  2. If there is no locol protcol that we want to use for streaming fallback to download policy. -> Curently for streaming p[rotocols we can specify Local and Remote protocols as:

     # List of protocols to use for streaming
    Protocols
    {
      # This list is used if the we are getting a file from a
      # StorageElement local to the site we are running on
      Local = file, xroot, root
      # This list is used if the SE is not local
      Remote = xroot, root
    }
    

    Lets say if the Remote part is empty and no local protocol available, then use the default policy.(i.e. DownloadInputData)

panta-123 avatar Feb 16 '23 18:02 panta-123

These are nice ideas, and we welcome PR ! The input data resolution being based on plugins, it should not be too difficult for you to extend. You should find a good starting point here https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/WorkloadManagement/InputDataResolution.html?highlight=inputdatapolicy Also, what you offer seems generic enough, and would probably benefit to other communities, so please consider implementing this in vanilla DIRAC. Let us know if you need more guidance.

chaen avatar Feb 16 '23 22:02 chaen