Copy without TPC through Webdav

[QLeB]

Hi,

I don't understand how to perform copy from a dCache server to the same server through webdav, without doing a third-party-copy. The documentation https://www.dcache.org/manuals/UserGuide-8.2/webdav.shtml only mentions third-party requests for COPY.

Basically I tried to make a COPY operation with curl, specifying the Destination URI in the header as documented. I am getting the following output:

< HTTP/1.1 202 Accepted
< Date: Tue, 10 Jan 2023 14:39:52 GMT
< Server: dCache/8.2.1
< Content-Type: text/perf-marker-stream
< Transfer-Encoding: chunked
< 
failure: rejected PUT: 401 Unauthorized

Any help would be appreciated. Thanks.

[paulmillar]

Yes, it looks like you are triggering an HTTP-TPC request, but without any kind of delegation. Without delegation, the dCache pool (containing the file) cannot authenticate, so the PUT request is made anonymously. It's therefore quite reasonable that the WebDAV door rejects this request.

This is, of course, silly.

What should happen is the WebDAV door should realise that the URL in the Destination request header is targeting a WebDAV door in the same dCache instance. Knowing this, the door should not use HTTP-TPC. This is (probably) a bug.

In order to reproduce the problem, I might be helpful to include the full curl command you used, along with the HTTP request headers that curl sends to dCache. The complete output from running curl with the -v option should provide this information.

As an alternative/work-around, you should be able to specify a path (an absolute path) as the destination; for example, Destination: /path/to/target. The WebDAV door should realise this cannot be an HTTP-TPC request, so it should try an internal copy, which shouldn't require any delegation.

That all said, I don't remember if dCache supports internal COPY requests. It could be that duplicating a file via a COPY request simply doesn't work.

[QLeB]

Thank you very much for your detailed answer. Indeed if I specify only the path, it says "Method COPY is not implemented." I will come back to you tomorrow with more details on the curl command and output.

Quentin

[paulmillar]

OK. It looks like there's two bugs here:

1. A COPY with a local URL is triggering an HTTP-TPC, even though this makes no sense.
2. The COPY command isn't implemented.

The second issue (COPY not implemented) is a feature request, but one that shouldn't be too hard to implement. dCache already supports internal copies, it's just not something you can trigger via the WebDAV protocol. You can achieve internal copies using either the dcap and SRM protocols.

The WebDAV library that dCache uses (Milton) does support COPY. If so, then adding support for COPY should only require some bridging code.

[QLeB]

Here is the full curl output:

$ curl -L -v -X COPY -k --capath /etc/grid-security/certificates -H "Authorization: Bearer $TOKEN" -H "Destination: https://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/users/leboulch/memory2.prof" https://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/users/leboulch/memory.prof
* About to connect() to ccdavrubinint.in2p3.fr port 2880 (#0)
*   Trying 2001:660:5009:320:134:158:227:2...
* Connected to ccdavrubinint.in2p3.fr (2001:660:5009:320:134:158:227:2) port 2880 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* 	subject: CN=ccdcacli449.in2p3.fr,OU=CC-IN2P3,O=CNRS,C=FR,O=GRID-FR
* 	start date: Dec 02 09:36:11 2022 GMT
* 	expire date: Dec 02 09:36:11 2023 GMT
* 	common name: ccdcacli449.in2p3.fr
* 	issuer: CN=AC GRID-FR Services,OU=GRID-FR,O=MENESR,C=FR
> COPY /pnfs/in2p3.fr/lsst/users/leboulch/memory.prof HTTP/1.1
> User-Agent: curl/7.29.0
> Host: ccdavrubinint.in2p3.fr:2880
> Accept: */*
> Authorization: Bearer XXX
> Destination: https://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/users/leboulch/memory2.prof
> 
< HTTP/1.1 202 Accepted
< Date: Wed, 11 Jan 2023 09:30:10 GMT
< Server: dCache/8.2.1
< Content-Type: text/perf-marker-stream
< Transfer-Encoding: chunked
< 
failure: rejected PUT: 401 Unauthorized
* Connection #0 to host ccdavrubinint.in2p3.fr left intact

[vokac]

This works for me after adding following headers

• Credential: none
• TransferHeaderAuthorization: Bearer $TOKEN

[paulmillar]

Thanks for the information @vokac!

This is (perhaps) not a surprise. If the COPY request is authorised using a bearer token (Authorization:) then that same bearer token may be delegated to dCache (TransferHeaderAuthorization:). This, then, allows the pool to to authorise its PUT request.

Personally, I would classify this as a work-around.

It is certainly useful, given regular COPY requests currently don't work. However, the underlying problem remains. dCache-internal server-side copying shouldn't require the user to delegate a credential.

[QLeB]

If I understand correctly, setting the TransferHeaderAuthorization header is equivalent to requiring a third-party copy?

[paulmillar]

Nope, setting this does NOT trigger (or require) the server to make an HTTP-TPC transfer.

You can set this header on any request (GET / PUT / DELETE / ...) it just doesn't have any effect. Even for COPY, it has no effect if dCache were to do an internal copy (internal copies are currently not supported by dCache).

HTTP-TPC has a feature that any header that starts TransferHeader is copied, without this prefix, to the corresponding data-bearing HTTP requests. So the value of the TransferHeaderAuthorization header on the COPY request becomes the Authorization header on the PUT request.

[airnandez]

Thanks @paulmillar and @vokac for your inputs. I would like to add more information about this issue for completeness hoping this may be useful.

Since Rubin is not yet completely ready to use bearer tokens, I checked the behavior of dCache's implementation of webDAV COPY operation when the the client uses X509 proxy certificates for identifying itself. Here is the trace of a curl command that shows a similar though not identical response: the COPY operation does not work neither but with a different response than the one observed when using bearer tokens. The response in this case is:

HTTP/1.1 401 client failed to delegate a credential

as opposed to

HTTP/1.1 202 Accepted
...
failure: rejected PUT: 401 Unauthorized

This is the trace:

$ curl -i -L -v \
   --capath /etc/grid-security/certificates \
   --cacert /tmp/x509up_u361 \
   --cert /tmp/x509up_u361 \
   -X COPY \
   -H 'Destination: https://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/users/fabio/tests/dir-b6dhc6uu/file-XXXXX' \
   https://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/users/fabio/tests/dir-b6dhc6uu/file-6fadgqzi
.....
> COPY /pnfs/in2p3.fr/lsst/users/fabio/tests/dir-b6dhc6uu/file-6fadgqzi HTTP/1.1
> User-Agent: curl/7.29.0
> Host: ccdavrubinint.in2p3.fr:2880
> Accept: */*
> Destination: https://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/users/fabio/tests/dir-b6dhc6uu/file-XXXXX
> 
< HTTP/1.1 302 Found
HTTP/1.1 302 Found
< Date: Mon, 23 Jan 2023 14:52:53 GMT
Date: Mon, 23 Jan 2023 14:52:53 GMT
< Server: dCache/8.2.1
Server: dCache/8.2.1
< X-Delegate-To: https://ccdcamcli21.in2p3.fr:8445/srm/delegation https://ccdcamcli22.in2p3.fr:8445/srm/delegation https://ccdcamcli20.in2p3.fr:8445/srm/delegation
X-Delegate-To: https://ccdcamcli21.in2p3.fr:8445/srm/delegation https://ccdcamcli22.in2p3.fr:8445/srm/delegation https://ccdcamcli20.in2p3.fr:8445/srm/delegation
< Location: https://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/users/fabio/tests/dir-b6dhc6uu/file-6fadgqzi?asked-to-delegate=true
Location: https://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/users/fabio/tests/dir-b6dhc6uu/file-6fadgqzi?asked-to-delegate=true
< Content-Length: 0
Content-Length: 0

< 
* Connection #0 to host ccdavrubinint.in2p3.fr left intact
* Issue another request to this URL: 'https://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/users/fabio/tests/dir-b6dhc6uu/file-6fadgqzi?asked-to-delegate=true'
* Found bundle for host ccdavrubinint.in2p3.fr: 0x1e9a200
* Re-using existing connection! (#0) with host ccdavrubinint.in2p3.fr
* Connected to ccdavrubinint.in2p3.fr (134.158.227.2) port 2880 (#0)
> COPY /pnfs/in2p3.fr/lsst/users/fabio/tests/dir-b6dhc6uu/file-6fadgqzi?asked-to-delegate=true HTTP/1.1
> User-Agent: curl/7.29.0
> Host: ccdavrubinint.in2p3.fr:2880
> Accept: */*
> Destination: https://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/users/fabio/tests/dir-b6dhc6uu/file-XXXXX
> 
< HTTP/1.1 401 client failed to delegate a credential
HTTP/1.1 401 client failed to delegate a credential
< Date: Mon, 23 Jan 2023 14:52:53 GMT
Date: Mon, 23 Jan 2023 14:52:53 GMT
< Server: dCache/8.2.1
Server: dCache/8.2.1
< WWW-Authenticate: Basic realm=""
WWW-Authenticate: Basic realm=""
< Transfer-Encoding: chunked
Transfer-Encoding: chunked

< 
* Connection #0 to host ccdavrubinint.in2p3.fr left intact

I am trying to understand what would be the best mechanism to turn around this issue when using X509 proxy certificates. I am aware that dCache offers the possibility of retrieving a macaroon (from a X509 certificate), which we could use as a bearer token. However, since the code of Rubin must be able to talk to other non-dCache webDAV servers, we would need to introduce code to handle specifically the case of dCache.

An alternative would be for us to implement the copy operation in the client: that means, the client would download the file from dCache to a local temporary file and then upload it again to dCache to the desired destination.

Both paths mentioned above seem possible albeit not entirely satisfying for different reasons. I would be interested in recommendations by dCache experts about possible ways forward for us.

[paulmillar]

Hi @airnandez ,

The behaviour you see is a consequence of how HTTP-TPC is implemented in dCache.

If an HTTP-TPC request is made using X.509 to authenticate to dCache then dCache assumes that the data-bearing request will require an X.509 credential. Therefore, dCache must have a valid X.509 credential before accepting the HTTP-TPC request. If dCache does not have such an X.509 credential that (or none that seems appropriate) then dCache will prompt the client to delegate a fresh X.509 credential using the GridSite delegation protocol. The delegation endpoint is provided by the X-Delegate-To header, with the Location header providing the URL to which the client should (re-)send the COPY request once the delegation has been made.

It is possible to disable the check for a delegated credential (and therefore prevent this response) by including the Credential: none header in the COPY request. Note that this does not fix the problem, because the data-bearing request will fail due to the lack of authentication.

In summary, what you are seeing is the same problem (the lack of delegated credential for HTTP-TPC). The failure mode is different because dCache assuming that a client using X.509 to authenticate with dCache will likely want to use X.509 to authenticate the data-bearing request, and delegation works differently between OIDC/bearer-tokens and X.509.