PROPFIND response excludes files below certain recursion level

[paulmillar]

Here is a simple subtree:

paul@celebrimbor:~/vtmp$ tree
.
├── dir-1
│   ├── dir-1-1
│   │   ├── dir-1-1-1
│   │   │   └── file-1-1-1
│   │   ├── dir-1-1-2
│   │   │   └── file-1-1-2
│   │   └── file-1-1
│   ├── dir-1-2
│   │   └── file-1-2
│   └── file-1
└── dir-2
    └── file-2

7 directories, 6 files
paul@celebrimbor:~/vtmp$ 

I copy this directory structure into dCache:

paul@celebrimbor:~/vtmp$ rclone copy . prometheus:/Users/paul/recursive-ls-test/
paul@celebrimbor:~/vtmp$ 

I then use curl to make a PROPFIND request, selecting only the displayname XML elements in the response that contain the text file:

paul@celebrimbor:~/vtmp$ curl -s -X PROPFIND \
        https://prometheus.desy.de:2443/Users/paul/recursive-ls-test \
        | xmllint -format -\
        | grep displayname.*file
        <d:displayname>file-1-1</d:displayname>
        <d:displayname>file-1-2</d:displayname>
        <d:displayname>file-1</d:displayname>
        <d:displayname>file-2</d:displayname>
paul@celebrimbor:~/vtmp$ 

Note that files file-1-1-1 and file-1-1-2 are missing in the response.

If I repeat the request, but targeting one directory deeper (i.e., the dir-1 directory) then file-1-1-1 and file-1-1-2 are included in the response:

paul@celebrimbor:~/vtmp$ curl -s -X PROPFIND \
        https://prometheus.desy.de:2443/Users/paul/recursive-ls-test/dir-1 \
        | xmllint -format - \
        |grep displayname.*file
        <d:displayname>file-1-1-1</d:displayname>
        <d:displayname>file-1-1-2</d:displayname>
        <d:displayname>file-1-1</d:displayname>
        <d:displayname>file-1-2</d:displayname>
        <d:displayname>file-1</d:displayname>
paul@celebrimbor:~/vtmp$ 

This behaviour seems to be at odds with the Depth header's description in RFC 4918.

This issue was first reported by Hubert Simma (thanks!). He also observed that StoRM WebDAV seems to exhibit the same behaviour. dCache and StoRM WebDAV both use the Milton library, which suggests this behaviour comes from Milton.