ios icon indicating copy to clipboard operation
ios copied to clipboard

Published 20 hours ago • verified publisher icon nextcloud

Folders with a very large number of files are showing as empty in the iOS app

Open johnhindley96 opened this issue 2 years ago • 11 comments

Steps to reproduce

  1. Open NextCloud iOS app
  2. Choose 'Files' tab
  3. Navigate to an external storage folder which contains a lot of media files (in my case ~20,000 photo and video files, totalling ~130Gb)

Expected behaviour

Similar to the behaviour in the web app, after a short wait (~20 seconds), a list of the files is displayed.

Actual behaviour

After a longer wait (~60 seconds), the iOS app indicates that there are no files in the directory.

Logs

There are 2 issues at play, which combine to cause the main issue discussed here:

  1. There appears to be a 60s timeout on the WebDAV call being made from the iOS app
  2. The WebDav call to retrieve the contents of this folder from the iOS app is taking a much longer time to respond than the equivalent (but different) call from the web app (~300 seconds vs ~15 seconds)

The reason I suspect that the WebDAV call from within the iOS app has a 60s timeout is due to the iOS logs consistently reporting a timeout after 60s. When I monitor http traffic via a web proxy, there is no timeout shown and the request runs to completion (meaning that the timeout is client side rather than server side).

iOS Logs:

2021-09-09 02:08:41 Network request started: PROPFIND http://redacted/remote.php/webdav/redacted/info/
2021-09-09 02:09:41 Network response request: http://redacted/remote.php/webdav/redacted/info/, result: failure(Alamofire.AFError.sessionTaskFailed(error: Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={_kCFStreamErrorCodeKey=-2102, NSUnderlyingError=0x281220780 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <133556B0-57FB-41D5-9F46-8C3996F5022F>.<118>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalDataTask <133556B0-57FB-41D5-9F46-8C3996F5022F>.<118>"
), NSLocalizedDescription=The request timed out., NSErrorFailingURLStringKey=http://redacted/remote.php/webdav/redacted/info/, NSErrorFailingURLKey=http://redacted/remote.php/webdav/redacted/info/, _kCFStreamErrorDomainKey=4}))

As mentioned earlier, in order to investigate this further, I redirected both the iOS app and the Web app through the same http traffic monitoring proxy. This monitor consistently shows the WebDAV request from the iOS app taking ~300s, and the equivalent request from the Web app taking ~15s. The two requests are different, but they ultimately appear to provide very similar responses (see below).

iOS WebDAV request:

PROPFIND /remote.php/webdav/redacted/info/ HTTP/1.1
Host: redacted
OCS-APIRequest: true
Accept: */*
Authorization: Basic redacted
Accept-Encoding: br;q=1.0, gzip;q=0.9, deflate;q=0.8
Accept-Language: en;q=1.0, fr;q=0.9, de;q=0.8, ja;q=0.7, nl;q=0.6, it;q=0.5
Content-Type: application/xml
Content-Length: 1527
Depth: 1
User-Agent: Mozilla/5.0 (iOS) Nextcloud-iOS/4.0.6
Cookie: redacted
Connection: keep-alive


<?xml version="1.0" encoding="UTF-8"?>
<d:propfind xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns">
    <d:prop>
        <d:getlastmodified />
        <d:getetag />
        <d:getcontenttype />
        <d:resourcetype />
        <d:quota-available-bytes />
        <d:quota-used-bytes />

        <permissions xmlns="http://owncloud.org/ns"/>
        <id xmlns="http://owncloud.org/ns"/>
        <fileid xmlns="http://owncloud.org/ns"/>
        <size xmlns="http://owncloud.org/ns"/>
        <favorite xmlns="http://owncloud.org/ns"/>
        <share-types xmlns="http://owncloud.org/ns"/>
        <owner-id xmlns="http://owncloud.org/ns"/>
        <owner-display-name xmlns="http://owncloud.org/ns"/>
        <comments-unread xmlns="http://owncloud.org/ns"/>
        <checksums xmlns="http://owncloud.org/ns"/>
        <downloadURL xmlns="http://owncloud.org/ns"/>
        <data-fingerprint xmlns="http://owncloud.org/ns"/>

        <creation_time xmlns="http://nextcloud.org/ns"/>
        <upload_time xmlns="http://nextcloud.org/ns"/>
        <is-encrypted xmlns="http://nextcloud.org/ns"/>
        <has-preview xmlns="http://nextcloud.org/ns"/>
        <mount-type xmlns="http://nextcloud.org/ns"/>
        <rich-workspace xmlns="http://nextcloud.org/ns"/>
        <note xmlns="http://nextcloud.org/ns"/>

        <share-permissions xmlns="http://open-collaboration-services.org/ns"/>
        
        <share-permissions xmlns="http://open-cloud-mesh.org/ns"/>
    </d:prop>
</d:propfind>

iOS WebDAV response (duration ~300s):

HTTP/1.1 207 Multi-Status
Date: Wed, 08 Sep 2021 23:55:55 GMT
Server: Apache/2.4.41 (Ubuntu)
Referrer-Policy: no-referrer
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Robots-Tag: none
X-XSS-Protection: 1; mode=block
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Security-Policy: default-src 'none';
Vary: Brief,Prefer
DAV: 1, 3, extended-mkcol
Keep-Alive: timeout=5, max=99
Transfer-Encoding: chunked
Content-Type: application/xml; charset=utf-8
Proxy-Connection: keep-alive


<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns">
...
</d:response>
<d:response>
    <d:href>/remote.php/webdav/redacted/info/09D71C68-5CCA-469F-AAA1-C23F2AC7DC0F.JPG</d:href>
    <d:propstat>
        <d:prop>
            <d:getlastmodified>Sun, 09 Dec 2018 19:11:41 GMT</d:getlastmodified>
            <d:getetag>&quot;9f46f89482e0a3f145f0226011a52747&quot;</d:getetag>
            <d:getcontenttype>image/jpeg</d:getcontenttype><d:resourcetype/>
            <oc:permissions>RMGDNVW</oc:permissions>
            <oc:id>00016539ocum9e0hf7du</oc:id>
            <oc:fileid>16539</oc:fileid>
            <oc:size>1198042</oc:size>
            <oc:favorite>0</oc:favorite>
            <oc:share-types/>
            <oc:owner-id>redacted</oc:owner-id>
            <oc:owner-display-name>redacted</oc:owner-display-name>
            <oc:comments-unread>0</oc:comments-unread>
            <oc:downloadURL></oc:downloadURL>
            <oc:data-fingerprint></oc:data-fingerprint>
            <nc:creation_time>0</nc:creation_time>
            <nc:upload_time>0</nc:upload_time>
            <nc:has-preview>true</nc:has-preview>
            <nc:mount-type>external</nc:mount-type>
            <nc:note></nc:note>
            <x1:share-permissions xmlns:x1="http://open-collaboration-services.org/ns">19</x1:share-permissions>
            <x2:share-permissions xmlns:x2="http://open-cloud-mesh.org/ns">[&quot;share&quot;,&quot;read&quot;,&quot;write&quot;]</x2:share-permissions>
        </d:prop>
        <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
    <d:propstat>
        <d:prop>
            <d:quota-available-bytes/>
            <d:quota-used-bytes/>
            <oc:checksums/>
            <nc:is-encrypted/>
            <nc:rich-workspace/>
        </d:prop>
        <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>
</d:response>
<d:response>
...
</d:multistatus>

Web app WebDAV request:

PROPFIND /nextcloud/remote.php/dav/files/user/redacted/info HTTP/1.1
Host: redacted
Accept: */*
X-Requested-With: XMLHttpRequest
Accept-Encoding: gzip, deflate
Accept-Language: en-gb
Content-Type: application/xml; charset=utf-8
Origin: redacted
Depth: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15
Content-Length: 653
Cookie: redacted
requesttoken: redacted
Connection: keep-alive


<?xml version="1.0"?>
<d:propfind  xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns" xmlns:ocs="http://open-collaboration-services.org/ns">
  <d:prop>
    <d:getlastmodified />
    <d:getetag />
    <d:getcontenttype />
    <d:resourcetype />
    <oc:fileid />
    <oc:permissions />
    <oc:size />
    <d:getcontentlength />
    <d:quota-available-bytes />
    <nc:has-preview />
    <nc:mount-type />
    <nc:is-encrypted />
    <ocs:share-permissions />
    <oc:tags />
    <oc:favorite />
    <oc:comments-unread />
    <oc:owner-id />
    <oc:owner-display-name />
    <oc:share-types />
  </d:prop>
</d:propfind>

Web app WebDAV response (duration ~15s):

HTTP/1.1 207 Multi-Status
Date: Wed, 08 Sep 2021 23:55:09 GMT
Server: Apache/2.4.41 (Ubuntu)
Referrer-Policy: no-referrer
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Robots-Tag: none
X-XSS-Protection: 1; mode=block
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Security-Policy: default-src 'none';
Vary: Brief,Prefer
DAV: 1, 3, extended-mkcol, access-control, calendarserver-principal-property-search, nc-calendar-search, nc-enable-birthday-calendar
Content-Encoding: gzip
Keep-Alive: timeout=5, max=95
Transfer-Encoding: chunked
Content-Type: application/xml; charset=utf-8
Proxy-Connection: keep-alive


<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns">
...
</d:response>
<d:response>
    <d:href>/nextcloud/remote.php/dav/files/user/redacted/info/09D71C68-5CCA-469F-AAA1-C23F2AC7DC0F.JPG</d:href>
    <d:propstat>
        <d:prop>
            <d:getlastmodified>Sun, 09 Dec 2018 19:11:41 GMT</d:getlastmodified>
            <d:getetag>&quot;9f46f89482e0a3f145f0226011a52747&quot;</d:getetag>
            <d:getcontenttype>image/jpeg</d:getcontenttype><d:resourcetype/>
            <oc:fileid>16539</oc:fileid>
            <oc:permissions>RMGDNVW</oc:permissions>
            <oc:size>1198042</oc:size>
            <d:getcontentlength>1198042</d:getcontentlength>
            <nc:has-preview>true</nc:has-preview>
            <nc:mount-type>external</nc:mount-type>
            <x1:share-permissions xmlns:x1="http://open-collaboration-services.org/ns">19</x1:share-permissions>
            <oc:tags/>
            <oc:favorite>0</oc:favorite>
            <oc:comments-unread>0</oc:comments-unread>
            <oc:owner-id>redacted</oc:owner-id>
            <oc:owner-display-name>redacted</oc:owner-display-name>
            <oc:share-types/>
        </d:prop>
        <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
    <d:propstat>
        <d:prop>
            <d:quota-available-bytes/>
            <nc:is-encrypted/>
        </d:prop>
        <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>
</d:response>
<d:response>
...
</d:multistatus>

Proposed Solution

There are two potential solutions for the two issues. Preferably both would be implemented, but implementing only one should fix the main issue described:

  1. Extend the iOS client timeout for WebDAV requests to allow for very long response times, which would cater to handling very large folders.
  2. Change the iOS app's WebDAV request for folders, in to bring it into line with the NextCloud web app, which would reduce the overall response time for this type of request significantly.

Reasoning or why should it be changed/implemented?

Currently folders containing a very large number of files are displayed as empty in the iOS app. This incorrect view of a folder could create many undesirable outcomes (e.g. deleting incredibly large, populated folders, because the user believes they are empty due to the app reporting this state). It is also a huge inconvenience if you can't access files which you safely stored in NextCloud.

Environment data

iOS version: iOS 14.7.1

Nextcloud iOS app version: 4.0.6

Server operating system: Ubuntu 20.04.3 LTS

Web server: Apache 2

Database: MySQL 8

PHP version: PHP 7.4

Nextcloud version: Nextcloud 22.1.1

johnhindley96 avatar Sep 09 '21 01:09 johnhindley96

Hi, thanks for your report but this issue has not solution, the only solution would be the paging of results but is not available.

marinofaggiana avatar Sep 15 '21 12:09 marinofaggiana

No problem, and thanks for your response. Are the two suggested solutions (extending the timeout, and/or using a call similar to the web app which is much quicker) not viable options ? Appreciate your work on the app.

johnhindley96 avatar Sep 15 '21 23:09 johnhindley96

Has there been any development on this? This is still an issue.

bnordio avatar Mar 01 '22 14:03 bnordio

Similar issue here: iOS version: iOS 15.3.1 Nextcloud iOS app version: 4.3.1 Server operating system: Ubuntu 20.04.4 LTS Web server: Apache 2 Database: Postgres 14 PHP version: PHP 7.4 Nextcloud version: Nextcloud 23.03

Upload a 2.7GB file failed with an error: Timeout code:-1001

lingyanmeng avatar Apr 20 '22 09:04 lingyanmeng

@marinofaggiana We have that issue too. You are telling us that there is no way to increase that timeout of 60 secs?

nis65 avatar Jul 06 '22 14:07 nis65

@nis65 why you don't use the Chunk ? (Settings->Advanced->Chunk size in MB) ?

marinofaggiana avatar Jul 06 '22 14:07 marinofaggiana

@marinofaggiana thanks for you very quick reply :smile:

I just have set the chunk size in my iOS App to the max of 100MB and the problem still occurs immediately after login. The issue is not one big file, the issue is that user has several hundred files and directories in its root directory.

We will try now to reorganize that root directory and move all those files to a new subdirectory, let's see if that helps.

nis65 avatar Jul 06 '22 15:07 nis65

it is rare to reply immediately :D :D

Yes ... better !! the root directory with several hundred files is not good 😌 and underperforming for the network, when the App goback from background reload always (forced = true) the / and this uses network and time, on the contrary the subfolders are reloaded from network only when some subfile is changed (etag).

marinofaggiana avatar Jul 06 '22 15:07 marinofaggiana

On my installation a (sub)folder with about 300 subfolders plus 300 files works fine with my iOS App. It takes some time, but all folders and files are shown after some minutes 👍🏻

thomasmerz avatar Jul 06 '22 15:07 thomasmerz

Update from the support team: Cleaning up the root directory helped and it seems to be (did not reproduce it myself) not the amount of files, but one bad file that cannot be processed that causes the request take so long (or even loop indefinitely). Thanks again for your responsiveness :smile:

nis65 avatar Jul 08 '22 15:07 nis65

@nis65 , what was so "special" about this "bad file" that it couldn't be processed? I have also some UTF "special" filenames and no problems at all. 🤔

thomasmerz avatar Jul 10 '22 16:07 thomasmerz