[Bug]: TarStreamer library is incompatible, downloads fail.
⚠️ This issue respects the following points: ⚠️
- [X] This is a bug, not a question or a configuration/webserver/proxy issue.
- [X] This issue is not already reported on Github (I've searched it).
- [X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
- [X] Nextcloud Server is running on 64bit capable CPU, PHP and OS.
- [X] I agree to follow Nextcloud's Code of Conduct.
Bug description
When downloading largeish folders (>90 files, subfolders, etc.) on MacOS, Nextcloud fails to create the tar file of the folder.
Nextcloud uses the TarStreamer library here: https://github.com/owncloud/TarStreamer
But it is apparently not compatible with current versions of PHP:
https://github.com/owncloud/TarStreamer/issues/23#issuecomment-1449700060
The library needs to be brought up to date in order to use Nextcloud with secure versions of PHP/MacOS.
Steps to reproduce
- Create a folder with 90 files in a subfolder
- Open a browser on MacOS
- Attempt to download the folder
- Receive a truncated tar file
Expected behavior
Receive a complete tar file of the folder.
Installation method
Community Manual installation with Archive
Nextcloud Server version
26
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.1
Web server
Apache (supported)
Database engine version
MariaDB
Is this bug present after an update or on a fresh install?
Updated from a minor version (ex. 22.2.3 to 22.2.4)
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
- [X] Default user-backend (database)
- [X] LDAP/ Active Directory
- [ ] SSO - SAML
- [ ] Other
Configuration report
{
"system": {
"instanceid": "***REMOVED SENSITIVE VALUE***",
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
"10.150.64.10",
"***REMOVED SENSITIVE VALUE***"
],
"datadirectory": "***REMOVED SENSITIVE VALUE***",
"dbtype": "mysql",
"version": "26.0.0.11",
"overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
"htaccess.RewriteBase": "\/",
"dbname": "***REMOVED SENSITIVE VALUE***",
"dbhost": "***REMOVED SENSITIVE VALUE***",
"dbport": "",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"skeletondirectory": "\/var\/www\/skeleton",
"trashbin_retention_obligation": "1, 2",
"ldapUserCleanupInterval": "30",
"log_type": "syslog",
"logfile": "",
"loglevel": 1,
"syslog_tag": "Nextcloud",
"enable_previews": false,
"log.condition": {
"apps": [
"admin_audit"
]
},
"default_phone_region": "CA",
"memcache.local": "\\OC\\Memcache\\APCu",
"filelocking.enabled": "true",
"memcache.locking": "\\OC\\Memcache\\Redis",
"redis": {
"host": "***REMOVED SENSITIVE VALUE***",
"port": 0,
"dbindex": 0,
"password": "***REMOVED SENSITIVE VALUE***",
"timeout": 1.5
},
"maintenance": false,
"ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory"
}
}
List of activated Apps
Enabled:
- activity: 2.18.0
- admin_audit: 1.16.0
- bruteforcesettings: 2.6.0
- cloud_federation_api: 1.9.0
- contacts: 5.2.0
- dav: 1.25.0
- federatedfilesharing: 1.16.0
- files: 1.21.1
- files_downloadactivity: 1.16.0
- files_external: 1.18.0
- files_pdfviewer: 2.7.0
- files_rightclick: 1.5.0
- files_sharing: 1.18.0
- files_trashbin: 1.16.0
- lookup_server_connector: 1.14.0
- oauth2: 1.14.0
- password_policy: 1.16.0
- privacy: 1.10.0
- provisioning_api: 1.16.0
- related_resources: 1.1.0-alpha1
- serverinfo: 1.16.0
- settings: 1.8.0
- text: 3.7.2
- theming: 2.1.1
- twofactor_backupcodes: 1.15.0
- user_ldap: 1.16.0
- viewer: 1.10.0
- workflowengine: 2.8.0
Disabled:
- circles: 26.0.0 (installed 22.1.1)
- comments: 1.16.0 (installed 1.12.0)
- contactsinteraction: 1.7.0 (installed 1.3.0)
- dashboard: 7.6.0 (installed 7.2.0)
- encryption: 2.14.0
- federation: 1.16.0 (installed 1.12.0)
- files_versions: 1.19.1 (installed 1.15.0)
- firstrunwizard: 2.15.0 (installed 2.11.0)
- group_everyone: 0.1.11 (installed 0.1.11)
- logreader: 2.11.0 (installed 2.7.0)
- nextcloud_announcements: 1.15.0 (installed 1.11.0)
- notifications: 2.14.0 (installed 2.10.1)
- photos: 2.2.0 (installed 1.4.0)
- recommendations: 1.5.0 (installed 1.1.0)
- sharebymail: 1.16.0 (installed 1.12.0)
- support: 1.9.0 (installed 1.5.0)
- survey_client: 1.14.0 (installed 1.10.0)
- suspicious_login: 4.4.0
- systemtags: 1.16.0 (installed 1.12.0)
- twofactor_totp: 8.0.0-alpha.0
- updatenotification: 1.16.0 (installed 1.15.0)
- user_status: 1.6.0 (installed 1.2.0)
- weather_status: 1.6.0 (installed 1.2.0)
Nextcloud Signing status
No errors have been found.
Nextcloud Logs
Relevant log lines:
nc_msg_Exception
TypeError
nc_msg_File
/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php
nc_msg_Line
72
nc_msg_Message
decoct(): Argument #1 ($num) must be of type int, string given
nc_msg_Trace
{file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php, line=72, function=decoct},
{file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=200, function=getHeader, class=ownCloud\TarStreamer\TarHeader, type=->},
{file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=166, function=writeLongName, class=ownCloud\TarStreamer\TarStreamer, type=->},
{file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=102, function=initFileStreamTransfer, class=ownCloud\TarStreamer\TarStreamer, type=->},
{file=/var/www/nextcloud/lib/private/Streamer.php, line=167, function=addFileFromStream, class=ownCloud\TarStreamer\TarStreamer, type=->},
{file=/var/www/nextcloud/lib/private/Streamer.php, line=136, function=addFileFromStream, class=OC\Streamer, type=->},
{file=/var/www/nextcloud/lib/private/Streamer.php, line=141, function=addDirRecursive, class=OC\Streamer, type=->},
{file=/var/www/nextcloud/lib/private/legacy/OC_Files.php, line=217, function=addDirRecursive, class=OC\Streamer, type=->},
{file=/var/www/nextcloud/apps/files/ajax/download.php, line=77, function=get, class=OC_Files, type=::},
{file=/var/www/nextcloud/lib/private/Route/Route.php, line=155, args=[/var/www/nextcloud/apps/files/ajax/download.php], function=require_once},
{function=OC\Route\{closure}, class=OC\Route\Route, type=->, args=[*** sensitive parameters replaced ***]},
{file=/var/www/nextcloud/lib/private/Route/Router.php, line=324, function=call_user_func},
{file=/var/www/nextcloud/lib/base.php, line=1055, function=match, class=OC\Route\Router, type=->},
{file=/var/www/nextcloud/index.php, line=36, function=handleRequest, class=OC, type=::}
Full Logs:
application_name
Nextcloud
facility
user-level
facility_num
1
level
3
message
{"reqId":"DxUGPZDJ1qX1WR8O81ll","level":3,"time":"2023-04-14T20:21:01+00:00","remoteAddr":"10.150.64.1","user":"***REMOVED SENSITIVE VALUE***","app":"index","method":"GET","url":"/index.php/apps/files/ajax/download.php?***REMOVED SENSITIVE VALUE***","message":"{\"Exception\":\"TypeError\",\"Message\":\"decoct(): Argument #1 ($num) must be of type int, string given\",\"Code\":0,\"Trace\":[{\"file\":\"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php\",\"line\":72,\"function\":\"decoct\"},{\"file\":\"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php\",\"line\":200,\"function\":\"getHeader\",\"class\":\"ownCloud\\\\TarStreamer\\\\TarHeader\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php\",\"line\":166,\"function\":\"writeLongName\",\"class\":\"ownCloud\\\\TarStreamer\\\\TarStreamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php\",\"line\":102,\"function\":\"initFileStreamTransfer\",\"class\":\"ownCloud\\\\TarStreamer\\\\TarStreamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/lib/private/Streamer.php\",\"line\":167,\"function\":\"addFileFromStream\",\"class\":\"ownCloud\\\\TarStreamer\\\\TarStreamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/lib/private/Streamer.php\",\"line\":136,\"function\":\"addFileFromStream\",\"class\":\"OC\\\\Streamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/lib/private/Streamer.php\",\"line\":141,\"function\":\"addDirRecursive\",\"class\":\"OC\\\\Streamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/lib/private/legacy/OC_Files.php\",\"line\":217,\"function\":\"addDirRecursive\",\"class\":\"OC\\\\Streamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/apps/files/ajax/download.php\",\"line\":77,\"function\":\"get\",\"class\":\"OC_Files\",\"type\":\"::\"},{\"file\":\"/var/www/nextcloud/lib/private/Route/Route.php\",\"line\":155,\"args\":[\"/var/www/nextcloud/apps/files/ajax/download.php\"],\"function\":\"require_once\"},{\"function\":\"OC\\\\Route\\\\{closure}\",\"class\":\"OC\\\\Route\\\\Route\",\"type\":\"->\",\"args\":[\"*** sensitive parameters replaced ***\"]},{\"file\":\"/var/www/nextcloud/lib/private/Route/Router.php\",\"line\":324,\"function\":\"call_user_func\"},{\"file\":\"/var/www/nextcloud/lib/base.php\",\"line\":1055,\"function\":\"match\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/index.php\",\"line\":36,\"function\":\"handleRequest\",\"class\":\"OC\",\"type\":\"::\"}],\"File\":\"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php\",\"Line\":72,\"CustomMessage\":\"--\"}","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/109.0","version":"26.0.0.11"}
nc_app
index
nc_level
3
nc_message
{"Exception":"TypeError","Message":"decoct(): Argument #1 ($num) must be of type int, string given","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php","line":72,"function":"decoct"},{"file":"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php","line":200,"function":"getHeader","class":"ownCloud\\TarStreamer\\TarHeader","type":"->"},{"file":"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php","line":166,"function":"writeLongName","class":"ownCloud\\TarStreamer\\TarStreamer","type":"->"},{"file":"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php","line":102,"function":"initFileStreamTransfer","class":"ownCloud\\TarStreamer\\TarStreamer","type":"->"},{"file":"/var/www/nextcloud/lib/private/Streamer.php","line":167,"function":"addFileFromStream","class":"ownCloud\\TarStreamer\\TarStreamer","type":"->"},{"file":"/var/www/nextcloud/lib/private/Streamer.php","line":136,"function":"addFileFromStream","class":"OC\\Streamer","type":"->"},{"file":"/var/www/nextcloud/lib/private/Streamer.php","line":141,"function":"addDirRecursive","class":"OC\\Streamer","type":"->"},{"file":"/var/www/nextcloud/lib/private/legacy/OC_Files.php","line":217,"function":"addDirRecursive","class":"OC\\Streamer","type":"->"},{"file":"/var/www/nextcloud/apps/files/ajax/download.php","line":77,"function":"get","class":"OC_Files","type":"::"},{"file":"/var/www/nextcloud/lib/private/Route/Route.php","line":155,"args":["/var/www/nextcloud/apps/files/ajax/download.php"],"function":"require_once"},{"function":"OC\\Route\\{closure}","class":"OC\\Route\\Route","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":324,"function":"call_user_func"},{"file":"/var/www/nextcloud/lib/base.php","line":1055,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php","Line":72,"CustomMessage":"--"}
nc_method
GET
nc_msg_Code
0
nc_msg_CustomMessage
--
nc_msg_Exception
TypeError
nc_msg_File
/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php
nc_msg_Line
72
nc_msg_Message
decoct(): Argument #1 ($num) must be of type int, string given
nc_msg_Trace
{file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php, line=72, function=decoct},
{file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=200, function=getHeader, class=ownCloud\TarStreamer\TarHeader, type=->},
{file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=166, function=writeLongName, class=ownCloud\TarStreamer\TarStreamer, type=->},
{file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=102, function=initFileStreamTransfer, class=ownCloud\TarStreamer\TarStreamer, type=->},
{file=/var/www/nextcloud/lib/private/Streamer.php, line=167, function=addFileFromStream, class=ownCloud\TarStreamer\TarStreamer, type=->},
{file=/var/www/nextcloud/lib/private/Streamer.php, line=136, function=addFileFromStream, class=OC\Streamer, type=->},
{file=/var/www/nextcloud/lib/private/Streamer.php, line=141, function=addDirRecursive, class=OC\Streamer, type=->},
{file=/var/www/nextcloud/lib/private/legacy/OC_Files.php, line=217, function=addDirRecursive, class=OC\Streamer, type=->},
{file=/var/www/nextcloud/apps/files/ajax/download.php, line=77, function=get, class=OC_Files, type=::},
{file=/var/www/nextcloud/lib/private/Route/Route.php, line=155, args=[/var/www/nextcloud/apps/files/ajax/download.php], function=require_once},
{function=OC\Route\{closure}, class=OC\Route\Route, type=->, args=[*** sensitive parameters replaced ***]},
{file=/var/www/nextcloud/lib/private/Route/Router.php, line=324, function=call_user_func},
{file=/var/www/nextcloud/lib/base.php, line=1055, function=match, class=OC\Route\Router, type=->},
{file=/var/www/nextcloud/index.php, line=36, function=handleRequest, class=OC, type=::}
nc_remoteAddr
10.150.64.1
nc_reqId
DxUGPZDJ1qX1WR8O81ll
nc_time
2023-04-14 16:21:01.000
nc_url
/index.php/apps/files/ajax/download.php?***REMOVED SENSITIVE VALUE***
nc_user
***REMOVED SENSITIVE VALUE***
nc_userAgent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/109.0
nc_version
26.0.0.11
process_id
1822
source
nc
timestamp
2023-04-14 16:21:01.331
Additional info
No response
I get this as well on Linux. Tried to download 5GB of multiple folders with smaller files, and the tar file comes back with 257MB only.
~If it's downloaded as a ZIP file (i.e. smaller download), it works as intended. The moment it crosses the threshold and it becomes a TAR file, the TAR file gets downloaded, but it's incomplete.~ I cannot replicate it consistently, though, but will happily troubleshoot this further. It might be related to the size of the files it's archiving? I'll look into this and try to come up with an example that's replicable.
This is the error:
[index] Error: TypeError: decoct(): Argument #1 ($num) must be of type int, string given at <<closure>>
0. /var/www/html/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php line 72
decoct()
1. /var/www/html/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php line 200
ownCloud\TarStreamer\TarHeader->getHeader()
2. /var/www/html/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php line 166
ownCloud\TarStreamer\TarStreamer->writeLongName()
3. /var/www/html/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php line 102
ownCloud\TarStreamer\TarStreamer->initFileStreamTransfer()
4. /var/www/html/nextcloud/lib/private/Streamer.php line 168
ownCloud\TarStreamer\TarStreamer->addFileFromStream()
5. /var/www/html/nextcloud/lib/private/Streamer.php line 137
OC\Streamer->addFileFromStream()
6. /var/www/html/nextcloud/lib/private/legacy/OC_Files.php line 215
OC\Streamer->addDirRecursive()
7. /var/www/html/nextcloud/apps/files/ajax/download.php line 77
OC_Files::get()
8. /var/www/html/nextcloud/lib/private/Route/Route.php line 155
require_once("/var/www/html/n ... p")
9. <<closure>>
OC\Route\Route->OC\Route\{closure}("*** sensitive parameters replaced ***")
10. /var/www/html/nextcloud/lib/private/Route/Router.php line 306
call_user_func()
11. /var/www/html/nextcloud/lib/base.php line 1048
OC\Route\Router->match()
12. /var/www/html/nextcloud/index.php line 36
OC::handleRequest()
GET /index.php/apps/files/ajax/download.php?dir=***redacted***&downloadStartSecret=w1riopogit8
from XXX at 2023-06-07T06:00:59+00:00
After further investigation, I am not convinced that this has anything to do with a PHP version incompatibility - at least not directly. The error does not happen all the time - I'm just finishing downloading a 4GB TAR archive and all is well.
EDIT: and, on the flipside, I can consistently replicate this with a specific folder, particularly in a file that does not get added.
It crashes when decoding the mtime of the file being compressed, so I will look into that and report back in the coming weeks.
In my case, this is fixed with the following change:
In
/var/www/html/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php
Change:
['a12', str_pad(decoct($this->size), 11, '0', STR_PAD_LEFT)],
['a12', str_pad(decoct($this->mtime), 11, '0', STR_PAD_LEFT)],
To:
['a12', str_pad(decoct((int)$this->size), 11, '0', STR_PAD_LEFT)],
['a12', str_pad(decoct((int)$this->mtime), 11, '0', STR_PAD_LEFT)],
to force the explicit int cast.
@szaimen since you tagged this issue, I'd appreciate guidance here. I'm happy to submit a PR, but should I do it upstream, assuming they'll take it, or do you have any other recommendation?
Thanks all.
Best to open a PR here then: https://github.com/owncloud/TarStreamer
also cc @nextcloud/server-backend
Thank you. I submitted the PR here:
https://github.com/owncloud/TarStreamer/pull/25
I'll keep you posted.
Thanks for the guidance - the PR has been approved.
Is there anything that needs to be done on the Nextcloud front (a PR or something) to incorporate those changes here?
I guess they would need to publish a new release. Afterwards we can update the dependency here.
The cast to int will fail on 32bit for files larger than 4GB, not sure if this is a big deal or not (and whether without the cast it will work or not on 32bits).
-> I tested and it seems even without the cast this will crash on 32bit for sizes above 4GB, so the PR does not break anything.
@come-nc Thanks for the heads up.
Just to make sure I'm reading this correctly, what you're saying is that TarStreamer fails on 32bit systems on sizes above 4GB, with or without the patch. Is that correct?
There's a lot of nuances on 32bit systems and 4GB file sizes indeed. Well spotted.
Just to make sure I'm reading this correctly, what you're saying is that TarStreamer fails on 32bit systems on sizes above 4GB, with or without the patch. Is that correct?
Yes I think so, because those sizes on 32bits will always appears as float and decoct expects an int.
Hello.
I can confirm i got this issue with 64bits system.
I can also confirm i have solved it with the proposal from @pjft ont TarHeader.
Also on mac with firefox.
Apparently there is a new release already: https://packagist.org/packages/owncloud/tarstreamer#v2.1.0
Thanks for pushing for that. I take it that, come the next Nextcloud update, it will include this updated package, is that it? Or is there a separate update process?
Best.
Or is there a separate update process?
There is. Cc @nextcloud/server-backend
Hi all.
25.0.9 was just released and the fix is still not there - I can't seem to tag the backend team on GitHub, but if there's anything I can do to help here just let me know. I have fixed it manually for me, but I imagine other users are suffering from this without a way to fix it.
Thanks!
Hi all. Apologies for the repeated message, but 25.0.11 was just released and I still had to manually patch this. Is there anything I need to do to get this backported to 25?