server
server copied to clipboard
nextcloud
[Bug]: 'Data directory protected' check failing
⚠️ This issue respects the following points: ⚠️
- [X] This is a bug, not a question or a configuration/webserver/proxy issue.
- [X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
- [X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
- [X] I agree to follow Nextcloud's Code of Conduct.
Bug description
The Data directory protected check DataDirectoryProtected.php still appears to try and check the URL even when the data directory is not in the webroot. The error also indicated that the .htaccess file is not working however it is. The runHead function is being passed $dataUrl which when I modified the code to log $dataUrl showed as just //var/nc_data and the .oc_data was also not appended.
Steps to reproduce
- Upgraded from Nextcloud 28.0.5 to 29.0.0.19 with data directory not in webroot
Expected behavior
The check should pass if data directory is not inside of the webroot.
Installation method
Community Manual installation with Archive
Nextcloud Server version
29
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.3
Web server
Apache (supported)
Database engine version
MySQL
Is this bug present after an update or on a fresh install?
Upgraded to a MAJOR version (ex. 22 to 23)
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
- [X] Default user-backend (database)
- [ ] LDAP/ Active Directory
- [ ] SSO - SAML
- [ ] Other
Configuration report
{
"system": {
"instanceid": "***REMOVED SENSITIVE VALUE***",
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
***Manually Removed***
],
"trusted_proxies": "***REMOVED SENSITIVE VALUE***",
"datadirectory": "/var/nc_data",
"dbtype": "mysql",
"version": "29.0.0.19",
"overwrite.cli.url": "***Manually Removed***",
"dbname": "***REMOVED SENSITIVE VALUE***",
"dbhost": "***REMOVED SENSITIVE VALUE***",
"dbport": "",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"activity_expire_days": 14,
"auth.bruteforce.protection.enabled": "true",
"simpleSignUpLink.shown": false,
"blacklisted_files": [
".htaccess",
"Thumbs.db",
"thumbs.db"
],
"cron_log": true,
"default_phone_region": "US",
"enable_previews": true,
"enabledPreviewProviders": [
"OC\\Preview\\PNG",
"OC\\Preview\\JPEG",
"OC\\Preview\\GIF",
"OC\\Preview\\BMP",
"OC\\Preview\\XBitmap",
"OC\\Preview\\Movie",
"OC\\Preview\\PDF",
"OC\\Preview\\MP3",
"OC\\Preview\\TXT",
"OC\\Preview\\MarkDown"
],
"filesystem_check_changes": 0,
"filelocking.enabled": true,
"htaccess.RewriteBase": "\/",
"integrity.check.disabled": false,
"knowledgebaseenabled": false,
"log_type": "file",
"logfile": "\/var\/nc_data\/nextcloud.log",
"loglevel": 2,
"logdateformate": "F d, Y H:i:s",
"logtimezone": "America\/New_York",
"log_rotate_size": 104857600,
"maintenance": false,
"memcache.local": "\\OC\\Memcache\\Redis",
"memcache.locking": "\\OC\\Memcache\\Redis",
"overwriteprotocol": "https",
"preview_max_x": 16000,
"preview_max_y": 16000,
"preview_max_scale_factor": 1,
"preview_max_memory": 512,
"redis": {
"host": "***REMOVED SENSITIVE VALUE***",
"port": 0,
"timeout": 0
},
"quota_include_external_storage": false,
"share_folder": "\/Shares",
"skeletondirectory": "",
"theme": "",
"trashbin_retention_obligation": "auto, 7",
"updater.release.channel": "stable",
"mail_smtpmode": "smtp",
"mail_smtpsecure": "ssl",
"mail_sendmailmode": "smtp",
"mail_from_address": "***REMOVED SENSITIVE VALUE***",
"mail_domain": "***REMOVED SENSITIVE VALUE***",
"mail_smtpauthtype": "LOGIN",
"mail_smtpauth": 1,
"mail_smtphost": "***REMOVED SENSITIVE VALUE***",
"mail_smtpport": "465",
"mail_smtpname": "***REMOVED SENSITIVE VALUE***",
"mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
"maintenance_window_start": 5,
"app_install_overwrite": [
"files_automatedtagging",
"imageconverter",
"files_archive"
]
}
}
List of activated Apps
- activity: 2.21.1
- admin_audit: 1.19.0
- bruteforcesettings: 2.9.0
- calendar: 4.7.1
- circles: 29.0.0-dev
- cloud_federation_api: 1.12.0
- comments: 1.19.0
- contacts: 6.0.0
- contactsinteraction: 1.10.0
- dashboard: 7.9.0
- dav: 1.30.1
- federatedfilesharing: 1.19.0
- federation: 1.19.0
- files: 2.1.0
- files_archive: 1.2.3
- files_automatedtagging: 1.19.0
- files_downloadlimit: 2.0.0
- files_external: 1.21.0
- files_pdfviewer: 2.10.0
- files_reminders: 1.2.0
- files_sharing: 1.21.0
- files_trashbin: 1.19.0
- files_versions: 1.22.0
- imageconverter: 2.0.1
- libresign: 9.0.0
- logreader: 2.14.0
- lookup_server_connector: 1.17.0
- mail: 3.6.0
- maps: 1.4.0
- nextcloud_announcements: 1.18.0
- notifications: 2.17.0
- oauth2: 1.17.0
- password_policy: 1.19.0
- photos: 2.5.0
- previewgenerator: 5.5.0
- privacy: 1.13.0
- provisioning_api: 1.19.0
- recommendations: 2.1.0
- related_resources: 1.4.0
- richdocuments: 8.4.0
- richdocumentscode: 24.4.103
- serverinfo: 1.19.0
- settings: 1.12.0
- sharebymail: 1.19.0
- support: 1.12.0
- systemtags: 1.19.0
- text: 3.10.0
- theming: 2.4.0
- twofactor_backupcodes: 1.18.0
- twofactor_totp: 11.0.0-dev
- updatenotification: 1.19.1
- user_status: 1.9.0
- viewer: 2.3.0
- weather_status: 1.9.0
- workflowengine: 2.11.0
Nextcloud Signing status
No errors have been found.
Nextcloud Logs
{"reqId":"OCPRKEMAJw49JilEK9oK","level":2,"time":"2024-04-28T19:51:20-04:00","remoteAddr":"10.0.0.70","user":"user-omitted","app":"no app in context","method":"GET","url":"/core/preview?fileId=604398&x=32&y=32&mimeFallback=true&a=0","message":"Transaction took 1.2166879177094s","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36","version":"29.0.0.19","exception":{"Exception":"Exception","Message":"Transaction took 1.2166879177094s","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/DB/ConnectionAdapter.php","line":154,"function":"commit","class":"OC\\DB\\Connection","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":523,"function":"commit","class":"OC\\DB\\ConnectionAdapter","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":404,"function":"handleChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":354,"function":"scanChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/LocalRootScanner.php","line":39,"function":"scan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":1341,"function":"scan","class":"OC\\Files\\Cache\\LocalRootScanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":1380,"function":"getCacheEntry","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Node/Root.php","line":208,"function":"getFileInfo","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Node/LazyFolder.php","line":161,"function":"get","class":"OC\\Files\\Node\\Root","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/AppData/AppData.php","line":98,"function":"get","class":"OC\\Files\\Node\\LazyFolder","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/AppData/AppData.php","line":147,"function":"getAppDataFolder","class":"OC\\Files\\AppData\\AppData","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Storage/Root.php","line":74,"function":"newFolder","class":"OC\\Files\\AppData\\AppData","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Generator.php","line":607,"function":"newFolder","class":"OC\\Preview\\Storage\\Root","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Generator.php","line":133,"function":"getPreviewFolder","class":"OC\\Preview\\Generator","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Generator.php","line":110,"function":"generatePreviews","class":"OC\\Preview\\Generator","type":"->"},{"file":"/var/www/nextcloud/lib/private/PreviewManager.php","line":187,"function":"getPreview","class":"OC\\Preview\\Generator","type":"->"},{"file":"/var/www/nextcloud/core/Controller/PreviewController.php","line":174,"function":"getPreview","class":"OC\\PreviewManager","type":"->"},{"file":"/var/www/nextcloud/core/Controller/PreviewController.php","line":142,"function":"fetchPreview","class":"OC\\Core\\Controller\\PreviewController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":232,"function":"getPreviewByFileId","class":"OC\\Core\\Controller\\PreviewController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":138,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":184,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":338,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/lib/base.php","line":1050,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/index.php","line":49,"function":"handleRequest","class":"OC","type":"::"}],"File":"/var/www/nextcloud/lib/private/DB/Connection.php","Line":691,"message":"Transaction took 1.2166879177094s","exception":{},"CustomMessage":"Transaction took 1.2166879177094s"}}
Additional info
The errors in logs appear when browsing various file heavy folders and/or the Photos app.
Modified function in DataDirectoryProtected.php:
public function run(): SetupResult {
$datadir = str_replace(\OC::$SERVERROOT . '/', '', $this->config->getSystemValue('datadirectory', ''));
$webRoot = $this->urlGenerator->getWebroot();
$dataUrl = $this->urlGenerator->getWebroot() . '/' . $datadir . '/.ocdata';
error_log('webRoot: '. $webRoot . ' DataDir: '. $datadir . ' DataUrl: ' . $dataUrl, 0);
$noResponse = true;
foreach ($this->runHEAD($dataUrl, httpErrors:false) as $response) {
$noResponse = false;
if ($response->getStatusCode() === 200) {
return SetupResult::error($this->l10n->t('Your data directory and files are probably accessible from > } else {
$this->logger->debug('[expected] Could not access data directory from outside.', ['url' => $dataUrl]); }
}
if ($noResponse) {
return SetupResult::warning($this->l10n->t('Could not check that the data directory is protected. Please chec> }
return SetupResult::success();
}
Error logs this:
Got error 'PHP message: webRoot: DataDir: /var/nc_data DataUrl: //var/nc_data/.ocdata'
Not sure why it doesn't get my webRoot directory though or why the if ($response->getStatusCode() === 200) is returning true as my thought would be the URL would make the runHEAD fail.
Not sure why it doesn't get my webRoot directory though or why the if ($response->getStatusCode() === 200) is returning true as my thought would be the URL would make the runHEAD fail.
Unless you've deployed Nextcloud in a subfolder, webroot is expected to be empty (well, or /). That part seems right.
What happens if you try to access the full URLs being used in the check by utilizing each of your configured trusted_domains from a web browser (or with curl)?
https://domain1.com/var/nc_data/.ocdata https://domain2.com/var/nc_data/.ocdata
Or, if you prefer:
https://domain1.com//var/nc_data/.ocdata
That would be the problem. One of my trusted domains was the box's internal FQDN but the box's DNS is set to public servers. Is there any reason for not bypassing the check if the data directory isn't inside of the webroot?
A check like I created in this possibly: https://github.com/nmbgeek/nextcloud-server/blob/master/apps/settings/lib/SetupChecks/DataDirectoryProtected.php
or at least have it only check the value in 'overwrite.cli.url'?
That would be the problem. One of my trusted domains was the box's internal FQDN but the box's DNS is set to public servers.
That shouldn't be a problem. We already anticipate that scenario; a similar check is used for .mjs files. If one of the domains tested isn't reachable we move on and try the others.
You didn't answer my question: what happens if you try to access each of the full URLs being used in the check? Does one of them succeed?
Is there any reason for not bypassing the check if the data directory isn't inside of the webroot?
Symbolic links come to mind (maybe? I haven't had my morning coffee yet). But it shouldn't matter either way, the test should pass unless we've either got a bug or an .ocdata file really is accessible via one of your URLs.
or at least have it only check the value in 'overwrite.cli.url'?
It doesn't matter if some of the trusted_domains aren't reachable/etc. We just look for one of them (trusted_domains or the overwrite.cli.url) to work.
That shouldn't be a problem. We already anticipate that scenario; a similar check is used for
.mjsfiles. If one of the domains tested isn't reachable we move on and try the others. You didn't answer my question: what happens if you try to access each of the full URLs being used in the check? Does one of them succeed?
The second domain resolves to a different webserver on public DNS due to wildcard subdomain dns record which returns a 200 status code for its generic default page. Internal DNS resolves it directly rather than having to hairpin at my router which then forwards it to Nginx Proxy Manager before hitting the nextcloud server. The nextcloud server's DNS is set to public DNS rather than internal so it is resolving the domain to a public address unlike clients using internal dns which resolve it directly to the nextcloud box.
It was setup some time ago like that for some obscure reason, I believe it had something to do with troubleshooting NPM, and I have removed that domain so the "issue" I was having of it reporting the erroneous error is technically resolved. I think the check could be done better as it never was the issue reported by nextcloud that my .htaccess wasn't working and that my data directory was possibly publicly available.
I can confirm that removing the network's central proxy server (also Nginx Proxy Manager) from the trusted domains in Nextcloud's config resolved the issue.
And I also forgot why I had to add the proxy server in the first place.
For me neither works. Everything in the config is correct but still for some reason the error is there. i have checked several times that the data directory is not accessible. folder is even deleted. Still the check fails. On the current system where it shows we only have one trusted domain. overwrite is correct.
curl -I https://
This is strange. Curl gives a 404 but the code it self suggest that its getting a 200 ok.
I think there is actually an improvement possible for the implementation of the check:
If your request is redirected, and you don't receive /.ocdata then some mechanism is protecting the data directory.
Posting this here as well https://github.com/nextcloud/docker/issues/2214#issuecomment-2112966891 since I think it's relevant for this issue.
What about if the check was for the .ocdata file with a non-forbidden status code and a content-length of 0? Since the .ocdata file is just an empty file if it is actually getting the real ocdata file it should have a content length of 0.
I have been testing around more. I added debug to the actual check itself to se what url and what host it checks. On this system it is only one host. And the complete url that it checks gives a 404 with curl.
Found it! I dont know why i missed this. Rewrite rule on port 80 for https generates a 301 as response. That is why its failing.
Found it! I dont know why i missed this. Rewrite rule on port 80 for https generates a 301 as response. That is why its failing.
So, the Nextcloud PHP code should be able to handle redirects. That's the solution.
Found it! I dont know why i missed this. Rewrite rule on port 80 for https generates a 301 as response. That is why its failing.
So, the Nextcloud PHP code should be able to handle redirects. That's the solution.
Or correctly generate Test-URLs honoring overwriteprotocol and overwrite.cli.url. In my setup both point to https so why does the test try to access the server using http anyway!
Found it! I dont know why i missed this. Rewrite rule on port 80 for https generates a 301 as response. That is why its failing.
So, the Nextcloud PHP code should be able to handle redirects. That's the solution.
Or correctly generate Test-URLs honoring
overwriteprotocolandoverwrite.cli.url. In my setup both point tohttpsso why does the test try to access the server using http anyway!
Good point!
it tests for missconfiguraton. Imagine if you forgot to protect stuff on port 80 but had a perfect config for ssl.
it tests for missconfiguraton. Imagine if you forgot to protect stuff on port 80 but had a perfect config for ssl.
Also good point! :laughing:
Found it! I dont know why i missed this. Rewrite rule on port 80 for https generates a 301 as response. That is why its failing.
So, the Nextcloud PHP code should be able to handle redirects. That's the solution.
I have only set trusted_domains to the domain nextcloud is running (no trusted_proxy) and I can confirm that once I remove 80 -> 443 redirect, the error is finally gone! \o/
you can add a redirect without generationg a 301 and it would work. Either a better rewrite rule or a Redirct statement
I have only set trusted_domains to the domain nextcloud is running (no trusted_proxy) and I can confirm that once I remove 80 -> 443 redirect, the error is finally gone! \o/
I'm not following why you would want to disable https redirects though.
you can add a redirect without generationg a 301 and it would work. Either a better rewrite rule or a Redirct statement
The default certbot rule that it creates for port 80 is below. Why wouldn't you want all http traffic redirected to https?
RewriteEngine on
RewriteCond %{SERVER_NAME} =example.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
Hi. Not sure if I should create another ticket or add to this, but since this seems to be related even without being totally the same, I think it's better to add here. Apologies if I'm wrong.
I also have the same issue but using nginx directly, not Apache. The test is failing, even though the data directory returns a 404, by design. It exists in the webroot, but nginx is configured to hide it by default:
# Rules borrowed from `.htaccess` to hide certain paths from clients
location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/) { return 404; }
location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) { return 404; }
(https://github.com/nextcloud/documentation/blob/master/admin_manual/installation/nginx-root.conf.sample#L135)
I'm confused as to why the .htaccess file is tested, since this will always result in a false negative when using nginx. I know nginx isn't officially supported by Nextcloud, but you can't take for granted in the code that Apache is used. If a request to .ocdata returns anything other that 200, the test shouldn't fail, right ? In my case a request to anything under /data does return 404 as expected, so the test should be valid since the data isn't accessible.
I'm confused as to why the .htaccess file is tested, since this will always result in a false negative when using nginx. I know nginx isn't officially supported by Nextcloud, but you can't take for granted in the code that Apache is used. If a request to .ocdata returns anything other that 200, the test shouldn't fail, right ? In my case a request to anything under /data does return 404 as expected, so the test should be valid since the data isn't accessible.
The .htaccess itself isn't tested. The current design does a curl for the location of your data directory appended to all of your trusted_domains from your config. Review your config file for trusted domains and where it locates your data directory as being (minus the server root portion of your data directory if applicable). Then append /.ocdata to that and perform a 'curl -l' from your server's cli. The actual check can be found here: https://github.com/nextcloud/server/blob/master/apps/settings/lib/SetupChecks/DataDirectoryProtected.php
I just think this check could be done much better and be made to be more robust.
I'm confused as to why the .htaccess file is tested, since this will always result in a false negative when using nginx. I know nginx isn't officially supported by Nextcloud, but you can't take for granted in the code that Apache is used. If a request to .ocdata returns anything other that 200, the test shouldn't fail, right ? In my case a request to anything under /data does return 404 as expected, so the test should be valid since the data isn't accessible.
The .htaccess itself isn't tested. The current design does a curl for the location of your data directory appended to all of your trusted_domains from your config. Review your config file for trusted domains and where it locates your data directory as being (minus the server root portion of your data directory if applicable). Then append /.ocdata to that and perform a 'curl -l' from your server's cli. The actual check can be found here: https://github.com/nextcloud/server/blob/master/apps/settings/lib/SetupChecks/DataDirectoryProtected.php
I just think this check could be done much better and be made to be more robust.
Thanks for the explanation. I'll admit the details went over my head when trying to read the discussion. However, I did the test and when testing with curl like so curl -l https://[a domain specified in trusted_domains]/data/.ocdata I correctly get a 404, both when testing from the server itself and outside.
I'm pretty sure my path to reach .ocdata is correct, because when I comment the corresponding lines in nginx, a following curl returns the file. So I do think the test in Nextcloud is a false positive, though I'm not sure why. If it does the same curl I did, it should've got a 404. Is there any way to log that specific test ?
@Tangeek42 form my tests i found that a 301 on port 80 made the test fail. It does test both 80 and 443 for the hostname.
Oh good point, I do have a 301 on :80. A curl on :80 returns 301, on :443 returns 404. So I'm guessing it stops at the first test and considers it failed when it shouldn't ?
@Tangeek42 yepp. If you change the rewrite so it doesnt send a 301 the problem is solved.
Not sure I want to disable the redirect from HTTP to HTTPS, though. I can live with a false positive now that I understand why it does what it does. :D
EDIT : Oh man... Re-reading the previous discussion, you all already got to that point. I'm sorry, I guess I wasn't awake enough last week to make the connection to my case.
change the rewrite to this:
RewriteEngine On RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [END,NE,R=permanent]