Support cloud storage

[zhanghai]

Like Google Cloud, Dropbox, etc.

[flocke]

Generic WebDAV support would be nice to have as well. That would also cover Nextcloud and ownCloud.

[J-Jamet]

The code for these APIs is not under free licence. So how did you think about their integrations? Do you have flavors that will not compile the program with these APIs or are you going to make a plugin system?

[zhanghai]

Maybe https://stackoverflow.com/questions/672678/java-webdav-client-side-library ?

I don't have a plan for a plugin system now though, because of IPC and GC difficulties, etc. Most other file managers (maybe except MiXplorer?) created plugins to make them paid, and for me that's extra complexity while I don't want to get paid.

[alexanderadam]

The code for these APIs is not under free licence.

There are FOSS apps in F-Droid that are using proprietary cloud services like Dropbox. The official Dropbox SDK is MIT licensed. And a search shows that there are Google Drive libraries using the MIT license as well (i.e. 1 and 2.

Or is MIT not free enough? :thinking:

[J-Jamet]

This is simply not the case for the main Google Drive API. If this is the case for the majority, that is a good thing.

[alexanderadam]

This is simply not the case for the main Google Drive API. If this is the case for the majority, that is a good thing.

With API you mean libraries that are accessing the API? Or are there internal Android APIs on each installation? Can you elaborate on that? Are the two mentioned Google Drive libraries (1 and 2) not 'good'?

[J-Jamet]

I may have misspoken, I didn't look in detail at the licenses of all the cloud connection APIs, here you clearly stops at Google and Dropbox (but if you note correctly, there is also the word etc), so it may be generalized to many clouds. I am not specifically familiar with the google APIs you mention, maybe they are great, that is not the issue for me.

I asked the question because some APIs are not under a free license and even if they are, it clearly promotes non-free services, so they are anti-feature in F-Droid. So we have to find a way to integrate them in a way that respects the free software philosophy.

[alexanderadam]

I may have misspoken, I didn't look in detail at the licenses of all the cloud connection APIs, here you clearly stops at Google and Dropbox (but if you note correctly, there is also the word etc), so it may be generalized to many clouds.

I am well aware of that. I agree that there might some libraries not available, yet I'm sure that there are libraries for the most common providers.

I asked the question because some APIs are not under a free license and even if they are, it clearly promotes non-free services, so they are anti-feature in F-Droid.

Anti-Features won't result in an exclusion. It's just a hint in the F-Droid store, so users are notified about the included functionality. There are a lot of FOSS apps in F-Droid that are promoting non free services. See alone five GitHub clients, Instagram clients (1, 2) and even a bunch of Facebook apps.

Also I'm sure it's clear to every user that a GitHub client will use the proprietary GitHub service. Which is probably similar to this issue here: if a user wants Dropbox integration, it's hopefully clear, that Dropbox is not a FOSS service.

Avoiding proprietary services in free software in general, will probably lead to the situation, that users will have to use proprietary clients that might spy on them (see mister-monster's comment for example).

rclone is probably a good example that shows that free software can access the most relevant providers anyway (btw there's also an MaterialFiles issue for it).

[J-Jamet]

Yes I agree with you, I was just asking the question to know what would be the best way to integrate them into Material Files. What was answered and plugins are not an option.

I personally don't like anti-features too much because it repels new users but as you say, it's only a warning. For my personal case, I will simply compile the application manually to remove the APIs code.

[Ged296123]

MaterialFiles is on its way to being a very competent libre open source application. I would prefer it had nothing to do with proprietary keys and cloud services. As far as I know, most cloud services can be accessed in other ways (i.e. WebDAV) and would prefer to see MaterialFiles 'clean' in code.

[alexanderadam]

@zhanghai maybe it would be easier to create an integration for @x0b's rcx and create a symbiosis with a shared interface?

This way MaterialFiles wouldn't need to implement any particular cloud implementation (only an interface to rcx would be needed) and the cloud stuff itself is maintained by the rclone maintainers. Also this would allow to use rclone's transparent encryption on untrustworthy storages and the issues #191, #215 and #95 could be closed at once.

It's just an idea and maybe I'm just naive to think that something like this might be realistic, so I'm very sorry in case this idea doesn't make sense at all.

[x0b]

@alexanderadam You do that right now by activating "Enable Content Provider Preview" in Settings > File Access in RCX. Then, open Material Files, select Add storage > External Storage and add RCX from there.

RCX	Material Files

Since it is a pre-release feature, some things don't yet work 100%. For example, the BFE8-5420 remote (a USB drive) has been removed in RCX, but is still shown in the Material Files app.

@zhanghai Do you know about buildChildDocumentsUriUsingTree? Because when listing a directory, I'm seeing queryDocument calls for each and every element in that directory from Material Files. You can improve that massively by doing a single child elements call on the parent. For RCX, this isn't that big of a problem since all those child element calls can usually be answered straight from cache, but for the system provider (for SD cards/USB drives), this causes actual I/O for each and every element.

[zhanghai]

Hi x0b,

Thanks for looking into this! RCX is indeed a great app, and the DocumentsProvider support is cool and (IMO) a long awaited feature for the ecosystem.

I'm aware of buildChildDocumentsUriUsingTree as it's the only way to get the child documents under a tree and used in this app. The reason you are seeing a query for each child is just how the FS abstraction works, similar to the Linux readdir() + lstat() loop, where directory listing only lists the names, and the detailed information for each child is retrieved with a separate syscall. I believed this works fine with local things like ExternalStorageProvider, and cloud storage provider should have its own caching, hence I kept it the same for DocumentsProvider backed storage.

As a point of reference, are you observing the same behavior with the system DocumentsUI app? If it actually batches these calls together, maybe I can do something similar. It's not a good abstraction but if DocumentsUI do this, I'll just follow their practice.

[x0b]

Sorry for starting this bike shedding.

As a point of reference, are you observing the same behavior with the system DocumentsUI app? If it actually batches these calls together, maybe I can do something similar. It's not a good abstraction but if DocumentsUI do this, I'll just follow their practice.

I've checked the behaviour against Android 7 and 11. In all cases, I started browsing from the provider root and then opened the first folder on that root. Here's what I observed:

Log of calls for DocumentsUI on Android 7 and 11

queryDocument(): rootDoc
queryChildDocuments(): parent=rootDoc
queryChildDocuments(): parent=rootDoc/subfolder

There does not seem to be a difference between those versions.

Log of calls for Material files

queryDocument(): rootDoc/sub2:
queryDocument(): rootDoc
queryDocument(): rootDoc
queryChildDocuments(): parent=rootDoc
queryDocument(): rootDoc
queryChildDocuments(): parent=rootDoc
    queryDocument(): rootDoc/sub2:
    queryDocument(): rootDoc/sub3:
    queryDocument(): rootDoc/sub4:
    queryDocument(): rootDoc/sub5:
    queryDocument(): rootDoc/sub6:
    queryDocument(): rootDoc/sub7:
    queryDocument(): rootDoc/sub6:
queryChildDocuments(): parent=rootDoc/subfolder:
queryDocument(): rootDoc/subfolder:
queryChildDocuments(): parent=rootDoc/subfolder:
queryDocument(): rootDoc
    queryDocument(): rootDoc/subfolder:/subsub1
    queryDocument(): rootDoc/subfolder:/Document.pdf
    ...
    continuing with queryDocument() for each element

For some DocumentProviders like RCX (and Seafile as well I think) this is not a problem, since the details are most likely still in RAM from the previous queryChildDocuments() call. The client app (Material Files in this case) then only incurs the IPC/Binder overhead, but no disk or network I/O. Providers backed by local storage (like the system providers or Termux) tend to cache less and will instead hit the disk for every queryDocument() call.

For both Termux and the system provider, it looks like Material Files causes double metadata reads from disk. Once in the queryChildDocuments() call, then again for the individual queryDocument() calls.

However, I'm not sure how much of a difference all of this makes in practice. I recommend some profiling.

[zhanghai]

Thanks for the confirmation! I found that DocumentsUI seems to be batching the call here: https://cs.android.com/android/platform/superproject/+/master:packages/apps/DocumentsUI/tests/common/com/android/documentsui/DocumentsProviderHelper.java;l=294;drc=2788bf12a94a9c7131bd16b83964abab8f5435e8

I'll look into batching the calls in Material Files as well.

[zhanghai]

The improvement has been implemented in master and I checked with loading Android/data which became much faster.

[zhanghai]

Note that Google Drive doesn't seem to support WebDAV so this is different from #191.

[ctrlbrk42]

@alexanderadam You do that right now by activating "Enable Content Provider Preview" in Settings > File Access in RCX. Then, open Material Files, select Add storage > External Storage and add RCX from there.

RCX	Material Files

Since it is a pre-release feature, some things don't yet work 100%. For example, the BFE8-5420 remote (a USB drive) has been removed in RCX, but is still shown in the Material Files app.

Wow thank you for this, wasn't aware of RCX or this ability to get Drive inside Material! It's a hack but better than nothing I guess...

[ctrlbrk42]

@alexanderadam You do that right now by activating "Enable Content Provider Preview" in Settings > File Access in RCX. Then, open Material Files, select Add storage > External Storage and add RCX from there.

RCX	Material Files

Since it is a pre-release feature, some things don't yet work 100%. For example, the BFE8-5420 remote (a USB drive) has been removed in RCX, but is still shown in the Material Files app.

Wow thank you for this, wasn't aware of RCX or this ability to get Drive inside Material! It's a hack but better than nothing I guess...

I take it back. Very hacky and doesn't work, especially with my larger repos.

[rihaq]

https://github.com/itszechs/DriveStream

[oguzhane]

it would be indeed amazing to have rclone support. I have been looking a neat android app with privacy focused cloud storage. I have found none that meets the expections.

Material files is only open source app that is user friendly with great design.

I would be open contribute for adding rclone support.

[urmomseksy]

I saw RCX mentioned a bunch and just wanted to share that Round Sync is an active fork with presumably more updated code.

https://github.com/newhinton/Round-Sync