Opening publications from the shared external storage on Android 10+ (scoped storage)

[mickael-menu]

I'm opening an issue to gather experiences from other integrators on interfacing with the shared external storage on Android 10+.

Background

Apps used to be able to write and read files from the shared external storage freely. But since Android 10, Google introduced scoped storage which limits its access to protect user privacy.

In this new model, an app can:

• freely read/write files in its App-Specific Storage, which users or other apps can't access
• request explicit read/write access to a specific directory or folder using the Storage Access Framework

Storing publications in the App-Specific Storage is less than ideal, because:

• they are deleted when the app is uninstalled
• they can't be directly shared with other apps
• Google explicitly states that ebooks should be stored in the shared external storage:

  Documents and other files: The system has a special directory for containing other file types, such as PDF documents and books that use the EPUB format. Your app can access these files using the platform's Storage Access Framework.

The Problem

The Storage Access Framework is an abstraction layer over the FS or Cloud-based storages. Unfortunately it doesn't provide direct access to file paths to the app. To read a file's content, we're given either:

• an InputStream
• a low-level file descriptor wrapped in a ParcelFileDescrptor

Both AndroidPdfium and the native PdfRenderer can read a file descriptor, so it could work for PDF publications.

However, we're stuck for any ZIP-based publications (EPUB, RWP, etc.) because the native ZipFile API requires a File or direct file path to operate.

A New Hope

Following community ~~outrag~~ feedback, Google backtracked a bit and restored access to file paths of media files in Android 11:

To help your app work more smoothly with third-party media libraries, Android 11 (API level 30) and higher allow you to use APIs other than the MediaStore API to access media files from shared storage. You can instead access media files directly using either of the following APIs:

• The File API.
• Native libraries, such as fopen().

Sadly, ebooks don't seem to be considered "media files" and I couldn't manage to read raw publication paths on Android 11.

See this Android issue for more information.

Possible Solutions?

Here are a few leads to address this issue:

• Storing publications in the App-Specific Storage. This sucks for the reasons outlined before but is the easiest solution right now. It's also how it's implemented in the test app.
• Working with the file descriptor. This is probably the solution intended by Google but it means that we need to find a third-party ZIP library. I could find some pieces working with file descriptors.
• Copying files to a temporary location upon reading, then delete them when closing the publication. It is slow and might fails if the disk space is low. This might get ugly with large audiobooks.
• Requesting the MANAGE_EXTERNAL_STORAGE permission. This provides an app with raw access to the whole external storage as before. But it is intended for special cases like a File Manager app and needs a special grant from Google, so highly unlikely for reading apps.

A temporary workaround for apps targeting Android 10 is to use the requestLegacyExternalStorage flag. This reverts back to the full access to the external storage file paths. Unfortunately, Google will require apps to target Android 11 by the end of the year, rendering this flag useless.

[mickael-menu]

I opened an issue on Android's issue tracker: https://issuetracker.google.com/issues/177246262

[mickael-menu]

Google answered:

We will consider file path access for SAF in a future release. At the moment it is not supported.

Not sure we can do much better than copying publications to the app-specific storage for now.

[krawa]

Use a ZipInputStream instead of a ZipFile

[mickael-menu]

@krawa ZipInputStream doesn't support rewinding or marks. It can be useful to read an in-memory ZIP from start to end but we need random access to render a publication. Do you know of a workaround for this using ZipInputStream?

[mrifni]

Have you seen this https://stackoverflow.com/questions/9287664/why-cant-a-randomaccessfile-be-casted-to-inputstream

// STEP 1:  Create random access file read-only
RandomAccessFile raf = new RandomAccessFile("/text.txt", "r");

// STEP 2:  Use Channels to convert to InputStream
InputStream is = Channels.newInputStream(raf.getChannel());

[mickael-menu]

We need access to a file path or File object to use ZipFile, input streams don't solve our problem in this case.

[krawa]

File API and ZipFile can no longer be used. It is deprecated. Here is my implementation of Archive https://gist.github.com/krawa/36b9eebd748dfff3a427069696550ec3

[mickael-menu]

@krawa

File API and ZipFile can no longer be used. It is deprecated.

I didn't see this, do you have a source?

Here is my implementation of Archive https://gist.github.com/krawa/36b9eebd748dfff3a427069696550ec3

Thanks that looks very interesting. Happy to see the PublicationAsset API put to good use. Since you implemented Archive, you can use ArchiveFetcher(uriArchive) directly. Did you see any performance impact from opening a new ZipInputStream for each requested resource?

If this pans out well we could consider adding your implementation in Readium if you want.

[qnga]

Solved in Readium 3.0 with a custom ZIP implementation.