`shutil.copy()` fails on external removable storage devices

[manuq]

In my app for ChromeOS I'm trying to use an external SD card or USB stick for storage because it needs to store many Gigabytes of content. The following code works for choosing the first external device that matches my criteria:

    target_dir = None
    # Find the first removable device:
    for external_file_dir in get_activity().getExternalFilesDirs(None):
        state = Environment.getExternalStorageState(external_file_dir)
        is_removable = Environment.isExternalStorageRemovable(external_file_dir)
        if is_removable and state == 'mounted':
            target_dir = external_file_dir
            break
 
    # Fallback to internal storage:
    if target_dir is None:
        target_dir = get_activity().getExternalFilesDir(None)

But I'm noticing issues when using this storage compared with the internal storage. One of them is that shutil.copy() fails with "PermissionError: [Errno 1] Operation not permitted". Here is how my code continues:

    shutil.copy(test_path, os.path.join(target_dir.toString(), "test.txt"))

This fails with:

04-28 15:55:41.534  6485  6548 I python  : Traceback (most recent call last):
04-28 15:55:41.534  6485  6548 I python  :   File "/path/to/src/main.py", line 4, in <module>
04-28 15:55:41.535  6485  6548 I python  :   File "/path/to/src/initialization.py", line 35, in <module>
04-28 15:55:41.536  6485  6548 I python  :   File "/path/to/src/android_utils.py", line 136, in get_target_dir
04-28 15:55:41.540  6485  6548 I python  :   File "/cache/.local/share/python-for-android/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/Lib/shutil.py", line 428, in copy
04-28 15:55:41.541  6485  6548 I python  :   File "/cache/.local/share/python-for-android/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/Lib/shutil.py", line 317, in copymode
04-28 15:55:41.544  6485  6548 I python  : PermissionError: [Errno 1] Operation not permitted: '/storage/D15429D1D0DFFFBDC65168977D6460FF8EC5E06F/Android/data/com.test.Test/files/test2.txt'

Note that this works fine using the internal storage. Ironically, the file is copied. So I tried a number of other ways to copy or write files and found that:

• shutil.move(), shutil.copytree(), shutil.copy2() and shutil.copystat() are giving the same PermissionError error.
• shutil.copyfile() works.
• Opening a file for read, write or append using with open() works.

I tried with different external SD cards and USB sticks and different formats, mainly NTFS. I am suspecting this Python bug https://bugs.python.org/issue28141

Versions

• Python: 3.8.5 and 3.9.12 (I tried 3.9 by changing .p4a to --requirements python3==3.9.12,hostpython3==3.9.12)
• OS:
• Kivy: Not using Kivy, using python-for-android v2022.03.13 directly.
• Cython: 0.26.1-0.4 (from apt package repository of ubuntu:bionic docker image)
• OpenJDK:8u162-b12-1 (from apt package repository of ubuntu:bionic docker image)

[manuq]

Update: @dbnicholson went to the bottom of this Android issue and found a workaround by monkeypatching: https://github.com/endlessm/kolibri-installer-android/commit/e963ac2a48cfc51109c4d443d10f7d0b68a91523

[Julian-O]

Closing as not an issue in p4a code.