Improved StorageFile APIs

[jonwis]

Proposal: Improved file-access APIs for UWP and AppContainer

Summary

Provide an updated version of StorageFile that fixes performance and usability issues in the current Windows.Storage.Storage* set of types. Support apps that think in terms of file paths & handles instead of StorageFile objects.

Rationale

The current Windows.Storage.Storage* types are designed to give programmatic access to regular files, shell namespace items, and other Windows entities backed by files. A single abstraction over all those spaces is interesting but comes with high costs - namely performance and impedence-mismatch with existing application code. Project Reunion should clearly explain why StorageFile is important and - where possible - provide lower-level access to the user's files.

• The BroadFileAccess capability is "scary" to customers but required even for limited UWP scenarios.
• Many libraries and helpers take file paths instead of objects (even instead of handles)
• Apps observe serious performance costs related to using StorageFile's built in functionality
• Low-level code can access the ...FromApp API variants, but higher-level code (JS, C#) interop for them is very limited.

Scope

Capability	Priority
App has direct access to any files the user has given it using low-level APIs	Must
Feature exposes an object-oriented view of ...FromApp platform APIs	Must
APIs in this feature are synchronous after user permission is acquired	Must
Feature provides standard IStream/ISequentialStream APIs for these files	Should
App can convert between existing W.S.Storage* and types in this functionality	Should
UWPs can access Shell Namespace types directly with user approval	Should
APIs taking or returning StorageFile should return/accept these objects instead	Should
App can prompt the user for broader access without requiring an app restart	Could
App can access user data without user intervention	Won't

Open Questions

Should UWPs be given access to the IShell* family of APIs?

As these APIs are not truly universal on all editions of Windows, if an app were to program against them directly the app would not work on editions like HoloLens or Xbox.

Should Reunion warn the user about apps using this capability?

Some platforms periodically remind users that "App X is using feature Y, are you OK with that?" How should Reunion make the user aware that a "low rights" application is using capabilities?

[MarkIngramUK]

For applications that are already compatible with sandboxing (i.e. only read from %APPDATA% or %PROGRAMDATA%, and only access files explicitly chosen by a user) I don't think a continuous reminder is warranted. This could perhaps be controlled via some sandbox capabilities options that are chosen at development time (similar to the macOS model).

[eklipse2k8]

A good test of this feature would be, I can create an SQLite database in the user's documents folder. I shouldn't need permission to do this if it's a new file. This is such a common scenario that the sandbox and the politics around it have never tried to solve for. An app should never need "broadFileAccess" but they should also make it so users can easily open documents and see their saved file right there.

[soumyamahunt]

A good test of this feature would be, I can create an SQLite database in the user's documents folder.

Why not create in AppData folder itself? A lot of users have complained apps creating random folders in their library folders like Documents, Music etc. introduces clutter to their workspaces and they have more trouble in finding their Work folders. Also, this kind of behavior from developers is what prompted MS to limit storage access in first place.

[benstevens48]

It’s really great that this is being looked into, and that it’s the first issue here. UWP file performance and permissions is the thing I’ve most wanted to see an improvement in since 2015! I imagine that any permissions changes will be difficult to deliver via NuGet, so hopefully something can happen on this as soon as possible in order to ensure we don’t have to wait too long before the users have the necessary OS updates.

Below I’ve put together a collection of scenarios related to file system performance and permissions that are problematic in UWP. These are things I have encountered while building my photo-viewing app. (Maybe it’s a bit selfish of me to concentrate on these problems, but hopefully they cover a wide range of use cases). If this post is too long, feel free to move it to a separate issue. Also, I am very happy to discuss any of these issues/ideas further. The scenarios themselves come first, followed by the problems with them in UWP and some possible solutions. I hope that this project will make time to think more creatively about how to deal with some of the file permissions issues. Some of my suggestions below are just initial ideas, so don’t judge them too harshly. It’s such an important subject that I think it’s worth spending a lot of time thinking about.

The scenarios

0. The user does not necessarily keep all their images in their Pictures library

I put this in as point 0 as it influences all the others. A key principle of my app is to allow users to organise their photos how they like and not to force them into any patterns. Therefore the app cannot assume that all the user’s images are in their Pictures library, and indeed it’s designed for viewing all images, even for web designers etc who are probably not keeping images in there. Also, I’m going to assume that broadFileSystem access is not used, because that is perhaps too unrestricted and also the current user experience for it is poor.

1. User selects a folder containing a large number of files (can include sub-folders) to view in the app

My app has quite a simple concept of allowing users to select files they want it to display. They can do this by selecting a folder, optionally including subfolders, or selecting individual files etc. They can make multiple selections. It is not uncommon for a folder selected to contain 10,000+ image files. The app needs to maintain a list of the selected files (including the order they were selected in etc), therefore it needs to be able to list files in folders, and the quicker and less resource-intensive this is, the better. Obviously the app needs to be able to access the contents of these files alter on, but at this stage a path and file name is all that’s needed.

2. The app needs to be able to watch for files added/removed/modified in the folders that the user has selected

My app has the option to auto-refresh the selected files to match the contents of the folders that the user selected to view. This can include modifications to files as well as files added/removed/renamed.

3. The app makes heavy use of StorageFile.GetBasicPropertiesAsync StorageItemContentProperties. RetrievePropertiesAsync to get properties such as DateTaken and Orientation

This should ideally be fast. Also, it expects to be able to read properties simultaneously.

4. The user can select files/folders via drag and drop, and expects to be able to edit these files

This is self-explanatory.

5. The app offers a ‘Duplicate file’ function

This is self-explanatory. There is a button in the app that allows the user to duplicate the file they are currently viewing in the app. (The file is one of those selected in scenario 1). This should place a copy of the file in the same directory as that file, with the name suffixed by ‘ Copy’ or similar. It should be a one-click operation as it’s a convenience function for the user (e.g. to make a copy to edit).

6. The app offers a button to load the other files in the same folder as the file that the user is currently viewing

The user may have selected an individual file to view in scenario 1. The app would like to show a button that would load the remaining files from the same folder in this case, ideally via a 1-click operation.

7. The app maintains a ‘quick access’ or ‘favourites’ list of files. Generally, clicking on a file in this list should also load the neighbors, as launching from File Explorer does.

Users who make heavy use of this list may add hundreds or even thousands of files to it.

8. The app would like to offer a feature that allows users to save their session to a file, e.g. to create a slideshow that they can bring up straight away

Such as slideshow may consist of thousands of images, which were originally selected in multiple different ways. The saved data file would store the file paths of these images, and when the data file is loaded it needs to be able to load the associated images.

9. The user can load files by double-clicking on a file in file explorer. This should load the neighboring files.

The user needs to be able to do the same things with these files that they can do with files selected another way. An important feature of this is that the neighboring files are ordered in the same way as in File Explorer. The neighboring files list should be resilient to problems with the file system indexer.

The challenges faced in a UWP app

Scenario 1

The challenge here is simply that listing the StorageItems in a StorageFolder is incredibly slow and resource intensive compared to using the standard .Net/Win32 API to list the file paths. A quick test showed that in .Net it takes about 0.003ms per file, whereas with UWP StorageItems it takes about 2ms per file, which is around 700 times slower. Also, looking at the Runtime Broker, UWP uses about 70kB memory per file, which is a huge cost when loading 10,000+ files, whereas .Net uses around 0.1kB per file (that’s a very rough estimate).

Scenario 2

UWP has the StorageFileQueryResult.ContentsChanged method for watching for file system changes. However, it has absolutely no configuration options for what sort of changes to watch for, so it will fire on any change, even something like last access time I think, and include subfolders. (I agree that if you have no configuration options that firing on any change is the best option). It also does not tell you what changed. Combined with the speed issues in scenario 2, this makes for a doubly bad experience. If you look at the .Net docs of file system watching, they give an example of watching for changes in “C:\”, which made me laugh. In UWP, assuming that you managed to get permission for this folder, in order to determine what changed, you would first have to enumerate all the contents, which would literally take several days, then on a change you would have to enumerate all the contents again, taking several days, and in fact several weeks if you also want to check the date modified to detect file modifications, and look at the difference. Also, at one stage, I tried adding a separate watching for each of 600 sub-folders that had been added to the app (in order to avoid having to query all the files for changes) and basically it ground to a halt – it seems to create a new thread for each watcher which is not necessarily very efficient.

Scenario 3

I added this because I’m not aware of a way to do this easily in .Net without writing interop code. Also, for any alternative to StorageFile in UWP, it needs to be possible to get these properties ideally without the overhead of creating a StorageFile. Also, there are some issues with these in UWP. Firstly, if you call any of the Get..PropertiesAsync methods apart from RetrievePropertiesAsync simultaneously on the same StorageFile instance, they will fail, even though they are read-only as far as I can see so have no reason to. Secondly, fetching System.Size and System.DateModified using RetrievePropertiesAsync after the first time does not return an up-to-date result. Thirdly, there can be issues with scenario 9, which I’ll mention later. Finally, using SorageFile.GetBasicPropertiesAsync to get the file modified data is incredibly slow compared to .Net. in .Net it takes about 0.05ms per file. In UWP, it takes about 5ms per file, which is 100 times slower, and also adds about an extra 60kB per file (if you keep the StorageFile), compared to about 0.05kB per file in .Net.

Scenario 4

Currently in UWP if a user drags and drop a file/folder from File Explorer into the app, the file/folder is read-only, so can’t be edited, unless the app has access to the file path via some other means.

Scenario 5

This duplicate button is not possible in UWP if the user opened the file individually or by double-clicking in File Explorer, unless the file is in the Pictures library (assuming the Pictures library capability). Worse, the app cannot even open the save file dialog to the current folder, since there is no way to set the starting folder of the save file dialog.

Scenario 6

This is a similar problem to scenario 5. If the user opened the file individually this is impossible, unless the file is in the Pictures library.

Scenario 7

The only general way to save access to files is via the FutureAccessList. However, this is beset with problems. It has quite a small limit of 1000 files which is easily exceeded. The developer has to write code to handle the case when this happens, which is hampered by the fact that any of the APIs surround the FutureAccessList are quite slow, typically taking several milliseconds, and hence such a function has to be carefully debounced so as not to impact performance of the app. Furthermore, the app has to maintain a list of the tokens it is using and ensure this is in sync with the list, which is awkward. It would be easier if the list could be accessed via file path, with a use count. This could be stored in the metadata currently but for some reason in order to obtain the metadata for a token you have to enumerate the whole list. I find that whenever I add a feature to my app that requires this list, it takes me a couple of days to get the logic right, and even then it’s full of annoying compromises. Compare this with a .Net app where you can simply forget about the whole thing and spend zero days on it.

Scenario 8

Due to the fact that the slideshow list would be stored in an external file which could then be deleted without the knowledge of the app, and also because this could exceed 1000 files, the FutureAccessList cannot be used for this purpose, so it is impossible unless the files are in the Pictures library.

Scenario 9

There are currently various issues with the neighboring files query, which I believe is related to the way it uses the file system indexer. Firstly, it is quite common for the file system indexer to stop working properly. On my old PC this happened after every major Windows update. In this case, the neighboring files query would contain no files or be missing some files. I found disabling and re-enabling file content indexing fixed the problem. Also, if you use RetrievePropertiesAsync to retrieve properties for files loaded in this way, you won’t be able to access some properties. Also, some properties like title will be limited to a certain number of characters (e.g. around 250 for System.Title). Another problem is that the allowed file extensions does not always include all media files .e.g. for some reason .webm video files are not included in the neighboring files list (as of 1909).

A few suggestions for solutions

Scenarios 1-3 are performance-related, and in the case of scenario 2 the APIs are simply incomplete for UWP and need to be completed. The performance issues should be covered by the suggestions in your proposal. In my opinion in this case it is important that all APIs involving StorageFile/Folder have versions which accept the lightweight alternative. This should include an alternative for RetrievePropertiesAsync. Also, once you have permission to access a file for the current app session, you should be able to access it via any means, unlike currently where StorageFile.GetFileFromPathAsync(myAlreadySelectedStorageFile.Path) will fail. (Memory usage should be taken into account if it becomes a problem, but hopefully it shouldn’t).

Scenario 4 is a simple matter of change the default behavior to read-write, and I think most people agree with this.

Scenario 9 could be covert by enabling the app to indicate that it doesn’t want the neighboring files query to use the file system indexer, or something like that. It could also be solved by just allowing access to the folder, as long as it was possible to get the current sort order. I know this is an increase in access rights, but it would also solve some other problems which is why I mention it. The issue of some media files being omitted could be solved by Windows having a list of such allowed extensions that could be updated independently of a feature update so we do not have to wait a year for such a simple change (assuming the list was then updated to add files such as .webm).

Scenarios 5-8 are more tricky. Basically all of these require more permissive file access or additional capabilities. I have sketched out a few ideas, some of which you might consider outrageous, but I think this is a very important area that needs exploring.

• Suggestion 1 – If the user double-clicks on a file in File Explorer which launches the app, allow the app full access to that folder. This would solve some of the issues with 5 and 7 and 9 (it is very common for the user to open files in this way). Alternatively, if technically possible, allow read-write access to that folder including creating new files but only for Pictures file extensions (if the app has the Pictures library capability).
• Suggestion 2 – A bit out on a limb. Add an ‘exendedPicturesLibrary’ (and videos etc) capability which would allow read-write access to all standard folders for the users, but would be limited to picture files. This would solve most of the issues for my particular app but perhaps is not so generally applicable. Also, the app may save some settings files etc in a non-image format, so maybe it should include other file types specific to the app – it could even be required that these extensions should be specially named (e.g. start with x- or something, to ensure the app does not gain access to other files).
• Suggestion 3 – Allow the app to prompt for access to files on a file-by file basis. The problem with broadFileSystemAccess, in my opinion, is that at the moment the app prompts for it, the user will assume the app is trying to access all their files, rather than just the ones they want it to, and that will put them off. It makes sense for the app for be able to tell the system ‘I want to access this list of file and folder paths’ and then the system can prompt the user (if the app does not already have access), and say ‘Do you want to give the app access to these files’. This dialog should ensure the list is displayed in an easy to read way that is not overwhelming even if the app prompts for multiple files. There could also be a checkbox ‘Always allow this app to access all my files’ which would be the same as broad file system access, except that now it would not seem like the app was unjustifiably tying to access all files. Ideally also the system should then automatically remember that the app was given access to those files so it does not have to prompt the next time. It might be difficult to maintain a large list however, so the system could simply maintain the most recent few thousand or something. This would address most of the file system issues and remove a lot of the need for the FutureAccessList, although it would still be necessary to add files to this to guarantee future access. It should not be necessary for the user to go into settings and restart the app to achieve any of this.
• Suggestion 4 – I’m not really sure this would work. In order to support scenarios like 8, I had this idea that once the app has access to a file, it can ask the system for a future access token, and the file access permission is then stored in the token itself. For example, this could be achieved with signing using the path of the storage item, the app’s identity and a secret system key, and maybe it could include a date stamp as well. However, it might be difficult to revoke access in this case. I guess it would be best used in conjunction with suggestion 3.
• Suggestion 5 – This is a really simple suggestion that would cost noting to implement but would make using broadFileSystemAccess where needed much less of a concern for me as a developer. In the app’s listing on the Store, if you use a capability like broadFileSystemAccess, then it says something like ‘This app can access all your files’. This makes it sound like by installing the app, it can access all your files. However, it requires an additional prompt. So why not say ‘This app can access all your files if you agree to an additional dialog’ or something like that? That would be must less off-putting.

Finally, my thoughts on periodic warnings for users about apps using capabilities - personally I find this kind of notification annoying, so I would be against it. Better to make it easy for a user to investigate this, without giving them an intrusive notification. Maybe it could go so far as something like the using your location icon in the task bar, although even that is potentially annoying if there are too many.

Thanks for your patience in reading that!

[eklipse2k8]

Why not create in AppData folder itself?

Sorry @soumyamahunt, but this is a limitation. What if the user wants to copy the file to a USB drive or put it in their OneDrive storage? I'm forced to build a file manager UI so they can "export" the document, when the file system provides all of this functionality already.

AppData is for caches and app metadata, but not user documents that they want to access, move around or do as they please with. AppData should be considered off limits to users mucking around with anyway.

[soumyamahunt]

Sorry @soumyamahunt, but this is a limitation. What if the user wants to copy the file to a USB drive or put it in their OneDrive storage? I'm forced to build a file manager UI so they can "export" the document, when the file system provides all of this functionality already.

AppData is for caches and app metadata, but not user documents that they want to access, move around or do as they please with. AppData should be considered off limits to users mucking around with anyway.

You don't have to build a file manager UI, user can go to the AppData folder of your application and copy it. If you are insistent in creating folder/file anywhere other than that you can ask user to create a folder with file picker and then add the folder/file in FutureAccessList. Automatically creating folder anywhere is huge no for me.

[MarkIvanDev]

These are the limitations I see in UWP file-access APIs besides the ones that are pointed out above

1. The broadFileSystemAccess capability limitations

• StorageLibrary APIs. You still cannot get the user's libraries when calling StorageLibrary.GetLibraryAsync even though technically, Libraries are part of the file system that is available to the user. Since the docs are saying that the app should not declare the capabilities for accessing the libraries (documentsLibrary, musicLibrary, picturesLibrary, videosLibrary) if it already uses the broadFileSystemAccess capability, declaring broad file system access should also grant access to the user's libraries through the StorageLibrary APIs

• Removable Storage Access. Although accessing removable storage requires another capability (removableStorage), broadFileSystemAccess should help lessen the restrictions. As of now, the app still needs to declare the file type associations just to access those specific file types in the user's removable storage devices.

• UNC folders. As with the previous point, accessing UNC folders needs a combination of capabilities (privateNetworkClientServer, internetClientServer, enterpriseAuthentication) to access them. Combining broadFileSystemAccess should also remove the file type restriction.

2. Low-level ...FromApp API although helps with performance is still severely limited

• Copy, Delete, Replace, Move APIs does not expose a way to monitor progress. CopyFileEx, although available to UWP, still only accepts a file path to work. If only it has a variant that accepts a file handle, a workaround can be made. Or better yet, expose a FromApp api that can monitor progress.

---

To answer the Open Questions

1. Should UWPs be given access to the IShell* family of APIs?

Since these APIs are not universally available, I think UWP should not be given access to these. It would add an added complexity of checking whether the device that the app runs on supports these or not. IMHO, the Windows.Storage is already a major improvement over the Storage APIs available in previous implementations. It just needs to be updated to have more APIs that will better align with the design of UWP

2. Should Reunion warn the user about apps using this capability?

UWP apps already has a robust way of requesting capabilities (i.e. Microphone, Camera) from the user that does not require a restart. I think this design should be implemented when requesting access for ANY capability. As it stands, changing access to the File System requires an app restart that users can mistake for a crash and does not provide a good look for the apps that use these kind of capabilities.

[eklipse2k8]

You don't have to build a file manager UI, user can go to the AppData folder of your application and copy it. If you are insistent in creating folder/file anywhere other than that you can ask user to create a folder with file picker and then add the folder/file in FutureAccessList. Automatically creating folder anywhere is huge no for me.

The problem is, AppData is hidden and I don't believe it's even easy to traverse because the final document would be behind your package folder that could be anything. If anything, it's there, but it's very unfriendly to users. That's why we're writing our software right?

The scenario I see is this,

• App creates a temporary document in AppData until the user chooses to save it somewhere on their system. You shouldn't need any kind of broadFileSystem access to do that. If they have write permission in the location of their choice, and they tell the app to put the document there, then that's it!
• Now, SQLite likes to create a journaling file, and potentially a backup file incase of a bad save. The SQLite that's bundled with Windows winsqlite3.dll will want to write those files next to the actual database file.
• This is the same behavior of Microsoft Office apps. They all get to save backups right next to the actual document file.

This is not about letting an app write junk all over the place, this is about a document based application creating a good user experience.

[lukeblevins]

Hi. I'm the developer of Files (UWP).

Our primary pain points with the current experience of integrating the native Win32 file-access APIs into a UWP app were the things the Win32 File APIs lack (at least for UWP apps): 1.) A simpler alternative to ReadDirectoryChangesW() 2.) Fast replacement for WinRT GetThumbailAsync() 3.) Viewing hidden items and shortcuts

Also, if we're planning on keeping the file/folder pickers, they should really have a modern UI to look familiar in our WinUI apps.

[jonwis]

@duke7553 - thank you!

Simpler alternative to ReadDirectoryChangesW

Have you tried out StorageFolder.TryGetChangeTracker ?

Hidden items and shortcuts

Does FindFirstFileExFromApp help here?

[lukeblevins]

@jonwis My app, Files, previously included a purely WinRT file-access implementation. (Meaning, we used StorageFile and StorageFolder) It was highly optimized at that because we retrieved storage items in batches, and prefetched properties for partial storage items, yet users complained this approach was slower than they were familiar with. As of January, we now use the FindFirstFileExFromApp() to load the items themselves, but we do still rely on StorageFile/StorageFolder to return thumbnails and other properties for the items in a directory.

Regarding StorageFolder.TryGetChangeTracker, my understanding of it was that it works only when the user is accessing libraries, so something like ReadDirectoryChangesW was needed here. I implemented it in Files (UWP) with the last update, v0.10.1 .

[lukeblevins]

Sure, we can access items with the hidden attribute this way, but they are invisible to all of the WinRT file operation methods (i.e. Cut, Copy, Delete)

P.S. I will admit my case is kind of extreme since I built a file explorer. :)

[jonwis]

@duke7553 - There's a whole raft of ...FromApp APIs like CopyFileFromApp, DeleteFileFromApp, and more. I get not wanting to re-implement StorageFile yourself from these primitives, of course. :)

Hey @smaillet-ms - does StorageFolder's change tracker work on any arbitrary storagefolder, or only on libraries?

[michael-hawker]

There's also some weird restrictions on existing capabilities, like access to the Downloads folder only lets you create a sub-directory, see doc issue here. If you're trying to make an alternate browser, this is a weird experience for anyone using any other browser. Maybe there needs to be a AllDownloads capability that's slightly broader?

(Similar to @benstevens48 Scenario 8 above), I think there's a challenge in having any sort of project system at all (and loading these types of file sets from any other source). If a file references the absolute or relative path of another file (or files think '*' wildcard in a csproj); you can't open those references in a UWP today unless you ask the user to select a parent folder that has access to all references. It does seem like a hard challenge to solve though with figuring out an access pattern for detecting and allowing this without needing the full BroadFileAccess capability though... I mean the fallback in UWP is have a folder based project system; however, that doesn't track well with consuming any external formats to your app.

[kmgallahan]

A StorageFolder.MoveAsync method would be highly useful. The current recommendation is... 🤷‍♂️

There is not currently a "MoveAsync" or similar method. One simple implementation of moving a folder might be to get the desired folder, copy it to the desired location, and then delete the original folder.

[soumyamahunt]

@jonwis ...FromApp apis doesn't work reliably either. I tried to implement GetFileAttributesExFromApp to check whether a file is read only when user opens file via picker or file activation, but this doesn't work when app opens the file for the first time (subsequent query works).

[shresthasource]

@duke7553 - There's a whole raft of ...FromApp APIs like CopyFileFromApp, DeleteFileFromApp, and more. I get not wanting to re-implement StorageFile yourself from these primitives, of course. :)

Hey @smaillet-ms - does StorageFolder's change tracker work on any arbitrary storagefolder, or only on libraries?

From what I remember, it is supposed to work on any folder. However, my last attempt to implement it failed. The issue I faced was that it did not trigger the event right after the changes. I remember reading something along the lines of ...it will wait till something else (don't remember what exactly) before the event is triggered.

[shresthasource]

P.S. I will admit my case is kind of extreme since I built a file explorer. :)

Considering WIndows Store has a dedicated subsection titled "File Managers", I would argue, this is not an uncommon scenario.

[ptorr-msft]

There's also some weird restrictions on existing capabilities, like access to the Downloads folder only lets you create a sub-directory, see doc issue here.

This is a missing feature (downloads folder capability) -- please add another issue for it. Thanks!

[brabebhin]

As a developer of media applications, I find the existing file system access capabilities for UWP are horrendous, and are probably the biggest reason the platform not actually taking off at all. Any kind of serious application will require decent levels of file system access, and decent performance for those APIs. The broad file system access capability does solve the problem to some extent, but this does not address the issue of performance.

Not only are the search / access / property APIs stupidly slow, but they are complimented by equally slow read & write stream APIs. It is unacceptable to wait for several seconds to open and parse 25 files, just because of some unknown limitation imposed at API design time. What is the point in having those APIs as async if the user has to wait ridiculous amounts of time just to perform basic operations? Better have the UI freeze for 1 second than wait 10 seconds staring at a loading bar.

An additional problem is the assumption that every use will put their music / video / pictures / documents in their respective libraries. Truth to be told, I have never seen a single user who actually follows the library pattern completely. Not a single one. Broad file system access should in no way be a restricted capability: it should be top and front. Simply restrict access to obvious danger places like C>Windows and other system folders, and you're golden. Any app that passes through the store should be considered safe, i don't see the point in adding additional security impediments.

These APIs need to go back to the drawing board and be designed for speed and scalability. If Storage.* APIs get deleted tomorrow, I will go out and dance on the recycle bin. They are the biggest reason no productivity app is currently following the UWP model.

[jtorjo]

As a developer of media applications, I find the existing file system access capabilities for UWP are horrendous, and are probably the biggest impediment on the platform not actually taking off at all.

This is definitely the biggest impediment. The whole "sandboxed" concept is a half baked idea of someone that probaly listened to some buzzwords at some point.

But as a developer that reimplemented pretty much all of his app in UWP, there's sooo many issues that will bring you to the "curse master" level.

An additional problem is the assumption that every use will put their music / video / pictures / documents in their respective libraries.

I've been saying the same thing for months. Having said all that, while I do hope this will be fixed, I'm almost 100% sure this will be dragged for years before MS will do anything about it (if they will do something ever). The whole "sandbox" is soooo broken, that no one at MS seeems to want to work on it. Just look at compile times - we've been promised they would be improved for close to a year. Promised again they would start working on compile times 2 months ago, but nothing to show for. But nothing is happening - just promises and delays.

[soumyamahunt]

Also, adding System.IO.File and other System.IO APIs support to UWP should be considered. While back System.IO.MemoryMappedFiles (may be System.IO.Pipes too) support was added by using ...FromApp methods explicitly for UWP, similar approach can be taken for other APIs. This will help dotnet developers bringing their existing projects to UWP without making any significant effort in learning new APIs.

[ZodmanPerth]

Hi folks. I'm the lead developer for the Drawboard PDF UWP app. Our application contains a file explorer that allows the user to pick a root folder using the filepicker and navigate up and down folders within that root. We surface a filtered view of the filesystem; only folders and (mainly) PDF files, for selection for loading.

I'm currently rewriting this section of our app and have discovered the pain of UWP filesystem APIs. I've tried various methods to improve the current experience, and have explored a number of approaches to the problem. At the end of the day, the only stable method is to continue to use GetItemsAsync which is the slowest file access I've ever seen (so I agree with others about performance, I average ~4 seconds for a folder of 1000 items, ~45 seconds for 10000 items).

The UWP APIs are terribly complicated and unfriendly. After spending a lot of time with them and believe I understand how they are intended to be used, I have still failed to get the features promising better performance to work. I don't believe they actually work at all. Further, I don't believe the exposing the file indexer is worth the maintenance cost for Microsoft; it's probably better the Reunion API monitor and decide whether or not it should use the indexer or not as an internal implementation detail.

In the end I've had to create my own folder wrapper class that performs two functions:

• Enumerate and cache the (filtered) contents of a folder the UWP app has access to,
• Monitor the Changed event for changes to the folder. This kicks off a background process to re-gather all (filtered) items in the folder using GetItemsAsync, then compare them to the cached list of items, to determine what the changes are. I surface the results through my own event using a lightweight item class. I use locks to ensure I only queue one of these processes at a time, no matter how many times the Changed fires.

This allows me to wrap up all the logic I need to process folders at the snail's pace the API provides, while keeping the UI somewhat responsive.

ASIDE: I'm working on a standalone prototype, which I am planning to share publicly when I've finished my investigation. Please let me know it would be helpful for this discussion and I'll post it here when it's done.

I've tried using FindFirstFileExFromApp which is blindingly fast in comparison, and more than acceptable for my needs. However it fails with Win32Exceptions on some folders (C:\ for example), meaning it isn't robust enough to rely on. I have abandoned it as an option.

What I expect from from a v1 Reunion API for filesystem access is:

• The ability to get the id+name+type of files/folders within a specified folder the UWP app has rights to, given filter options. I don't care if it's an enumerator, collection, or stream of simple objects, I just want it to be very fast. It should be asynchronous.
• A callback/event to notify me when the contents of a folder has changed, given filter options. The args should include the id+name+type of files/folders that have been added/renamed/removed, and grouped as such. It could also provide some id to pass to an async function to get results and allow the callback/event to complete.
• A function (or functions) to call with the id of a file/folder to obtain more details. This could be file properties, image data etc.

I feel that would cover the needs of many UWP apps; perhaps many non-UWP apps as well.

Having spent significant time analysing the UWP tools for working with the containerised filesystem model (StorageItem, FutureAccessList, AppData folders etc) I have to say I think the model has a lot of merit. The core issue seems to be that engineers have trouble transitioning from a culture where they can access anything in the filesystem to the containerised model. It's not an intuitive leap, and despite numerous articles and documentation I think Microsoft have failed to onboard engineers into right mindset to work with the containerised model.

I don't have an opinion on BroadFileAccess as at present; our app simply hasn't had the need for it.

I agree with what others have said about the upper limit of the FutureAccessList being too small. It probably made sense when UWP was in a much smaller container and was only capable of producing simple apps. If we want to allow UWP the capability to create serious tools then performant filesystem access is key, whether it's using containerised permission or broad access.

With WinUI 3 on the verge of being delivered it may be that engineers will gravitate towards Win32/Winforms/WPF apps using WinUI 3, and UWP would remain only able to successfully compete with a subset of apps.

I feel resolving filesystem access with a Reunion API will mostly flatten the playfield of what can be achieved on UWP and give it the competitive support it needs and deserves.

[brabebhin]

Having spent significant time analysing the UWP tools for working with the containerised filesystem model (StorageItem, FutureAccessList, AppData folders etc) I have to say I think the model has a lot of merit. The core issue seems to be that engineers have trouble transitioning from a culture where they can access anything in the filesystem to the containerised model. It's not an intuitive leap, and despite numerous articles and documentation I think Microsoft have failed to onboard engineers into right mindset to work with the containerised model.

The container itself is fine. The fact that it takes 100+ms to open a file on a SSD is not (while arguably using native code, when a managed application can achieve 100 times better performance). Other OSes managed to get containers and file access at a decent speed, yet somehow UWP failed to do that.

The StorageFile idea has a lot of merit, especially since it can also abstract away a lot of problems (like files on network drives or streamed files), but this has come at the expense of performance. Microsoft needs to address the performance problem, otherwise nobody will bother with the container.

With WinUI 3 on the verge of being delivered it may be that engineers will gravitate towards Win32/Winforms/WPF apps using WinUI 3, and UWP would remain only able to successfully compete with a subset of apps.

I personally plan to migrate my UWP app to a desktop app + winUI just to get decent file performance and implement some features which are simply not feasible with the current performance of the Storage API. I can continue using the Window.Storage API for most of the application, but performance critical sections need better APIs

[ZodmanPerth]

I've completed my prototype and have added it to a public repository at uwpfileexplorer. It serves as a case study into what UWP engineers have to do to achieve performant file access in the context of a file explorer.

The performance is good enough for an in-app file explorer, but not good enough for filesystem-crawling file processing utilities. The code includes a PowerShell script to create large folders that can be used to highlight performance issues with the API. In particular, note how long it takes to refresh a large folder when changes are made to the folder outside of the app.

It also highlights the inadequacies of the UWP filesystem APIs, and how much work needs to be done to wrap them into something useful.

[shresthasource]

To my experience, UWP apps also do not get access to various items such as URL shortcuts or App shortcuts, and hidden files/folders via StorageSpace API. Access to such items would help.

[DrusTheAxe]

@shresthasource > UWP apps also do not get access to various items such as URL shortcuts or App shortcuts

What sort of access are you interested in?

What

• create shortcuts
• update shortcuts
• delete shortcuts
• other?

Who

• shortcuts in user's profile
• shortcuts for all users
• other?

Where

• shortcuts on Start Menu
• shortcuts on desktop
• shortcuts on tray
• other?

There's gaps between shortcuts and tiles. Some of these are old features that may not make much sense these days (Run as Windows95 etc compatibility options?) and some are functional e.g. Secondary Tiles have no RunAsAdministrator property. Do you have specific gaps you'd find helpful?

Is it the functional behavior you're interested in or the physical shortcut itself i.e. a .LNK file?

There's also some gaps between 'primary' and secondary tiles. The latter are created/manipulated via runtime APIs, the former are automagically created when a package containing <Application>s is registered for a user and deleted when the package is removed, an <Application> only gets 1 primary tile e.g. you can't create a tile to launch an app and another tile with uap10:Parameters="--safemode" etc to create a similar tile with different arguments.

Are there particular gaps you're interested in?

On the primary/secondary tile front, would it be helpful if you could specify tiles in appxmanifest.xml e.g.

<Application Id="App" Executeable="catvideos.exe" ...>
    ...
   <Extensions>
        <Extension Category="windows.tile" uap10:Parameters="--type=kittens" uap10:CurrentDirectoryPath="$(knownfolders.videolibrary)\Cats">
            <Tile Logo="kitty.png">

to produce 2 tiles. The 'primary' today for the <Application> runs "...pkgdir...catvideos.exe" with currentdirectory=pkgdir, and a 2nd tile that runs "...pkgdir...\catvideos.exce --type=kittens" with currentdirectory=c:\users\username\videos\Cats

[shresthasource]

This should clarify most of the answers:

Use Case: A user browses to the Desktop Folder using a UWP file manager and do not see the Application shortcuts that launches Applications such as Edge Browser. A URL shortcut is created from Context Menu -> New -> Shortcut and puts a URL or other paths.

Issue: A user does not see such shortcuts in a folder and might end up deleing the folder thinking it's empty. But some URL shortcuts or other such items might have useful/important contents. A user blames developer for a file manager that did not show items that were there.

What sort of access are you interested in?

What

• create shortcuts
• update shortcuts
• delete shortcuts
• other?

All are relevant/useful. I tend to think the StorageSpace API does offer something to create a link/shortcut to a different folder (just like copy/cut/paste, there was something along the line of link, haven't tried it though). At very least such links/shortcuts should be visible. Best case scenario, they should have similar methods as other storageitems. Also, I am not sure if storage item type of neither folder nor file is actually implemented or usable for anything as of now.

Who

• shortcuts in user's profile
• shortcuts for all users
• other?

Again, I am looking mostly at from a location where shortcuts are rather than build in shortcuts to something. However, another issue I had is accessing various folders via StorageSpace. As of now, we can access Music, Videos, Documents, and Pictures. However, places such as Desktop and Downloads are not accessible from the API. So, best a developer can do is to try to do some magic with user name and generate a default path (e.g. C:\Users\username). But, such workaround wouldn't work for all the cases.

Where

• shortcuts on Start Menu
• shortcuts on desktop
• shortcuts on tray
• other?

Shortcut items wherever they are located (in StorageSpace). Desktop definitely. StartMenu maybe. Tray, probably not important unless tray shortcuts are also stored somewhere. Others -> Wherever user decides to create shortcuts to files/folders/applications.

There's gaps between shortcuts and tiles. Some of these are old features that may not make much sense these days (Run as Windows95 etc compatibility options?) and some are functional e.g. Secondary Tiles have no RunAsAdministrator property. Do you have specific gaps you'd find helpful?

Probably not much relevant answer in my context here.

Is it the functional behavior you're interested in or the physical shortcut itself i.e. a .LNK file?

Not entirely sure about the question, but I tend to think I am talking about physical shortcuts.

[benstevens48]

@DrusTheAxe I think at a minimum the physical shortcut itself (.lnk file) should be listed when listing the items in its parent folder. I suppose this could be optional but I don't see a reason not to include it. Maybe there could be an option to also resolve shortcuts when listing files in a folder, but this wouldn't work if just file names were returned and I wouldn't want it to impact too much on performance, so I don't think this is essential in a low-level API. There also needs to be a way to get the target file path of the shortcut. Should permission to access the target file automatically be granted? I'm not sure, but it would be difficult to do anything useful if not.

[Qws]

People don't want apps to dump files on random system locations behind the scene.

Only Devs think it's nice to dump their files wherever they want, it's a selfish use case and cause horrible Windows user experience (maybe not immediately, but it will in the long run).

Many people find Windows a confusing system, compared to iOS and even Android. Downloading and installing legacy Win32 app is even a more scarier and confusing process, but on iOS and even Android it's a fun process.

The reason why UWP is considered a fail? Let's just look how you get UWP apps installed on the PC in the first place.

The Windows Store itself a horrible app, look at the Windows Store review, people calling out how terrible the app is. The Windows Store is slow, freezing, horribly designed has boring UI for years. The whole Store needs to be revamped. It feels like it's a sluggish Surface advertisement board + some games.

Please don't blame UWP when the place where you get UWP apps is barely functional.

Another problem is, building an UWP app is still confusing, you don't need to know XAML and C#. Let people know how to make UWP apps with other tech stack, and further broaden the ways to build UWP apps.

UWP has limitation, sure, some of them are problematic, sure. But the file access is not one of them, it's one of the better limitations. Once you break this limitation, it will be abused... then you get back to the old Win32 legacy chaotic software and worst case literally spyware/malware. You just end up walking circles.

Keep UWP tightly locked and safe for the users to use... broaden the ways to build UWP apps, and important one, fix the store.