Learn from Mac OS X how to suck less at system integration

[probonopd]

Current situation

Desktop Linux, mostly through freedesktop.org specifications, currently provides only static and inflexible ways to integrate applications with the system. It is assumed that applications are installed into fixed locations into the system. The system is not really built for dynamically ever-changing per-user applications. It also assumes that every application is only present in one version.

Deficiency of the status quo: Dealing with applications that come and go on the fly

This creates challenges for situations in which applications "come and go", e.g., when an external partition (or, say an AppImage, a Snap, a file server share...) is mounted that contains applications. One has to manually copy around desktop files, icon files to locations like $HOME/.local/icons/hicolor/ with varying subdirectories that may or may not exist and be recognized depending on whether the subdirectories were already present when the desktop environment was launched, MIME files to locations like $HOME/.local/share/mime/packages, and then trigger runs of tools like update-desktop-database and update-mime-database, which is a real pain:

• https://github.com/AppImage/appimaged/issues/34
• https://github.com/AppImage/AppImageKit/issues/45
• https://github.com/AppImage/appimaged/issues/46

Some desktop environments then also need to update their own databases and caches, e.g., Sycoca on KDE Plasma. Whether or whether not this happens automatically and on which occasion has been a complicated mess since essentially forever.

If too many applications come and go at the same time, the system can come to a grinding halt due to the various databases and caches being updated a lot of times in short order:

• https://github.com/AppImage/appimaged/issues/12

Once the external partition containing applications is unmounted, one has to figure out which of the files that were previously copied around need to be deleted, and has to manually trigger various update-.... commands again.

This reminds one of the state of networking before WLAN, where everything was configured statically under the assumption that there is a static networking setup that is erected when the machine is installed. Once people had mobile devices and roamed around, NetworkManager was introduced and made network management dynamic, assuming that network connections would come and go. The same has not happened on "Desktop Linux" for applictions yet, but needs to happen.

Deficiency with the status quo: Multiple versions of applications

Many applications have different icons for different versions of the same application. Firefox, for example, has constantly been evolving their icon.

The current XDG standard is entirely ignorant of this, assuming all Firefox instances on the system are sharing the same Firefox icon. macOS does a much better job because icons are stored inside each application, rather than at one central place in the filesystem.

Similarly, when a HTML file is about to be opened, then there is no clear concept which version of Firefox will get to open it under the current XDG regime.

Current workarounds in AppImage

As an AppImage-specific workaround for the inflexible, static way of how applications are integrated with the system, we have (at least) 4 different, inclomplete ways to integrate AppImages with the system (which still suffer from the shortcomings of the status quo described above but try to work around them):

• A shell script that is contained inside some AppImages and that asks the user whether the AppImage should be integrated when an AppImage is executed for the first time. This is deprecated and no longer recommended nor supported
• The optional appimaged daemon which needs to be installed in the system. This will watch certain directories (most notably, /Applications and $HOME/Applications) for AppImages, and will integrate them into the system automatically. By design, it will never move anything anywhere and will never ask the user any questions. (Compared to the Mac, this solution is not yet ideal because it does not integrate AppImages stored in random locations yet and watching the entire system with inotify would be computationally too expensive.)
• The third-party AppImageLauncher which needs to be installed in the system (but not in addition to appimaged). It will ask the user with a popup to move every AppImage into $HOME/Application whenever an AppImage is launched that is not in that location yet. (Personally I find this too intrusive as I keep my AppImage in non-standard locations but apparently some users like this.)
• The AppImage desktop integration built into Nitrux OS.

Current workarounds in snapd

(to be written)

Current workarounds in Flatpak

(to be written)

What needs to be done

To come up with a much better (and more seamless) system than what we have now, we should

• First, propose to standardize (e.g., via freedesktop.org) some infrastructure for the Linux Desktop that is not AppImage specific but does cater to the AppImage use case (as it would for Snappy, Flatpak, traditional packages, applications installed via .tar.gz or .zip files, AppDirs and that would have to be implemented by desktop environments like GNOME, KDE Plasma, XFCE, Elementary OS, and the like and/or existing system-wide infrastructure such as systemd. We should do this in an open discussion involving the greater Desktop Linux community, including desktop environment projects, application authors, etc.
• Second, as a fallback, come up with something that does not require the collaboration of the desktop environments and/or systemd (but possibly hook into them using the existing interfaces they provide as of today, such as dbus, thumbnailers, and such)

The objective should be to get proper dynamic desktop integration for applications (like on the Mac) into the default installation of distributions. This pretty much excludes any solution that is AppImage-specific.

Learning from the Mac

Classic Mac OS, the various NeXT systems, and Mac OS X are still unparalleled in terms of ease-of-use when it comes to integrating applications into the system by mere drag-and-drop. It just seems to work phenomenally well, without the user having to fiddle around with desktop files, icon files, and the like.

I have written about this before.

It is about time that Desktop Linux gets the same level of usability, almost two decades later.

We have been attempting to re-create a similar level of simplicity with appimaged, but frankly, so far we have not reached the same level of "it just works" (and performance) yet. Possibly some changes are required on deeper levels of the system and the file managers.

Almost two decades ago, then Apple User Experience Architect @arnog gave insight into some of the concepts of how Mac OS X does what we would call "system integration" today in the WWDC 2000 Session 144, Application Packaging and Document Typing. The man must know, as he has been the "Architect and lead engineer of a complete rewrite of the Finder for Mac OS X", and according to his CV "Participated in weekly design meetings with Steve Jobs (and had a good time doing it)".

Applications on Mac OS X live in application directories, a subclass of what they interchangeably referred to as "bundles" and "packages" back then. Our equivalent would be AppDir. Those are often shipped inside dmg disk images, our rough equivalent of which would be AppImages.

How the system recognizes an application bundle

1. By a filesystem attribute ("bundle bit"). It is explicitly mentioned that this is not robust across different filesystems. Our equivalent would probably be something like an extended attribute, but like on the Mac, that would fall flat on non-native filesystems. Hence the following additional measures also make the system recognize an application bundle
2. A directory that ends in .app. Our equivalent would be .AppDir. It is explicitly mentioned that you are not forced to use that extension, since there is also a third option
3. A file called PkgInfo in the top-level diretory of the bundle. It is explicitly mentioned that this holds type and creator information redundant what is stored elsewhere, for performance reasons

What triggers system integration

1. When you put an application into one of several "well-known" Applications directories (most likely, /Applications, /Network/Applications, /Users/$USER/Applications. Our equivalent would be to watch /Applications, $HOME/Applications, and possibly /Applications on each newly mounted partition using inotify. Note that the Downloads directory is not part of those. It is explicitly mentioned that while applications get integrated into the system immediately when you put them there, you can also put them into other locations where they will be picked up by the following additional measures
2. When you launch an application. Our equivalent would be to catch each application launch, possibly using binfmt, and silently integrate the application into the system when it is launched. It is mentioned that once they integrate an application, they also check the directory containing that application for other applications (but most likely not the directory's subdirectories to prevent situations like these).
3. When you "look at" an application. Presumably they mean viewing a directory in the Finder. Our equivalent would be a thumbnailer (normally used to render preview icons) that gets executed whenever one "looks at" a directory.

Registers an application, designated by URL, in the Launch Services database.

According to (Singh, 2006) system integration is also triggered

5. When the system is booted (What exactly happens then?)
6. When a user logs in (What exactly happens then?)
7. When the Finder locates a new application, such as on a newly mounted disk image - say, one that has been downloaded from the Internet (What exactly happens then?)

Additionally,

8. When an application explicitly triggers integration. An application, such as a web browser downloading an application or an archive manager unpacking an application, can call LSRegisterURL(_:_:). Note that one cannot rely on applications doing this. Our equivalent would be to offer a dbus message that when called would trigger system integration.

What happens when you double-click

• Each application has a type and a creator code (four characters each - this concept is inherited from Classic Mac OS)
• The system keeps a database, built from metadata that travels inside each AppDir's Info.plist file, containing, among other things, the file types the application can handle and the application version. Our equivalent would roughly be the desktop file amended with the X-AppImage-Version field. (Possibly we should settle on a YAML file that would combine what is in .desktop and AppStream metainfo files today.) The database is populated by the triggers described above
• When a file ("document" in Apple terminology) is double-clicked, the database is queried for the newest version of an application that can handle this type of file
• It is possible to tell the system that certain files shall be opened by certain applications (e.g., one jpg in Photoshop and another jpg in Dreamweaver)
• When you double-click a document for which there is no known application, then you are presented with a file chooser window that will select you the application to open the file with
• No icons are copied around. Presumably the system queries the database to display application and document icons

Note that the only time the system asks the user something is when it doesn't know which application to open a file with, which rarely happens in practice, given the combination of the three triggers described above.

The database

The Launch Services registration database can be dumped like this:

(Singh, 2006)

Additional information

A year later, the concepts were covered again in WWDC 2001 Session 114, Application Packaging and Document Binding.

"Document Binding is what associates, on the fly, documents in the file system with the applications that can handle them, it's what makes double click work".

In Desktop Linux, we are not doing this "on the fly" yet. We should!

Bindings Database

Note that applciations also can make use of "packages", that is folders displayed as files.

The relevant Property List keys start with CFBundle (it's all defined by Core Foundation).

It gets populated by reading the Info.plist files that travel inside the applications (and other types of packages). Today, one would probably use YAML or JSON instead of XML.

Info.plist key	Corresponding desktop file key
CFBundleName	Name
CFBundleIdentifier	"ID" (the name of the desktop file sans the .desktop extension)
CFBundlePackageType=APPL	Type=Application
CFBundleSignature	n/a (the concept does not exist yet?)
CFBundleVersion	X-AppImage-Version
CFBundleIconFile	Icon=
CFBundleExecutable	Exec=
CFBundleDocumentTypes	MIME= keys plus corresponding mimetype xml files plus corresponding icon files. The Apple implementation is much cleaner
CFBundleURLTypes

Applications and documents have type and creator codes. The type of an application is always APPL. The creator code is what generally binds, e.g., text files which have the type TEXT (roughly equaling a MIME type) to one certain application that can handle text files (e.g., the SimpleText application). This concept seems to be missing in Desktop Linux entirely, which is why documents of the same MIME type never get opened by the application that created them, but rather by some "random" application that happens to also handle the same MIME type.

Do we need to invent/specify "MIME creators" in addition to MIME types to match what the Mac has had for a long time?

1. The user can set a certain document to open with a certain application manually
2. The system checks for the creator code of the document and looks up applicaion(s) with that type. In case there are multiple apps with that type, the latest version is taken. If there are multiple apps with the same version, the app with the latest modification date is taken
3. Extensions. User can say "open all html files in the browser"
4. OSType

"Launch Services is the engine that does document binding." "It maintains the Bindings Database." (Not to be confused by the deprecated Classic Mac OS "DesktopDB".) On Desktop Linux, xdg-open is similar to using Launch Services to open something ("Essentially the same as double-clicking"), and mimeinfo is similar to using Launch Services to get the binding (but not quite yet!).

TODO: Check LaunchServices.h header documentation and the Launch Services TechNote.

How do you ensure that you are in the Bindings Database?

The Finder (file manager) is solely responsible for updating the Bindings Database. Whenever the user logs in, the Finder looks in some well-known places auch as the main Applications folder and the user's application folder. It scans the sub-folders as well. Other applications are added to the database when launched from the Finder.

On Desktop Linux, file managers don't do this (yet). So we have to have some other part do this job, e.g, something like appimaged in combination with thumbnailers?

Application installers can send kAFSync AppleEvent. The Desktop Linux equivalent would be to send a dbus message to appimaged.

---

Linux needs a bindingsd to handle

• Applications
• Documents
• Icons
• MIME types
• URL schemes
• Creators and how they work together, DYNAMICALLY ON THE FLY.

Without the collaboration of file managers, we can abuse thumbnailers to do part of the job.

How can we "properly" achieve similar results on Desktop Linux?

• Desktop Linux so far seems to be lacking the equivalent of Launch Services. Could systemd be a place for it? I've talked about this at https://media.ccc.de/v/ASG2018-174-2018_desktop_linux_platform_issues
• Would the XDG specifications need to be changed or amended?

---

Singh, A. (2006). Mac OS X Internals. Boston: Addison Wesley Professional, pp. 828-830.

Possible stakeholders

System integration for applications and "document binding" a.k.a. file association is a community-wide topic and should be addressed as such. Possible stakeholders might be:

• Snappy. snapd currently needs to copy around files in order to achieve desktop integration. Snappy would benefit from a Launch Services equivalent
• Flatpak. It, too, currently needs to copy around files in order to achieve desktop integration. Flatpak would benefit from a Launch Services equivalent
• GNUstep. A free software implementation of the NeXT/Apple desktop for Unix-like operating systems. Especially we should check what their GWorkspace file manager does in terms of desktop integration, and whether there are .app bundles/AppDirs of some sort. http://www.gnustep.org/experience/screenshots/GWorkspace-screenshot.jpg has a promising-looking screenshot with /System/Applications. Does it work like the real deal, though?
• https://github.com/darlinghq/darling. A runtime for Mac apps on Linux. If the new proposed system could recognize existing Mac bundles, such apps could be integrated into the system properly
• File manager developers (e.g., Nautilus, Dolphin, etc.)
• Sycoca/kded developers. It is a per-user database for servicetypes, mimetypes and services) that gets updated when certain triggers occur. Might be turned into or replaced by the system described above. Currently seemingly limited to KDE (not cross-desktop like systemd, dbus and the like)
• Deepin Linux. A Linux distribution with some AppImage support in their file manager
• Nitrux OS. A Linux distribution that comes with AppImage integration out of the box
• Electron app authors. Electron apps are often produced by cross-platform developers who may or may not be Linux users, and often distribute applications for Windows, Linux, and the Mac e.g., in zip files. Maybe @develar as the maintainer of widely-used Electron tools would be interested to represent this group

What would be a suitable forum for driving this topic community-wide?

• The Linux Foundation. Probably not a good match since what we colloquially call the "Linux" desktop is not really limited to that particular kernel. But then, the Linux Foundation is also driving other projects that are not strictly limited to the kernel of the same name. However, they seem to focus on topics that are relevant to paying member corporations, which currently happen to be more interested in server-side technologies such as containers and container orchestration rather than the desktop. Yet still, Linus Torvalds claims the desktop is relevant to him... Looking at https://www.linuxfoundation.org/projects/hosting/#getstarted it seems they are open to hosting open source projects, but out topic would initially be more of a discussion and conventions-forming effort. It is currently unknown whether that group is interested in the subject.
• freedesktop.org, specifically XDG. A group that standardizes conventions for the "Linux Desktop" for use in desktop environments such as GNOME, KDE, and Xfce. It is currently unknown whether that group is interested in the subject but according to their Wiki, they "host any 'on-topic' software projects" and "developers working on any Linux/UNIX GUI technology are welcome to participate". Looking at Specifications currently in the planning/requirements-gathering stages it seems like this topic would definitely be a good fit there, even at this early stage. The shared-mime-info-spec describes a database for aspects of this overall topic (and has an implementation) and might be folded into or replaced by something with a broader scope than just MIME types. A list of the current "owners" of the XDG specs can be seen here.
• The systemd project. So far concerned with integrating services and dynamically launching services on Linux. Could possibly be extended to handle desktop applications as well. To do this, has much of the infrastructure in place that might be needed for a Launch Services and "document binding" equivalent. It is currently unknown whether that group is interested in the subject. Being hosted on freedesktop.org, it may be part of the technical solution but it seems that discussion of this topic should occur in the umbrella project.

So, on first sight it appears that freedesktop.org might be a suitable place for this kind of discussion and standardization, if we can get "non-distribution" people (like application authors and users) involved there, too.

Scope

What exactly would be the scope of "this topic"?

• A per-user dynamic application database used by the system (file manager and other components) to store information about applications available to the user (installed on the system or otherwise accessible, e.g., on attached storage media or network shares), application metadata, file associations, MIME types, icons, including user choices about per-file application/document associations. This should be generic enough as to also apply to "traditional" systems that do not use AppImage, AppDir, Snappy, Flatpak or any such systems. The spec should define the logic of how this database gets populated/updated, the meaning of its contents, and how the information should be used by the system. It should also define how to interact with this per-user database, e.g., over dbus. It does not necessarily have to define the inner workings of the database, which can be considered implementation details.
• A specification for application (bundle) metadata, e.g., an updated AppDir specification. This should describe how metadata that shall travel along with applications (e.g., in a bundle, a zip file, a snap, etc. - everywhere except in a package manager package) should be stored, so that the dynamic application database can read it. Possibly this could be made generic enough so as to also cover the metadata coming with Mac OS X bundles, Snaps, Flatpaks etc.

Somewhat related, but we should take care to keep this separate:

• Possibly even the AppImage specification (which would basically say that it is a self-mounting fileystem containing an AppDir). This should be updated to point to the above where required.

[probonopd]

Prior Art

Other systems

Classic MacOS

In the original 1984 Macintosh desktop, every application was a single file that could be freely moved around in the file system. No installation was necessary. The system used a desktop database that automatically associated files with the applications that had created them (using the type and creator resrouces). The desktop information was stored in two files, Desktop DB and Desktop DF).

Source: Wikipedia

Note that when after the merger with NeXT, Apple went on to create Mac OS X, they explicitly stated in 2000 (16 years after Classic Mac OS) that it was their objective to match that level of simplicity for modern applications.

(Source: Apple WWDC 2000 Session 144 - Mac OS X: Application Packaging and Document Typing, 3:22)

NeXTStep, NeXTSTEP, OPENSTEP

NeXTSTEP was an operating system launched by Steve Jobs' NeXT Computer in the late 1980s and is the predecessor to today's macOS. It had a concise concept application system integration. Let's the system describe itself:

Source: https://medium.com/@probonopd/make-it-simple-linux-desktop-usability-part-6-1c03de7c00a9

This is how NeXT Workspace Manager (in 1989) automatically handled file associations using the Apps directory. Note that no files (other than the application itself) have to be copied around, no databases have to be manually triggered.

The applications themselves were directories that the file manager showed like files, but the terminal showed like directories:

Source: https://medium.com/@probonopd/make-it-simple-linux-desktop-usability-part-6-1c03de7c00a9

NeXTSTEP 1.0 filesystem in Terminal vs. Directory Browser GUI

Mac OS X

When Mac OS X was created around 2000, Application Packaging and Document Typing merged the simplicity of Classic Mac OS single-file applications with the benefits of NeXT-style application directories, and introduced a new bundle format and Launch Services. This is explained in detail in

• Apple WWDC 2000 Session 144 - Mac OS X: Application Packaging and Document Typing
• Apple WWDC 2001 Session 114 - Application Packaging and Document Binding

Today's macOS, it is still possible to associate a single file to a certain application, and for the user to override this on a per-file basis. This is not possible on Linux?

Source: https://eclecticlight.co/2017/08/11/launch-services-database-problems-correcting-and-rebuilding/

The Launch Services Daemon, lsd, manages the Laucnh Services database. Check /private/var/db/lsd/ for more information.

"Desktop Linux"

What we call "Desktop Linux" is more correctly described as the X Desktop, since it can run on various kernels, including Linux. But as the vast majority of users will encounter this desktop in conjunction with the Linux kernel, we are commonly referring to it as "Desktop Linux".

ROX Filer

The ROX Desktop is a graphical desktop environment with ROXFiler as its file manager. Its UX is was inspired by RISC OS. ROX Filer is centered around the drag-and-drop philosophy of managing applications. To allow for this, it uses the AppDir format. Our new spec should attempt to make this format work across all file desktops, too?

More information: https://en.wikipedia.org/wiki/Application_directory

GNUstep GWorkspace.app

GNUstep is a Free Sotware framework that Apple's Cocoa (formerly NeXT's OpenStep) APIs but is portable to a variety of platforms and architectures. It comes with a file manager called GWorkspace.app that uses application bundles. GNUstep is available in many Linux distributions, but it has somewhat faded into abscurity, due to what I suspect is the dated 80s looking user interface (whereas Apple has constantly updated the user experience during all those decades).

GWorkspace.app application bundles live in /usr/lib/GNUstep/Applications/. Conceptually, they are similar to macOS application bundles, but somewhat different in the actual implementation (e.g., there is no Contents directory, and Info-gnustep.plist is not using XML like macOS Info.plist. (To be investigated: Perhaps this was an earlier NeXT format.)

GWorkspace running on Xubuntu 16.04; on the right hand side are the same applications when viewed with today's XFCE Thunar file manager. Our new spec should attempt to make this format work across all file desktops, too?

Source: Own screenshot (2018)

To be investigated: Does GWorkspace.app allow for drag-and-drop application management including document binding?

[hideout]

really nice read! thanks @probonopd hope you are well. And i agree with a lot of your statements

[arnog]

The situation we had to solve with MacOS X was a bit different than the contemporary situation for Linux. For one thing, we had to deal with the legacy architecture introduced by the original Mac OS in 1984, chiefly the type/creator system (a set of two four-character strings that identified respectively the type of a given file and the application that created it). We also mostly had to concern ourselves with HFS, the primary Mac filesystem that could store some limited amount of metadata (bundle bit, type/creator, etc...). However, we also increasingly had to deal with other filesystems, including from Windows and Unix that did not support this, so we were in a bit of a transitional stage.

Nowadays, the situation would be simpler. Dealing with multiple flavors of filesystems is a requirements, so using features that are specific to a given filesystem is out. And there's no 'legacy' architecture to worry about. You can then simplify the solution by having something that solely relies on filenames and directory structures, e.g. use the filename extension of a directory to indicate it's a bundle (no need for PkgInfo, etc...).

The goal of all this was to simplify for users the task of managing and installing applications. In particular, we wanted to make it very, very, hard for a casual user to 'break' the system by inadvertently moving a file somewhere, or deleting it because they didn't know what it was for (something that happened a lot with version of Mac OS before Mac OS X).

We wanted the task of installing an application to be as simple as dragging an app icon from somewhere (a server, an external drive) onto your computer's disk and be done with it. To do this, all the information that the system needed to know to handle the application was stored inside the app bundle, including its icons, its version information, the type of file it could handle, and the type of URL schemes it could handle. Then this information for 'centralized' in the Icon Services and Launch Services database. To maintain performance, applications were 'discovered' on a limited basis in a few 'well known' locations: the Applications system and user folders, and a few others, as well as automatically when the user navigated in the Finder to a directory containing an app. In practice, this worked very well.

Once you had all this info, you could give users options: the ability to associate certain file types (or URL schemes) to specific applications (if more than one could handle them) either on a global/default basis or per file (you would need metadata stored with the file for this), as well as manage multiple version of applications (by default we would always use the latest known version, but a user could specify in the Open With menu another version).

I'm happy to answer questions you may have about what we did back then. Like I said, some of it was done for legacy reasons that probably don't apply today, so I'd be glad to clarify.

[probonopd]

Thank you very much @arnog for taking the time to comment here. That the basic concepts are still used almost 20 years later in macOS and iOS shows that they have really passed the "test of time".

Yes, I did notice that some of what was described in those WWDC sessions were workarounds to support legacy systems, and I admire the careful approach that was taken to not breaking legacy while building redundancies and robustness into the system. (The Linux desktop today would also need to take into consideration the "legacy" XDG way of doing things.)

We wanted the task of installing an application to be as simple as dragging an app icon from somewhere (a server, an external drive) onto your computer's disk and be done with it.

This is precisely what the AppImage format for Linux is all about, inspired (in deep admiration) by .app in .dmg (but hopefully, eventually, even better - with things like binary delta updates and such). It is now becoming apparent that we need to tackle application desktop integration next.

I'm happy to answer questions you may have about what we did back then.

Thanks for your generous offer, I will attempt to keep the questions at a minimum so as to be mindful of your time, but can't thank you enough to be open to answer questions that may arise as we continue to think though this.

[TheAssassin]

CC @ximion this might be interesting for you as well, perhaps you want to share some thoughts?

[shoogle]

So basically a bundle is something that looks like a file (mainly in file managers, but not always) but behaves like a directory (mainly in the shell, but not always). This is not easy to get right. Perhaps a "bundle" should be a whole new kind of filesystem object; something that answers "yes" to the question "are you a file?" (for safety & backwards compatibility), but also answers "yes" to the question "are you a bundle?", allowing programs that know about bundles to be able to navigate inside it. I don't know how feasible this is, and it would probably take a long time for support to filter down into desktop environments and applications (and even longer for it to reach distributions).

Another way to implement it might be to create automatic/virtual mount points in the same directories as AppImages, so if the user installs ~/Applications/MyApp.AppImage then the mount point ~/Applications/.MyApp.AppDir (notice the leading dot) is automatically created, and automatically destroyed when the AppImage is deleted. This is obviously not perfect, but maybe it doesn't have to be. Careful reading of @arnog's comment reveals one or two places where they took shortcuts for the sake of simplicity, but as he says:

"In practice, this worked very well."

One thing that the Mac doesn't quite get right is the fact that .dmg and .app files both exist. I happen to own a MacBook (which I triple-boot with Linux and Windows) and my macOS desktop is littered with .dmg files that I never got round to deleting after installing the .app file inside. (Some users might not even realise that it is safe to delete the .dmg after you have installed the app.) It would be nice for users to be able to just drag around AppImage files without having to think about the AppDir inside it. That's something the automounting feature would facilitate quite nicely.

[shoogle]

In terms of standardisation, it appears that Freedesktop.org is basically just a wiki and anyone can request edit access to modify specifications or create new ones. There's even a proposed Application Package Specification. Here's what the specifications page has to say about it:

The Application Package Specification is designed to provide a desktop neutral way to package an application. Installation is not required.

According to the edit history it hasn't received much attention, so maybe you could adopt it?

[probonopd]

I was hoping that we could implement the

When you "look at" an application

aspect by a thumbnailer that would write to the application database. However, turns out that GNOME now runs thumbnailers in a bubblewrap sandbox, severely restricting what they can do. https://askubuntu.com/questions/1088539/custom-thumbnailers-don-t-work-on-ubuntu-18-10-and-18-04

[Pointedstick]

I don't want to derail the conversation, but I feel the need to point out that based on my experience being an Apple user since 1993, a full-time macOS user for 16 years (2000-2016) an Apple engineer for 7 years (2009 - 2016), and providing user support for an entire friends-and-family group for all of that time...

Nobody knows how to install apps via drag-and-drop.

While drag-and-drop app management sounds awesome in theory and has a certain conceptual elegance to it, in practice it is a usability catastrophe and has been since its inception. People who have been using Macs for decades are still capable of messing it up on a regular basis. I have seen this time and time again in the less-technical world of the humanities in academia where I've spent a lot of time. People just don't get it.

The ways they mess up are varied: People run apps from inside disk images; people drag-and-drop the disk image or its volume or the zip file into the Applications folder instead of the app itself; people drag-and-drop the app onto their desktop instead of the Application folder; people forget to delete the disk image or zip file and run out of disk space (macOS apps are enormous compared to Linux apps); in short almost nobody ever does this right. The number of implementation details exposed by the drag-and-drop app installation experience is actually quite high: compressed archives; disk image containers vs volumes; filesystem structure, and so on). Easing the burden of app installation by hiding as many of these implementation details was one of the major reasons why Apple abandoned the concept for iOS and encourages the use of an App Store on macOS now.

System integration is important, and various features that you point out would indeed be nice, including versioned apps, per-version app icons, transient apps on other disks, etc. But I humbly suggest that having the explicit goal of managing apps via drag-and-drop using the filesystem is a distraction. Apple tried that and it didn't actually work in practice. Even if we make a perfect system that actually does support drag-and-drop app installation to the extent that macOS does, for in the interests of usability, I would highly suggest leaving it as an experts-only feature and encouraging the use of App Store type apps instead, so that regular computer users can actually install software painlessly and properly! :smile:

[probonopd]

Thanks @Pointedstick for joining the discussion.

You seem to start from the assumption that it is a good idea (and necessary) to install software, whereas I start from the assumption that installation should no longer be needed. We used to install (hardwire) networks. Today, we just switch on WLAN and "it works". This is what we need for apps. Boot a random distribution, attach a disk full of apps, and be able to run them - with no installation. This is my vision.

Lately, Apple and other proprietary vendors are requiring users to register, log in, be online all the time to get apps from app stores - this is exactly the reason why I am turning my attention to the Linux Desktop (again).

People run apps from inside disk images; people drag-and-drop the disk image or its volume or the zip file into the Applications folder instead of the app itself

Hence, unlike the Mac, for AppImage we never extract the application directory from the disk image. We leave it inside all the time. Should remove this class of user errors.

people drag-and-drop the app onto their desktop instead of the Application folder

Why shouldn't they? On the Mac, you can put an app anywhere and it will work. So why not on the desktop?

people forget to delete the disk image or zip file and run out of disk space (macOS apps are enormous compared to Linux apps)

Since with AppImage we never extract the application directory from the disk image, this class of user errors should also not affect us.

But what I am envisioning here is not an AppImage-specific solution, but something generic enough that allows systems like AppImage, Apple .app bundles, ROX AppDirs to be hooked in (e.g., using plugins).

[shoogle]

You seem to start from the assumption that it is a good idea (and necessary) to install software, whereas I start from the assumption that installation should no longer be needed.

You should be able to run applications without installing them, but menu and icon integrations will require some form of installation - at least it should anyway! If I have a folder full of nightly builds of my favourite application then I don't want them all to appear in the system menu. Only the ones in the dedicated ~/Applications folder should appear in the menu. This is just the same as being able to run command line programs from anywhere but the ones in $PATH are the most convenient.

with AppImage we never extract the application directory from the disk image [...]. But what I am envisioning here is not an AppImage-specific solution, but something generic enough that allows systems like AppImage, Apple .app bundles, ROX AppDirs to be hooked in (e.g., using plugins).

So the container format is packaging-specific but the contents (or some of the contents) are standardised? The plugin's job is to mount the container and expose the contents. The system then reads the contents and does the integration. The integration simply involves list the files in the system database: no files are moved from the package to the filesystem.

[arnog]

One way to deal with this is to automatically "install" (i.e. register icons, menu commands, etc...) for application in well known locations, and "install" (do the same registration, potentially give the chance to the app to install drivers, etc..) apps when they are launched, regardless of their location.

[shoogle]

"install" apps when they are launched, regardless of their location.

Would part of that installation process involve moving the app to the well-known location? If not, how would the sytem deal with the app being moved or removed?

[Pointedstick]

I fully support the goal of self-contained, drag-and-droppable apps. I just don't think that those technical features should be advertised to users and exposed as the normal method for acquiring and managing software. Again, on macOS, this is a persistent source of user confusion. Expert users should be able to take advantage of these features, sure, but regular casual users prefer central app store type solutions. Getting AppImages to integrate with them (like the Nitrux app store does) should be a primary goal IMHO.

[probonopd]

We can also learn a lot from Haiku, which seems to have solved most if not all of this:

• Binaries can embed resources such as icons right inside them, and the file manager displays them. No separate icon files needed!
• There is a registrar process which seems to do roughly what Launch Services is doing - keep track of the applications as they come and go, their file type associations and such. No separate desktop files and MIME type files needed!
• Binaries get the executable bit set automatically upon certain file manager (Tracker) operations (e.g., when being double clicked). No manual chmod +x or equivalent GUI action needed!

It looks like the Haiku people share pretty much our vision of how a desktop should operate.

Thanks @waddlesplash for pointing me there.

[Pointedstick]

That makes sense, since Haiku is inspired by BeOS, which was created by Jean-Louis Gassée, one of the original Mac guys.

[probonopd]

I've just completed writing a whole series of articles on it:

1. https://medium.com/@probonopd/my-first-day-with-haiku-shockingly-good-8930cad4bbb0
2. https://medium.com/@probonopd/my-second-day-with-haiku-still-thrilled-but-not-ready-to-switch-just-yet-2310330d104b
3. https://medium.com/@probonopd/my-third-day-with-haiku-reality-is-settling-in-47b551216550
4. https://medium.com/@probonopd/my-fourth-day-with-haiku-installation-and-boot-woes-bd4f637c3ba0
5. https://medium.com/@probonopd/my-fifth-day-with-haiku-lets-port-some-apps-e0f26487cea7
6. https://medium.com/@probonopd/my-sixth-day-with-haiku-under-the-hood-of-resources-icons-and-packages-abec8d0e4ec6
7. https://medium.com/@probonopd/my-seventh-day-with-haiku-printing-scanning-networking-fb56c1dc8ee7
8. https://medium.com/@probonopd/one-more-thing-haiku-application-bundles-1803a11f9748

[probonopd]

Turns out that GNUstep's GWorkspace file manager had figured this stuff out already decades ago:

http://wiki.gnustep.org/index.php/WrapperFactory.app

• Associate file-types with your bundles
• Customize your bundle with icons for both the app itself and associated filetypes
• Each file-type you create can have a custom icon

Indeed the GWorkspace file manager does show application icons on .app bundles (GNUstep's variant of AppDirs), and does launch the application when the .app bundle is double-clicked.

It's a shame that the format is not supported by all file managers.