Missing _NET_WM_WINDOW_TYPE
When using the i3 window-manager (or any other tiling window manager, i suspect), fixed-size dialog-windows like the "connecting"-message, or the preferences window, are scaled by i3, instead of being displayed in floating mode.
I suspect this is due to the missing _NET_WM_WINDOW_TYPE, which tells the window manager how to properly handle this window. The correct value for mentioned windows should probably be _NET_WM_WINDOW_TYPE_DIALOG.
You can find more information about this topic here: http://standards.freedesktop.org/wm-spec/1.3/ar01s05.html
This is not as simple as you would think as internally there is no difference between the main Steam client window and the Settings (or Friends) modeless dialogs. We could set the dialog type on them but then the main window would be a dialog also, which is probably not what you want.
Oh, I didn't know that. Is there any hope that this will be resolved until the final release? Even on Ubuntu, some folks are using tiling WMs.
There are two parts: is there an actual difference between the Steam client window and its settings window (or the connecting status window)? They are both top-level windows with a title bar that you can move around. I'm not sure that they are different in terms of what kind of window they are. What makes one a dialog while the other is a normal window?
The other part would be whether substantially different things like modal dialogs could be marked as dialogs and that's somewhat easier to do, but doesn't seem to address your concerns.
The settings window and status windows would classify as dialogs, because for one, they are not permanent (you use them and close them, unlike the main steam window or the friends list, which may be kept open for a while), and more importantly, they do not scale well. You can maximize settings or status windows, but it does not make any sense. And in case of windows with progress bars (like preparing an installation), they even have rendering errors when maximized.
If by modal dialogs you mean the "right-click"-context-menus (Sorry, I don't know the correct name), yes, they are already handled properly by i3.
We don't have a reliable way of drawing distinctions like the examples you mentioned, so we'll discuss this more but I'm not sure we'll make any changes here.
I forgot to mention, the exit dialogue "Synchronising with Steam Cloud" is displayed and hanlded perfectly. I just don't know which Window Properties it has, because it displays too short for me to invoke xprop on int ;)
Even making them all dialogs would be reasonable I think. Since steam is designed around a floating window manager making all windows float would be fine. When using a tiling manager it's easy enough to change but the default is broken. Especially for the initial login dialogs.
Here's what it looks like when the windows are tiled vs floating.

It's probably better to think of the dialog hint as a floating hint too. It's crazy and unintuitive but there's really no other way to tell the window manager that your windows should always be floating.
I think the main window should not be kept floating, it integrates very well with tiling.
By the way, I found a quick workaround for i3, based on the window titles: Just add
for_window [class="Steam"] floating enable
for_window [class="Steam" title="^Steam$"] floating disable
to your i3-config. It will put every Steam window into floating mode, except for the main window, and login message.
And one stupid question for the developers: Wouldn't it be possible to set individual window hints along with the window titles?
UPDATE: because regex is hard
Replying to https://github.com/ValveSoftware/steam-for-linux/issues/597#issuecomment-12034933
For new visitors, steam's class is lowercase now, use:
for_window [class="steam"] floating enable
for_window [class="steam" title="^Steam$"] floating disable