The Steam Deck is a nice candidate for some light exploration because it's not just a default install of some standard Linux distro (SteamOS 3.0 is based on Arch, but has been customized), and at the same time it is unlike your usual embedded target: wide open, comes with all the usual system tools we'd immediately strip normally (or not even build in the first place) when making a Linux for a device, and intended to allow breaking out of the safety net and using it as a general-purpose computer. To do this I enabled SSH access to the Deck, because I don't have a USB-C adapter for a keyboard and the on-screen keyboard, while not entirely terrible, really isn't nice to use for shell stuff. So I only used it to set a password for the default "deck" user with passwd and turned on SSH temporarily with sudo systemctl enable sshd. By default the Steam Deck ships with a read-only rootfs, and while you can disable this it is warned that updates will reset it. At the same time, it clearly needs some places to have games/settings/user data, so those will be mounted elsewhere. So lets look at the block devices:

(deck@steamdeck ~)$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0 476.9G  0 disk
├─nvme0n1p1 259:1    0    64M  0 part
├─nvme0n1p2 259:2    0    32M  0 part
├─nvme0n1p3 259:3    0    32M  0 part
├─nvme0n1p4 259:4    0     5G  0 part
├─nvme0n1p5 259:5    0     5G  0 part /
├─nvme0n1p6 259:6    0   256M  0 part
├─nvme0n1p7 259:7    0   256M  0 part /var
└─nvme0n1p8 259:8    0 466.3G  0 part /var/tmp
                                      /var/log
                                      /var/lib/systemd/coredump
                                      /var/lib/flatpak
                                      /var/lib/docker
                                      /root
                                      /var/cache/pacman
                                      /srv
                                      /opt
                                      /home

A small-ish root partition, small /var and then a large partition holding all the rest. And a pile of unmounted partitions, several of which are paired in size. Likely bootloader etc, and I'd guess the pairs are for A/B updates, where the updater writes to whichever one currently isn't in use. That way the current one is preserved and available for boot if anything goes wrong during the update or the update is faulty. This is very common in embedded and appliance setups. At least the latter for sure is writeable and holding data - certainly makes sense for /var/tmp, /var/log, /var/lib/systemd/coredump. /var/lib/flatpak also isn't surprising, given that Flatpaks are the recommended way of installing apps outside the Steam ecosystem. The desktop environment ships with the KDE Discover "app store", and Flatpaks are nicely self-contained without dependencies on the rest of the OS that might change. /var/lib/docker … does this thing ship docker for whatever reason?

(deck@steamdeck ~)$ sudo ls -Al /var/lib/docker
total 0
(deck@steamdeck ~)$

directory is empty at least.

(deck@steamdeck ~)$ which docker
which: no docker in (/usr/local/sbin:/usr/local/bin:/usr/bin:/var/lib/flatpak/exports/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl)
(1)(deck@steamdeck ~)$ which dockerd
which: no dockerd in (/usr/local/sbin:/usr/local/bin:/usr/bin:/var/lib/flatpak/exports/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl)
(1)(deck@steamdeck ~)$

Doesn't seem like it. Maybe they use it in development setups for some reason, or planned to have it and this was left over, who knows.

/srv and /opt are also basically empty, but I guess it just makes things easier for people that do manually install things if they exist (and reduces the chances something gets messed up when they try to fix it). /root just has a smathering of default-ish dotfiles.

I've already installed quite a few GB of games, so where are those?

(deck@steamdeck ~)$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.3G     0  7.3G   0% /dev
tmpfs           7.3G  592M  6.7G   8% /dev/shm
tmpfs           2.9G  9.8M  2.9G   1% /run
/dev/nvme0n1p5  5.0G  3.3G  1.5G  69% /
/dev/nvme0n1p7  230M   32M  182M  15% /var
overlay         230M   32M  182M  15% /etc
/dev/nvme0n1p8  466G   83G  383G  18% /home
tmpfs           7.3G  1.2M  7.3G   1% /tmp
tmpfs           1.5G  124K  1.5G   1% /run/user/1000

Ok, in /home. Let's leave digging deep into that for later, we still don't know what all those unmounted partitions are. Sometimes those are mounted by labels, lets check if it is so nice to list those in the fs …

(deck@steamdeck /)$ ls -Al /dev/disk/by- [tab]
by-id/        by-label/     by-partlabel/ by-partsets/  by-partuuid/  by-path/      by-uuid/

… partlabel? partsets? What's that?

(deck@steamdeck ~)$ ls -Al /dev/disk/by-partlabel/
total 0
lrwxrwxrwx 1 root root 15 May  6 23:06 efi-A -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 May  6 23:06 efi-B -> ../../nvme0n1p3
lrwxrwxrwx 1 root root 15 May  6 23:06 esp -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 May  6 23:06 home -> ../../nvme0n1p8
lrwxrwxrwx 1 root root 15 May  6 23:06 rootfs-A -> ../../nvme0n1p4
lrwxrwxrwx 1 root root 15 May  6 23:06 rootfs-B -> ../../nvme0n1p5
lrwxrwxrwx 1 root root 15 May  6 23:06 var-A -> ../../nvme0n1p6
lrwxrwxrwx 1 root root 15 May  6 23:06 var-B -> ../../nvme0n1p7

Well, that confirms the assumption about there being A/B boot for updates. Given that we above saw that nvme0n1p5 and nvme0n1p7 have mountpoints right now, we clearly are booted into the B image. The Arch wiki confirms that ESP is also an UEFI thing (EFI System Partition), even though it suggests other mount point names. Apropos, in / there are an /esp and /efi, why did lsblk didn't see them?

(deck@steamdeck ~)$ mount
[...]
systemd-1 on /efi type autofs (rw,relatime,fd=47,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=12040)
systemd-1 on /esp type autofs (rw,relatime,fd=51,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=12043)
[...]

Automounts!

(deck@steamdeck ~)$ ls /esp
ls: cannot open directory '/esp': Permission denied
(deck@steamdeck ~)$ ls /efi
ls: cannot open directory '/efi': Permission denied

(deck@steamdeck ~)$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0 476.9G  0 disk
├─nvme0n1p1 259:1    0    64M  0 part /esp
├─nvme0n1p2 259:2    0    32M  0 part
├─nvme0n1p3 259:3    0    32M  0 part /efi
[...]

There they are! Note for the future: check mount earlier, because automounts don't show up in lsblk. They also were quickly gone, because the automount is set (as visible in the output of mount above) with timeout=60, so it gets quickly unmounted again. Is this a safety feature to sync the filesystems to disk quickly, especially since those are probably FAT-something and nothing modern and crash-resistant? Just a sign of "you shouldn't need this"? I'm not sure. Just quickly, what were those other by-* groups?

(deck@steamdeck ~)$ ls -Al /dev/disk/by-label
total 0
lrwxrwxrwx 1 root root 15 May  6 23:06 efi -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 May  6 23:06 esp -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 May  6 23:06 home -> ../../nvme0n1p8
lrwxrwxrwx 1 root root 15 May  6 23:06 rootfs -> ../../nvme0n1p5
lrwxrwxrwx 1 root root 15 May  6 23:06 var -> ../../nvme0n1p7

(deck@steamdeck ~)$ ls -Al /dev/disk/by-partsets
total 0
drwxr-xr-x 2 root root 100 May  6 23:06 A
drwxr-xr-x 2 root root 200 May  6 23:06 all
drwxr-xr-x 2 root root 100 May  6 23:06 B
drwxr-xr-x 2 root root 100 May  6 23:06 other
drwxr-xr-x 2 root root 100 May  6 23:06 self
drwxr-xr-x 2 root root  80 May  6 23:06 shared

Ok, so by-label is only the currently active ones, with the prefix stripped, and by-partsets has them grouped by the "absolute" set (A or B), relative to the current one (self, right now B, and other, right now A), common being always used and all again being everything.

(deck@steamdeck ~)$ ls -Al /dev/disk/by-partsets/self
total 0
lrwxrwxrwx 1 root root 18 May  6 23:06 efi -> ../../../nvme0n1p3
lrwxrwxrwx 1 root root 18 May  6 23:06 rootfs -> ../../../nvme0n1p5
lrwxrwxrwx 1 root root 18 May  6 23:06 var -> ../../../nvme0n1p7
(deck@steamdeck ~)$ ls -Al /dev/disk/by-partsets/B
total 0
lrwxrwxrwx 1 root root 18 May  6 23:06 efi -> ../../../nvme0n1p3
lrwxrwxrwx 1 root root 18 May  6 23:06 rootfs -> ../../../nvme0n1p5
lrwxrwxrwx 1 root root 18 May  6 23:06 var -> ../../../nvme0n1p7

(deck@steamdeck ~)$ ls -Al /dev/disk/by-partsets/shared
total 0
lrwxrwxrwx 1 root root 18 May  6 23:06 esp -> ../../../nvme0n1p1
lrwxrwxrwx 1 root root 18 May  6 23:06 home -> ../../../nvme0n1p8

I don't know off-hand what kind of mechanism is in use here to implement this and on what level these things get re-labelled and added, but the principle is quite clear.

Back to mount, there were two other things:

(deck@steamdeck ~)$ mount
[...]
/dev/nvme0n1p5 on / type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/)
[...]
overlay on /etc type overlay (rw,relatime,lowerdir=/sysroot/etc,upperdir=/sysroot/var/lib/overlays/etc/upper,workdir=/sysroot/var/lib/overlays/etc/work)

The second one is fairly straight forward: /etc gets overlayFS so it can be written for things that insist on hardcoded path while being inside the read-only root for most of its files (/etc/resolv.conf is a classic example of a file that gets updated at runtime that really needs to be in this place, symlinks don't cut it). With the former, why "rw"? Aren't we in a read-only root filesystem?

(deck@steamdeck work)$ touch /x
touch: cannot touch '/x': Read-only file system

Appears so. Is that something btrfs can do independently? How exactly is the unlocking done if requested? According to the docs there is a command sudo steamos-readonly disable, and with a bit of luck …

(deck@steamdeck work)$ file `which steamos-readonly`
/usr/bin/steamos-readonly: Bourne-Again shell script, Unicode text, UTF-8 text executable

… that's a shell script. And a quick skim shows that indeed, that's a btrfs setting (key parts extracted):

# mark root partition writable
read_write_btrfs() {
    mount -o remount,rw /
    btrfs property set / ro false
}

# mark root partition read-only
read_only_btrfs() {
    btrfs property set / ro true
}

From above we still have some open questions about the large/home (et al) partition. Where exactly is the meat of things, games and such, and where exactly to all these sub-mounts live.

Games a large, so the former should be easy to answer:

(deck@steamdeck ~)$ du -a /home | sort -n -r | head -n 20
79973096        /home
77606552        /home/deck
77208228        /home/deck/.local
77208176        /home/deck/.local/share
77206516        /home/deck/.local/share/Steam
74580616        /home/deck/.local/share/Steam/steamapps
72520748        /home/deck/.local/share/Steam/steamapps/common
33437560        /home/deck/.local/share/Steam/steamapps/common/Wreckfest
31347448        /home/deck/.local/share/Steam/steamapps/common/Wreckfest/data
15509760        /home/deck/.local/share/Steam/steamapps/common/Wreckfest/data/vehicle
14933888        /home/deck/.local/share/Steam/steamapps/common/Wreckfest/data/art
12532840        /home/deck/.local/share/Steam/steamapps/common/Portal 2
10486772        /home/deck/.local/share/Steam/steamapps/common/Portal 2/portal2
8737256 /home/deck/.local/share/Steam/steamapps/common/Wreckfest/data/art/levels
6921936 /home/deck/.local/share/Steam/steamapps/common/Cloudpunk
6870380 /home/deck/.local/share/Steam/steamapps/common/Cloudpunk/Cloudpunk_Data
5590476 /home/deck/.local/share/Steam/steamapps/common/Portal Reloaded
4357676 /home/deck/.local/share/Steam/steamapps/common/Aperture Desk Job
4357672 /home/deck/.local/share/Steam/steamapps/common/Aperture Desk Job/game
3799168 /home/deck/.local/share/Steam/steamapps/common/Portal Reloaded/portal2

would be where. And presumably the wine/proton-runtimes are also in there somewhere?

(deck@steamdeck ~)$ find -name "*proton*"
./.local/share/Steam/steamapps/common/Proton 7.0/dist/lib64/wine/vkd3d-proton
./.local/share/Steam/steamapps/common/Proton 7.0/dist/lib64/wine/vkd3d-proton/libvkd3d-proton-utils-3.dll
./.local/share/Steam/steamapps/common/Proton 7.0/dist/lib64/gstreamer-1.0/libprotonmediaconverter.so
./.local/share/Steam/steamapps/common/Proton 7.0/dist/lib/wine/vkd3d-proton
./.local/share/Steam/steamapps/common/Proton 7.0/dist/lib/wine/vkd3d-proton/libvkd3d-proton-utils-3.dll
./.local/share/Steam/steamapps/common/Proton 7.0/dist/lib/gstreamer-1.0/libprotonmediaconverter.so
./.local/share/Steam/steamapps/common/Proton 7.0/proton
./.local/share/Steam/steamapps/common/Proton 7.0/proton_3.7_tracked_files
./.local/share/Steam/steamapps/common/Proton 7.0/proton_dist.tar

Indeed they are. And actually I omitted some errors from the du output above, which already give us a good hint about where the /var/tmp, /var/log, /var/lib/*, /root, … are:

du: cannot read directory '/home/lost+found': Permission denied
du: cannot read directory '/home/.steamos/offload/var/log/private': Permission denied
du: cannot read directory '/home/.steamos/offload/var/tmp/systemd-private-720d8377f2b248368e65c4b7a6f69ee5-systemd-logind.service-dILMLM': Permission denied
du: cannot read directory '/home/.steamos/offload/var/tmp/systemd-private-720d8377f2b248368e65c4b7a6f69ee5-iwd.service-JOU9yF': Permission denied
du: cannot read directory '/home/.steamos/offload/var/tmp/systemd-private-720d8377f2b248368e65c4b7a6f69ee5-upower.service-2jH9Wu': Permission denied
du: cannot read directory '/home/.steamos/offload/var/tmp/systemd-private-720d8377f2b248368e65c4b7a6f69ee5-systemd-timesyncd.service-bDMH37': Permission denied
du: cannot read directory '/home/.steamos/offload/root': Permission denied

Quick check:

(deck@steamdeck ~)$ sudo ls /home/.steamos
offload
(deck@steamdeck ~)$ sudo ls /home/.steamos/offload/
opt  root  srv  usr  var
(deck@steamdeck ~)$ sudo ls /home/.steamos/offload/var/
cache  lib  log  tmp
(deck@steamdeck ~)$ sudo ls /home/.steamos/offload/var/lib/
docker  flatpak  systemd

As guessed. Their mounts are are all handled through systemd:

(deck@steamdeck ~)$ ls /usr/lib/systemd/system/*.mount
/usr/lib/systemd/system/boot.mount                     /usr/lib/systemd/system/sys-kernel-tracing.mount
/usr/lib/systemd/system/dev-hugepages.mount            /usr/lib/systemd/system/tmp.mount
/usr/lib/systemd/system/dev-mqueue.mount               /usr/lib/systemd/system/usr-lib-debug.mount
/usr/lib/systemd/system/etc.mount                      /usr/lib/systemd/system/usr-local.mount
/usr/lib/systemd/system/opt.mount                      /usr/lib/systemd/system/var-cache-pacman.mount
/usr/lib/systemd/system/proc-sys-fs-binfmt_misc.mount  /usr/lib/systemd/system/var-lib-docker.mount
/usr/lib/systemd/system/root.mount                     /usr/lib/systemd/system/var-lib-flatpak.mount
/usr/lib/systemd/system/srv.mount                      /usr/lib/systemd/system/var-lib-machines.mount
/usr/lib/systemd/system/sys-fs-fuse-connections.mount  /usr/lib/systemd/system/var-lib-systemd-coredump.mount
/usr/lib/systemd/system/sys-kernel-config.mount        /usr/lib/systemd/system/var-log.mount
/usr/lib/systemd/system/sys-kernel-debug.mount         /usr/lib/systemd/system/var-tmp.mount

They all are straightforward bind mounds putting various folders from /home/.steamos/offload in the right places. A few (like /usr/lib/debug) are masked and thus not active, which explains why there are more than we'd expect. This covers the filesystem side of this. Potential next points: What software/services are running here? What does the deck-specific hardware look like to Linux?

Not part of the main specification, but important for private feeds and worth documenting.

I see three models:

1. WebSub informs all users about all changes to the feed

When the feed changes, the site triggers a notification to all subscribers. If the feed change is in a private post, it does not include it in the ping, either sending the last public state of the feed or an empty ping (effectively announcing an empty diff). Authorized subscribers would take this as a signal to fetch the feed with authorization attached.

This leaks the fact that *something* private happened to all subscribers. The hub is not involved in handling private information at all, and thus can safely be external.

• Do all/most hubs allow this? A hub that wants to create the diff itself might reject sending out an empty notification, but at least for non-Atom content I don't think hubs do this.
• How do subscribers handle empty pings? Does it cause them to fetch the page, assuming a "thin ping"? This could be mitigated by separating out authorized subscribers as described below. (WebSub does not know "thin pings", but I think pubsubhubbub did and clients might support them)

2. individual topics for private subscribers with fat pings

The site could give different topic URLs (capability URLs) to private subscribers, and send matching notifications to them. (Compare how the WebSub spec recommends returning different rel=self URLs for different content types. Potentially, fat pings could be used then.

A site could use an integrated hub for private subscribers and still let a public hub handle everyone else.

• The hub here can't fetch the private topic URLs (unless it has special support/is integrated).
• If the capability URL leaks, others can subscribe to it and would receive notifications. This would compromise fat pings. Subscribing applications and hubs would need to take care to not leak this, but hubs developed assuming public feeds might not do this. Integrated hubs could only allow one subscription per topic URL, which could mitigate this when each time a different capability URL is submitted.
• integration with token expiry/revocation is needed: the link between token and capability URL must be maintained, and topics associated with invalid tokens not updated anymore.

3. individual topics for private subscribers with thin pings

Compared to 1, it at keeps activity private and solves the issue mentioned above of subscribers potentially fetching needlessly. Compared to 3, it removes complexity, trust in the hub and leaking the cability URL is less problematic, but requires feed fetches on notification.

conclusion

I think 2. is too much complexity. I think it makes sense to document 3., and potentially 1. as an easier option. Testing how it works with existing clients and hubs is needed.

Thoughts/comments?