Supporting virtual desktop/display creation across different Linux window systems

[wheaney]

Moving the discussion from xrealAirLinuxDriver #23 here.

It looks like Wayland makes it difficult to create virtual displays that are commonly used in screen sharing apps and apps like XREAL Nebula that want to render multiple monitors to a single display (i.e. the glasses). We'll probably have to take a multi-pronged approach, targeting each Window System with their own custom desktop creation logic:

• X11 would just use xrandr
• Wayland would depend on the specific compositor:

1. KDE provides a custom Wayland Protocol (and this library seems to provide an integration but I think we can probably do something more lightweight since we just want to create and destroy displays)
2. GNOME provides a tool called gsettings that seems to allow for easily creating a new virtual monitor (e.g. gsettings set org.gnome.desktop.remote-desktop.rdp screen-share-mode extend [source])

[wheaney]

One crazy idea I just had for Wayland: run a docker container or VM that also runs a Wayland compositor, set it up so there are multiple "monitors" hooked up as virtual hardware, then hook up waypipe so that it can launch and render applications from the host machine, and then finally use the display output from the VM/docker as your "virtual displays."

[wheaney]

I wonder if we could even kick off a second Wayland compositor (without docker or VM). Any extension of wlroots would have the ability to run headless displays that could be used for our purposes.

[xdave]

This would be awesome

[wheaney]

Today I was playing around with the GNOME mutter library for spawning a headless compositor. It can take virtual display params right on startup, and even add more virtual displays on-the-fly. I found that launching it without displays was best, actually, because I plugged in this script for screen casting which seems to create its own display. I modified that script to send the gstreamer pipeline through vulkanupload and then vulkansink elements, then if I kick it off with ENABLE_VKBASALT=1 as a launch variable, it enables my breezy vulkan shader injection and I actually get head movement tracking on a virtual display that doesn't require xrandr or some kind of dummy plug. So this is one possible solution, but there would still be a lot to solve:

• The gstreamer pipeline doesn't send any video when there's nothing changing on-screen, so the virtual display won't move unless something on-screen is moving. I tried forcing a videorate but that caused the pipeline to fail; I'm sure I can use something like this to force a constant framerate, though.
• Need a way to launch applications from mutter. One way I came across but didn't play with yet was to launch an application that acts as a launcher, and send it off to mutter using WAYLAND_DISPLAY in the launch params. This isn't super ideal because it doesn't match the way you would be launching apps in your native desktop, but I don't really care as long as it's something that would work and doesn't suck to use.
• Need a way to send mouse/keyboard events to mutter. Something like Synergy would be awesome but that doesn't work with Wayland yet. Mutter does support remote desktop/VNC, so using an existing RemoteDesktop library (for the control side, not screen casting) or building one that minimally utilizes the Remote Desktop protocol for just mouse/keyboard events should be do-able, but possibly a lot of work.

[wheaney]

I'm gonna close this discussion. After a lot of research and experimentation I've landed on doing this in GNOME as an extension, and eventually also using GNOME in nested mode for use with other compositors (similar to my previous comments about headless).