Using the SlicerVR extension on Acer SpatialLabs Display

[markleoryan79]

I've been asking around on the Slicer3D forum about this for a while,:

https://discourse.slicer.org/t/spatiallabs-stereoscopic-display/34800/13

I purchased the Spatial Labs Pro monitor made by Acer (https://www.acer.com/us-en/monitors/spatiallabs), which uses 2 eye tracking cameras to send images to each eye and create a 3D image. Up until this point, I have only been able to view segmentations exported as .STL or .OBJ files in the included model viewer., and the quality of the 3D image is excellent. In my case I rely a lot on the volume renderings, since the level of detail tends to be much higher. Up until this point I had not had any success in transmitting 3D images to the monitor., which is listed as being OpenXR compatible. Slicer 5.6.2 crashed whenever I would attempt to connect to the hardware.

However, yesterday I tried the extension again using the 5.7 preview build and was able to connect by setting StearmVR as the default OpenXR runtime and enabling AcerXR (their included OpenXR extension). I've attached a short video to show it functioning in the display, but there is some distortion due to picking up the images being sent to each eye. The device changes the viewing angle with head movement, so you are able to look around corners within a certain viewing angle.

I've gotten this far, but I'm a surgeon with no programming background. I wanted to see if any of the developers thought these ideas would be possible:

1. The image moves with my head movement rather than being static. So as long as I don't move my head the image is good, but I lose the ability to look at the object from different angles. Obviously this software was designed with a VR headset in mind, but I was hoping there was some way to alter the camera(s) to keep the image in one place.

2. For obvious reasons, there is not any functionality if I try to manipulate the image using the mouse. I am able to change opacity/presets for the volume rendering, which updates right away. However, if I rotate the image I have to then switch to the VR options tab and click on "set virtual reality view to match reference view". Is there any way to manipulate the image directly using the mouse? Would I have more luck with an XBox controller? I have a Quest 3 I can use, but I'm not sure if I can pair the controllers without also using the headset as a display.

3. The view shown in the 3D window within 3D slicer does not match the one on the 3D monitor. I have to position the rendering at the very bottom of the window for it to appear. I've setting the camera to a point on the image and centering the view, but no luck so far. I suspect the perspective on a VR headset is different somehow.

Anyway, thank you for all your hard work on this extension. If you haven't tried one of these monitors I highly recommend it. The images are compelling and not needing glasses makes it far more accessible. If there is any tech I would bring into the OR so far, this is it.

Untitled.video.mp4