Oct 18, 2024

opinion Augmented RealityEmerging TechnologyVirtual Reality Mainstream desktop and laptop PCs should fully support AR screens and hand gestures right out of the box. Credit: Thinkstock Apple released a claustrophobic submarine movie for Apple Vision Pro earlier this month, and it gave me a sinking feeling. The movie Submerged was neither augmented reality (AR) nor virtual reality (VR) nor 360-degree video. It was more like 180-degree video. According to reviews, the experience was as immersive as a submarine with a hole in it. But wait a minute, you might say. I thought Apple Vision Pro was supposed to be mainly AR. While the tech media and even the mainstream press point out that Apple Vision Pro  sales aren’t anywhere near as high as Apple expected , a community of diehards still use the product heavily every day. The main rationale for these heavy users, the most practical and productive usage, is to expand the desktop of a MacBook Pro to gigantic proportions as AR displays floating in the air. That general concept — using AR for desktop computing rather than video games, holographic hijinks, or teleportation meetings — is what Sightful’s Spacetop computing platform was all about. The company’s Spacetop G1 was a one-of-a-kind augmented reality laptop that used AR glasses instead of a screen. The system displayed a virtual 100-in. display in front of the user anchored in space relative to the keyboard. The system ran on a proprietary Android-based operating system called SpaceOS; it functioned like a Chromebook, running apps from the cloud instead of desktop applications running from the device. I  first told you about it in June . I use the past tense here because, while the Spacetop G1 was scheduled to ship this month, it’s been killed off. The company emailed people who pre-ordered it and informed them their deposit would be refunded and the product would be canceled. Instead, Sightful is now working on software that enables Windows PCs to project their displays onto Xreal’s AR display glasses, the same brand that shipped with the Spacetop. Now, AR virtual desktops are a feature, not a product Sightful has discovered that AR virtual desktop screens are a feature, not a product. Unfortunately, their offering will be limited and, essentially, unnecessary. While their integrated package was one-of-a-kind, their software will be joining a small but crowded market. Xreal, for example, already supports both Windows and MacOS using its own Nebula software. (It’s unclear why users might want Sightful’s software instead of Xreal’s.) Viture One supports Windows and MacOS, enabling users to set up virtual displays in different configurations. TCL NXTWEAR S is compatible with Windows devices, allowing users to view their PC screen through the glasses. Lenovo ThinkReality A3 works with Windows machines to output virtual desktops with up to three virtual screens. And there are others. Another interesting new product is the Visor, a lightweight spatial computing device vaguely similar to the Apple Vision Pro developed by Immersed, an Austin-based startup. It features two 4K OLED screens with a 100-degree field of view, 6DoF tracking, and hand and eye tracking capabilities. Weighing only 186g, it’s significantly lighter than competitors like the Meta Quest 3 and the Vision Pro. And like the Vision Pro, Visor is VR, creating the illusion of AR via passthrough video. Powered by the Qualcomm XR2+ Gen 2 chip, the Visor is designed primarily for productivity. It allows users to work with multiple virtual screens in both passthrough and fully virtual modes. That said, recent demos have shown issues with readability, heat management, and passthrough quality. The product gives users five HD screens on Windows, macOS, and Linux. Starting at $1,050 for outright purchase, with a subscription model also available, the Immersed Visor is set to begin shipping limited “Founder’s Edition” units next month, with wider availability in April 2025. While promising, potential buyers should approach with caution, given the device’s ongoing development and the challenges observed in recent demonstrations. The home run the industry hasn’t hit yet Some of the AR glasses mentioned above support hand gestures via sensors in the glasses, and most anchor the desktop screens fully in place. Sightful’s Spacetop G1 had something going for it that none of these other solutions did: Integration with the laptop. The Spacetop G1’s keyboard base, which contained its own Qualcomm processor, served as a spatially tracked anchor for the AR glasses. This enabled the system to maintain a consistent positioning of the virtual displays relative to the physical keyboard. The device had a depth camera mounted above the keyboard that tracked the user’s position relative to the keyboard and where they looked. This provided a significantly better AR experience when sitting in front of the keyboard. The Spacetop G1 incorporated sensors that mapped the physical environment to ensure virtual elements were accurately anchored in the real world. And it used customized XREAL glasses, with cameras specifically designed to detect the Spacetop. In summary, the system considered both the user’s position and the surrounding environment, and the glasses and the base unit worked together to improve screen anchoring and gesture control. Positioning sensors on a laptop and glasses is far better than on the glasses alone. What we need from the industry is laptops with sensors built in and designed to work in concert with AR glasses for screen position, eye-tracking, and gesture control. Apple doesn’t want to support cheaper, non-Apple solutions, so forget about Apple doing this. The Vision Pro experience is already amazing — though the price of admission is $3,500. (Apple is expected to ship lower-priced Vision Pro models next year at the earliest.) The two best platforms for AR glasses support The two best platforms for dedicated hardware support for AR glasses are Windows and ChromeOS. ChromeOS is especially interesting because that platform is often used by schools and others looking for very low-cost hardware. By optimizing AR glasses for virtual desktops, you not only get a very low-cost laptop, but also a very low-cost giant display. Or two. Or three. Microsoft is uniquely positioned to offer Windows laptops with built-in AR glasses support using Microsoft-branded glasses. In fact, the entire Windows and Linux OEM market should be developing or partnering with companies that make AR glasses. As a digital nomad who travels abroad while working and loves giant screens that are too big to carry, I’m super excited about virtual desktop screens. All we need now is to optimize the experience by laptop-glasses integration. Sightful had the right idea, but it may have been the wrong company. It’s extremely difficult for a startup to enter the PC market with a usage model that has yet to be tried. That kind of leap can normally come only from an established giant with money to burn. Speaking of which, somebody [cough!] Google! [cough!] should buy Sightful, its five or so patents, two founders, 60 employees, and vision for AR laptops, and convert their ideas into a mainstream OS, such as ChromeOS. AR is definitely going to happen. The concept of using it to give tiny laptops gigantic virtual screens is also going to happen. We need the industry to optimize and mainstream this idea so that it doesn’t belong exclusively to a few hundred thousand Apple Vision Pro enthusiasts. Laptops should just come with the built in sensors and OS support, with the glasses available separately as just another display option. A high-quality experience with virtual desktop screens in a laptop shouldn’t cost $7,000 (the combined price of a Macbook Pro plus Apple Vision Pro). It should cost $700 for a low-end Chromebook and $1,000 for a Surface laptop. Virtual screens are far better for mobility, far better for the environment and far better for the wallet than giant physical displays. By improving the user experience (via laptop-embedded sensors and OS integration) and lowering the price, virtual screens could become serious  mainstream options for how we all use our laptops. Related content