Stefano Baldassi is a man on a mission - to build the world's first truly human computing machine. And with a handpicked team of neuroscientists and technologists, he is laying the foundation for what will undoubtedly become one of the most transformative cultural innovations of the 21st century.
Stefano is the Director of Neuroscience & Analytics at Meta, a key pioneer of augmented reality technology, and the scrappiest member of the "3Ms" - Microsoft, Meta, and Magic Leap. With recent advances in near-field displays, hand tracking technology, and artificial intelligence, Meta is building a tool from science fiction fantasy - a wearable head-mounted computer that projects 3D holograms in space that you can control with your hands.
Fresh off the release of their latest product - the Meta 2 DK - I recently sat down with Stefano to discuss Meta's past and future, designing for the human brain, and why the kinesthetic response times of athletes in the future will be 50% faster than they are today.
A Revolution in Your Hands
Just how did Meta go from a modest Kickstarter project in 2013 to the $300MM golden child of a multi-billion dollar computer revolution?
"The basic idea behind Meta was to create a see-through display with hand control, to allow users to touch the hologram," says Baldassi.
"It may sound simplistic, but dreams are simple. I want to design a machine that will help people be more human."
While 2D gestures such as swipe and pinch have become massively popular, building an input system around 3D gestures is technically daunting and philosophically controversial. Many of Meta's competitors have chosen to follow different paths.
Microsoft, for instance, has chosen to ship its augmented reality computer, the Hololens, with a clicker users can use much like a mouse. And Google recently bought EyeFluence, a platform for allowing users to control computer programs with their eyes. Integration of eye tracking into Google Glass and other computational platforms is sure to follow.
"We don't want remote click or gaze-based gestures that the competitors have," Stefano says. "Try moving your eyes and imaging that you are moving things in the real world, and tell me how long it takes for your eyes to get strained. Your eyes are not designed for moving things around, unless you are locked into a wheelchair. In the real world, people use their eyes to select things for their hands to reach."
Whether or not Meta's handy interface will pay off remains to be seen. But in an industry that has rewarded user experience innovators like Apple with billions in profits, a deep focus on creating an immersive natural experience is perhaps the most important thing that separates Meta from the competition.
And this is where Stefano and his team of neuroscientists steps in.
The Brain: A Human Machine
The Bayesian Brain is a model that scientists use to predict human behavior based on "priors" - i.e., a person's prior experiences and mental models. Stefano and his team are using the Bayesian brain to guide Meta's user interface and design philosophy.
To design the ultimate brain-computer interface, Stefano has assembled a team of specialists from neuroscience, mathematics, computer science, data analytics, and user experience. His team is responsible for user research and product requirements for Meta's optical engine, sensors, hand tracking, and ergonomic design.
"If you are strapping it to your eyes, you are going to have to take neuroscience into account."
"In my team, we work from the very low-level properties of sensors to the interactions that flow from those sensors - the specific hand manipulations, visual interactions, even some marketing research. Any research that involves users, we own it," says Baldassi.
The team is responsible for setting the company's design philosophy. Many of their product requirements come directly from Meta's current user base, primary developers at Fortune 100 companies. These early adopters - from the fields of architecture, medicine, commerce, and other verticals - are eager to build software on top of Meta's platform to gain a competitive edge.
The team is also tasked to demonstrate use cases for Meta's technology. For example, Stefano's team recently showed that users given a lego assembly task performed up to 20% faster when given 3D augmented reality instructions vs traditional paper-based instructions. The study was published in the proceedings of the 2016 IEEE ISMAR conference in Merida, Mexico. Groundbreaking articles in popular journals such as Nature and Journal of Neuroscience further round out their learnings.
The result? A novel design approach deeply rooted in the very soil of the human experience. "We think that design should be led by deep understanding of neuroscience vs. the arbitrary way of designers that maybe have good aesthetic sense and intuition. If you are strapping it to your eyes, you are going to have to take neuroscience into account."
Spatial Design: The New Frontier
Stefano's team is working on a set of principles for augmented reality they call "Spatial Design". Spatial Design is a UX design philosophy that emphasizes realism in tools and content, while seeking to minimize abstractions such as menus and icons.
If Stefano's team has learned one thing from all their research, it's that the human brain is a deeply intelligent system. And if you wish to tap into the full power of the brain in real-world scenarios, you need to create a truly immersive interface.
"Through a tiny window, you can nail every bit of technology, but the experience will not exist."
"We understood from Meta 1 that developers wanted a large field of view. So we worked hard to get a 90 degree field of view, which is more than twice as large as the competing technology," says Baldassi. "Through a tiny window, you can nail every bit of technology, but the experience will not exist."
With a new 3D platform to design around, the team is positioned to challenge long-held approaches and beliefs about computer interfaces. Software designers have been using affordances such as windows, menus, and icons to display content for nearly the entire history of computing, dating back to the invention of the graphical user interface in the early 70's at Xerox PARC. In the world of augmented reality, however, Stefano's team believes designers need to throw all that out and start from scratch.
"We want to create natural interactions derived from neuroscience," says Baldassi. "So we have worked over the last two years on these design principles called spatial interfaces. The basic idea is about moving from the WIMP paradigm (windows, icons, menus, & pointers) to content and tools."
The team is authoring a manifesto of sorts, a set of user experience guidelines they plan to distribute to developers and the design community at large. Their current working draft includes guiding principles, patterns and anti-patterns, and supporting neuroscience readings. The document will likely evolve into a concrete pattern library, much like Google has done for VR. The goal, of course, is to help developers build software applications that are as natural to use as throwing a ball or writing with pen and paper.
The team plans to give their library away for free. "These principles, the science we are doing, we don't just want to keep it for Meta," says Baldassi. "We are giving it to the world. We want to be thought leaders, but we are not hiding our learnings from our competitors."
Unlocking Human Potential
The true proof of concept for augmented reality may be Meta's own plans for it: to replace all computers at their company with augmented reality machines by Q2 of this year. Will mission-critical work still get done?
This year, Meta is making a radical move to prove that augmented reality technology is not only the future, but is 100% usable by today's IT workforce.
"One year ago today, our CEO gave a TED talk," says Baldassi. "He promised that by TED 2017 [in late April], Meta would get rid of physical displays to use only augmented reality displays. And in the next 3-4 months, we will complete a full transition of the company, even including our IT folks."
"At this point, our competitors are traditional computers and not our AR competitors. If one wins, we all win."
Meta is not the only company drinking the augmented kool-aid. Meta just announced a partnership with Solidworks, the largest 3D CAD software company in the world. And computing infrastructure companies such as Intel, NVidia, and Qualcomm are making heavy investments in Meta's underlying technologies - displays, GPUs, and transmission bandwidth for real-time operations - on the belief that AR and VR will be a huge part of their business in five years.
Hospitals are also experimenting with augmented reality. Stefano's personal passion for healthcare leads him to brainstorm ways to improve surgery and patient care with Gerry Grant, Chief of Neurosurgery at Stanford's Lucille Packard Hospital. "Families of children going into surgery - for example, for limb or facial reconstructions - struggle to understand what the outcome is going to be like," Stefano explains. "You can use augmented reality to overlay images on their bodies. Seeing the outcome on an actual body is much more informative than seeing it on a screen."
Professional athletes are also making use of VR and AR to improve their results, with companies such as StriVR, Halo Neuroscience, and Kadho Sportsproving it out. Baldassi explains, "I started my Neuroscience career because I wanted to be the Olympic coach of the National Italian Volleyball team. Maybe if we get something from the glasses, volleyball rallies will last 5 minutes, not 20 seconds. I want players to be players, not robots. The goal is to have have fun playing at higher and higher levels."
Wrapping It Up
While it's clear that augmented reality has massive potential to positively impact society, it's still a few years away from becoming truly mainstream. "Our technology is currently at the developer kit level," says Baldassi. "It will be another 3 years or so before we hit prosumer. And prosumer is BIG."
But will Stefano's Meta gamble pay off? After all, Stefano left a full tenure-for-life position at the University of Florence to join this small startup, with a downside risk that none of it will go as planned. But for true visionaries such as Stefano and his team of radical neurohackers dreaming about the next human revolution, perhaps it's more about the journey than the outcome.
"In the trajectory of what technology can give to humans, we've gone down," Stefano says. "But now we have a chance to go up again, to return to our natural selves. It may sound simplistic, but dreams are simple. I want to design a machine that will help people be more human."
And that would be a very beautiful thing, indeed.
https://transcend.ghost.io/you-are-the-operating-system-metas-stefano-ba...
