Introduction
On January 31, Microsoft Research announced via its social media channels that their Seeing AI app, already a very useful tool for blind users, now included indoor wayfinding on its World Channel. The World Blind Herald staff found this to be incredibly exciting, so, within 20 minutes of the announcement, we jumped into testing it out as much as we could in a variety of settings ranging from simple situations around our own homes to stress testing the software in a pretty complicated shopping mall with a lot of things going on in a lot of different directions.
It should be noted that this is the first release of this feature in Seeing AI is very explicit that this new software is experimental. The Microsoft description of this experiment is quite specific that it works best in places where there is little movement and goes so far as to say that a shopping mall where there are a lot of people and a lot of motion is beyond its current capabilities. We like to push the limits to see how far we can go with new technology and did both simple and relatively complex tests and, as you will read below, Seeing AI outperformed both our expectations and did better than the MS disclaimer would imply.
What follows are our first impressions of the software from several points of view, including those who travel extensively on business as well as those who travel only far enough to get things done. World Blind Herald plans on following this story as it unfolds, this article describes our initial testing and our readers can look forward to more about Seeing AI and its indoor wayfinding feature in upcoming issues. The next article will go into greater detail on the various features it provides and a detailed description of the user experience.
If you've been experimenting with the Seeing AI indoor wayfinding features, we'd love to hear from you. Please share your experiences with us via our contact form and we'll try to include them in upcoming articles.
Our Informal Testing Effort
We tested Seeing AI's indoor navigation in several scenarios, including simple tests in and around our own homes and more complicated settings including a hospital, a very complicated upscale mall, with a guide dog and with canes, and though Microsoft has made it clear that the indoor wayfinding feature is experimental, even when pushing the limits of the description provided by MS, we experienced terrific success with it This new feature We found that even in the mall, which was split-level and with lots of bends and curves and a lot of people moving around, we found that we were able to build two separate complicated routes, save them, and use them after save to get to where we were our destination.
The Basic Tests
When Laura Legendary, the WBH managing editor and social media coordinator saw the announcement that Seeing AI had added indoor wayfinding, she immediately informed the rest of the WBH staff and along with her own testing, Chris Hofstader and Sina Bahram started playing around with the app in and around their homes. We did this using four different iPhone models (an SE30, a 14 Pro, a 13 Pro and an iPhone 12 Mini), two of which include support for lidar two of which do not. Lidar is kind of like sonar but uses light instead of sound to determine distances and such. The testing with the phones that do not have the lidar support was interesting as it gave very good results even though the Microsoft documentation suggests that the phones that support lidar will do a better job. As the testing we did on those phones were relatively simple situations, we didn't notice much difference between the fancier phones and the one that didn't have the more advanced hardware.
These tests were all rather simple, we created basic routes in our own homes (a safe place to start as we're blind and, if something went wrong with the app, we wouldn't be lost) and on all four hardware configurations performed better than we anticipated, even given the fact that this is experimental software.
The Complex Settings
Even though Microsoft's documentation about this new feature suggested that it would work poorly in places where a lot of motion was present, we decided to see how far we could push the limits of the Seeing AI indoor wayfinding features. Chris Hofstader gave it its first complex test at a hospital near his home in St. Petersburg, Florida on an older iPhone SE30. As he was at the hospital to get some simple blood work done, he created a route from the hospital door to the test lab on this older model phone and the SeeingAi indoor wayfinding performed relatively well.
The following day, Chris and his wife drove across Tampa Bay to the International Mall in Tampa. There, Chris went to the Apple store and bought a new iPhone 14 Pro and a pair of AirPod Pro ear buds. After the phone and AirPods were set up, he and his sighted wife went into the mall and created his first route. For a complex setting, this was a relatively simple route and the mall wasn't especially crowded so, while there was motion from other people walking around, it was a slow time of day and the Seeing AI indoor wayfinding performed admirably.
On Thursday, Chris and his wife returned to the same mall and tried more stressful tests of the software. Again, the indoor wayfinding performed admirably in a situation where there was a fair amount of motion, inconsistent lighting, a lot of reflective surfaces and lots of noise which could be confusing for the user.
What Is A Route?
It is obviously impossible that Microsoft could have trained its AI on every inch of every indoor location in the world. Thus, the SeeingAI indoor wayfinding features need one to first create a route, save it and then follow the route from the starting point to the destination, save it and then load the route the user had saved to be guided to the destination. One of the best aspects of this feature is that routes can be shared. So, in the mall setting, Chris' sighted wife could leave him on a bench outside the Apple Store, walk using Seeing AI to create a route, shoot it to Chris' phone and he could open it and, not knowing where his wife had gone, follow the route to where she was located. Even though Microsoft's disclaimer about the app being experimental and warning against using it in places like malls, it did a pretty good job of getting him to find her. To say the least, this is incredibly impressive as we tried to push the limits and, while it definitely performed better in the simple situations, it did quite well in a complex environment and this is only the first few days after the software was released and we can expect it will improve over time.
When Seeing AI was first released a number of years ago and all of its features were experimental, it performed pretty poorly in some places. In a test Chris did with that first version, it said that his wife was a 90+ year old man (she's female and a lot younger than that) and it said Chris was a 45 year old woman (he does tend to have long hair) which didn't impress him at all. But, today, literally every member of the WBH team uses Seeing AI in any of a number of different ways. For most of us, it's our go to solution for OCR and a variety of other tasks and we've all enjoyed watching it improve consistently over time. We expect the indoor wayfinding feature to improve rapidly as that's the history we've witnessed with the other excellent Seeing AI features.
Hardware
While we were able to get some usable results on an iPhone SE30, the best results came on the iPhone 14 Pro, the iPhone 13 Pro (both have lidar) and on the iPhone 12 Mini which does not. The newer iPhone models have faster processors and better cameras so this is not at all surprising.
We also noted that the best results come when one is using Apple AirPod earbuds and recommend the AirPod Pro models as they include things like head tracking and allow for noise reduction while also being able to hear the ambient sounds around you. Seeing AI can follow ones head and know which way the user is pointing if they are using an AirPod Pro model and results with other earbuds or headphones is an unknown as we didn't test with any other brands yet.
Not Perfect Yet
As this is the first release of software that Microsoft won't even call "beta" but use the term "experimental" instead, it is by far not perfect but we oughtn't expect it to be either. It's great that MS is letting us give it a test drive while other companies may have kept it hidden in secret until it was further along in its development. Some of the problems we encountered, though, had more to do with bugs we found in the software's interface than in the AI used to perform the indoor wayfinding. For instance, on one of Chris' experiments at the mall, some aspects of the software like object detection worked great but somehow the wayfinding portion stopped working entirely. In one of Laura's tests, she tried to save a route and the software failed to do so. Once again, this is experimental software that only made it to the public less than an hour before we started our testing so bugs are to be expected and one who is testing this new version of SeeingAI should report any problems or anomalies back to Microsoft so their developers know what needs to be fixed; Microsoft is letting us play with an experimental piece of software, we should repay them with reports of our results in the field so they can a-dress the problems in future updates.
One issue with the current version of Seeing AI for indoor wayfinding is that one need walk around holding their phone in front of them. We understand that there are products on the market that allow one to attach an iPhone to their body for hands free use (apparently Aira users do this) but that is a kludge of a solution. If one needs to hold the phone in their hand, walking around is pretty awkward when one's other hand is grasping a cane or dog's harness. In Chris' tests at the mall, his dog wanted to take the corridor to the food court (Labrador's tend to want to head toward the smell of food) and correcting the dog with one hand holding a phone in front of him was a bit frustrating. I think we all agree that this would be best if it could be on a pair of smart glasses so the user needn't have the phone in their hand or somehow attach it to their clothing. Both Microsoft and Apple have smart glasses in the pipeline and we're looking forward to testing out Seeing AI when it's available in a head mounted device.
The Future
A couple of members of the WBH staff were at the Assistive Technology Industry Association (ATIA) conference this past week in Orlando, Florida. One of them had a conversation with Dr. Gregg Vanderheiden, a WBH advisor and a major figure in the history of AT. There, Dr. Vanderheiden recalled a conference in 2021, only two years ago that discussed new user interface concepts and the implications for accessibility that came with them. Vanderheiden said, "Two years ago, the consensus among the accessibility specialists was that we would have what would at best be described as a 'poor to mediocre' prototype of an AI based indoor wayfinding solution in about ten years." He continued, "It looks like Microsoft has beaten our expectations both in terms of time as they are 80% ahead of our presumed schedule and in terms of quality as it's already much better than just mediocre."
Sina Bahram, also a WBH advisor, said, "I wouldn't have believed it until I tried it out but this is the most important innovation in accessibility that we've seen in years. This is a tremendous game changer and it's only going to get better in the future."
Laura Legendary wrote after testing the software, "This is amazing, simply amazing!"
Chris Hofstader said, "This is the biggest step forward in accessibility innovation since Apple released the iPhone 3GS and has totally changed my mind as to the state of artificial intelligence in the accessibility arena. Innovation has been pretty slow related to technology blind people use and this is a giant step forward."
With this release of the first ever entirely AI driven indoor wayfinding solution, AT has entered the artificial intelligence age in a big way and Seeing AI, even in its experimental stage, is the looking glass through which we can see the future. The combination of Microsoft's software and Apple's mobile hardware, we're stepping into the next generation and it's happening faster than expected.
Conclusions
It's hard to form too many conclusions about a very new bit of technology only days and a few hours of testing since it was released but the WBH team as a group is incredibly impressed with this software as it stands today. As we've been following the progress of Seeing AI since the day it was First released and have only seen it get better over time, we expect this trend to continue over the next few years and that this will be the model for indoor wayfinding into the future.
–End