In the past month or so I came to own Amazon Alexa and Google Assistant devices. I purchased a 7 inch Fire tablet on a Black Friday special to run a home automation system. Then I purchased both a Google Home and Pixel phone. The Pixel and Home arrived this week, meaning I now have both Alexa and Assistant at my disposal; of course I immediately wanted to do a side by side test.
I have tried to trick Google Now. I mentioned in an earlier post that Google wasn’t stumped when I asked it to show me pictures of Blue Helmets. So, how would Alexa and Assistant do when faced with the same questions?
Since I cannot figure out how to wake Alexa on the Fire tablet by voice (whether this is user error or a short coming of the system is beyond me, but a problem with the device either way), I decided to use Assistant on my Pixel, so that I could summon both by touch, to make the experiment as equal as possible.
I set up the test by drafting a set interactions (commands and questions) and expected outcomes, grouped them by theme. I included some informational questions as well as commands to perform tasks for me. Also, there are some functions (e.g. email) that I wanted to test but couldn’t because I don’t have the Fire logged into my Google account.
I administered the test by posing each interaction, one by one, alternating between systems, using the same wording for both.
Both systems could satisfy the basic informational interactions, such as providing the date and headlines. They also could perform the task interactions like setting alarms and timers. They aced the conversational test, where I first asked who the queen of England is, followed by asking her age. Google Assistant did a little better with many of the challenges, for example, while both systems could set a ten minute timer, Google Assistant accurately gave the timer a label to boot.
Amazon Alexa used Bing to power its searches rather than Google, as Assistant did. Assistant had no problem with translations and it handled transportation questions with ease, even automatically launching Maps. Assistant got the Blue Helmets question, no sweat, but Alexa was flummoxed.
Alexa has Amazon’s product catalog behind it, and immediately followed up my battery challenge by asking if I wanted to place an order (when I said “no” to the first product offering, it moved on to another battery listing). Google simply pulled up a link to Amazon.
Of the 25 interactions I drafted, I was able to pose 20 in the test. Alexa got 12 correct, or 60%, a D-. Assistant got 18 correct, earning 90%, an A-. These findings are consistent with other comparisons I’ve read, even the ratio of the scores (Assistant beat Alexa by 50%).
Assistant has the full force of Google to respond to challenges, a decided advantage. Alexa excelled when it came to placing an order for a product, to be expected when considering the company’s core competencies. Also, although conversational capabilities were just pushed to Alexa, it did well. As its third party skills weren’t tested, I cannot speak to how well it leverages them.
All told, Google’s Assistant is superior both in what it can do out of the box and in how it does it.
In the interest of full disclosure, I own both Alexa and Assistant devices.
© Peter Roehrich, 2017