Alexa and Assistant Square Off

In the past month or so I came to own Amazon Alexa and Google Assistant devices. I purchased a 7 inch Fire tablet on a Black Friday special to run a home automation system. Then I purchased both a Google Home and Pixel phone. The Pixel and Home arrived this week, meaning I now have both Alexa and Assistant at my disposal; of course I immediately wanted to do a side by side test.

I have tried to trick Google Now. I mentioned in an earlier post that Google wasn’t stumped when I asked it to show me pictures of Blue Helmets. So, how would Alexa and Assistant do when faced with the same questions?

Experiment Design 

Since I cannot figure out how to wake Alexa on the Fire tablet by voice (whether this is user error or a short coming of the system is beyond me, but a problem with the device either way), I decided to use Assistant on my Pixel, so that I could summon both by touch, to make the experiment as equal as possible. 

I set up the test by drafting a set interactions (commands and questions) and expected outcomes, grouped them by theme. I included some informational questions as well as commands to perform tasks for me. Also, there are some functions (e.g. email) that I wanted to test but couldn’t because I don’t have the Fire logged into my Google account.

I administered the test by posing each interaction, one by one, alternating between systems, using the same wording for both.

Results

Both systems could satisfy the basic informational interactions, such as providing the date and headlines. They also could perform the task interactions like setting alarms and timers. They aced the conversational test, where I first asked who the queen of England is, followed by asking her age. Google Assistant did a little better with many of the challenges, for example, while both systems could set a ten minute timer, Google Assistant accurately gave the timer a label to boot.

Screenshot of Google Assistant.
Google Assistant understands the figurative reference to UN troops.

Amazon Alexa used Bing to power its searches rather than Google, as Assistant did. Assistant had no problem with translations and it handled transportation questions with ease, even automatically launching Maps. Assistant got the Blue Helmets question, no sweat, but Alexa was flummoxed.

Amazon Alexa tripped up when it came to describing Blue Helmets as UN peacekeeping forces.
Amazon Alexa tripped up when it came to describing Blue Helmets as UN peacekeeping forces.

Alexa has Amazon’s product catalog behind it, and immediately followed up my battery challenge by asking if I wanted to place an order (when I said “no” to the first product offering, it moved on to another battery listing). Google simply pulled up a link to Amazon.

Scores

Of the 25 interactions I drafted, I was able to pose 20 in the test. Alexa got 12 correct, or 60%, a D-. Assistant got 18 correct, earning 90%, an A-. These findings are consistent with other comparisons I’ve read, even the ratio of the scores (Assistant beat Alexa by 50%).

Verdict

Assistant has the full force of Google to respond to challenges, a decided advantage. Alexa excelled when it came to placing an order for a product, to be expected when considering the company’s core competencies. Also, although conversational capabilities were just pushed to Alexa, it did well. As its third party skills weren’t tested, I cannot speak to how well it leverages them.

All told, Google’s Assistant is superior both in what it can do out of the box and in how it does it.

In the interest of full disclosure, I own both Alexa and Assistant devices.

© Peter Roehrich, 2017

Author: Peter Roehrich

I'm a railroader, business analyst, and trained scientist.

One thought on “Alexa and Assistant Square Off”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s