When you have a successful interaction with conversational AI, it seems like real communication is happening. You say what you need, the machine gives you a reasonable response. The Natural Language Processing algorithms that power these interactions are doing their job and it doesn't seem important how they got to the right response.
All this changes when the conversation hits a snag. When conversational AI fails, it's suddenly obvious that the machine doesn't really get what's happening. These failures are more than simple misunder-hearing (you said Boston, it heard Austin).
Instead, these are cases when you ask for the same thing in a slightly different way than last time, and the machine responds like it has no clue what you're looking for. Why can't the machine figure out that the two requests were for the same thing?
Part of the answer is that today's NLP is built on statistical patterns of words and phrases without considering the meaning of the words. Conversational AI sometimes gives us meaningless responses because it doesn't understand meaning, so our intuition that the machine doesn't get it are absolutely valid. These failures are annoying if you're trying to turn on a smart bulb or listen to music, but they can be much more serious in financial or healthcare applications.
Researchers have developed a new way to test the common sense of NLP algorithms and demonstrate that even with recent advances, we have a long way to go.