Will artificial intelligence transform communication tools for the non-verbal?
by Amy S.F. Lutz | Originally posted on Psychology Today
The first app to use artificial intelligence to power an alternative and augmentative communication (AAC) tool for profoundly autistic and other minimal- and non-speaking users was recently unveiled by Howard Shane, a professor at Harvard Medical School and the former director of the Autism Language Program and Center for Communication Enhancement at Children’s Hospital in Boston.
QuickPic AAC was created by Shane and Christina Yu from Boston Children’s Hospital and Mauricio Fontana de Vargas, a human-computer interaction (HCI) research scientist. The app facilitates the instantaneous generation of topic displays—sets of vocabulary relevant to a particular image—that used to take teachers and speech language pathologists up to half an hour to create. For example, if a therapist wanted her client to work on building sentences and answering questions about a birthday party, QuickPic could search the web for a photograph of a child blowing out birthday candles or even upload a photograph from that child's birthday party. Then AI would interpret the image and arrange it on a display alongside picture icons representing relevant words the client might choose, such as girl, boy, bake, cut, light, blow, yummy, cake, etc. (see accompanying image for an example). A facial recognition feature allows the inclusion of friends and family members.
“AI doesn’t make the sentence,” Shane explained to me in an interview. “The child still has to choose from the icons to build a sentence.”
For profoundly autistic children—those whose vocabularies are typically dominated by nouns—this can be very difficult. But it’s a critical skill. Shane notes that research has found that syntax is “like the holy grail… If we can teach autistic kids syntax, we can greatly expand their ability to understand and use language.” Since many autistic kids have better visual than auditory processing, tools like topic displays can be particularly effective. Building them through an app not only frees up the time of speech language pathologists for more direct client interaction, but it increases accessibility to parents, teachers, and others who may want to generate new topic displays to promote conversation in more natural settings.