Wearable Cognitive Assistance (Gabriel)


Today, over 20 million Americans are affected by some form of cognitive decline that significantly affects their ability to function as independent members of society.  This includes people with neurodegenerative conditions such as Alzheimer’s disease (~4.5M) and mild cognitive impairment (>6M), survivors of stroke (~2.5M) and people with traumatic brain injury (~5.3M).  These numbers are expected to grow significantly due to an aging population and an increase in long-term post-traumatic stress disorders arising from occupational and social causes.  Cognitive impairment can manifest itself in many ways, including the inability to recognize people, locations and objects, loss of short- and long-term memory, and changes in behavior and appearance such as decreased attention to personal hygene. Among the many challenges faced by older Americans, cognitive decline often has the largest negative impact on them and their family members.  The potential cost savings from even modest steps towards addressing this challenge are enormous: it is estimated that just a one-month delay in nursing home admissions nationwide could save $1.12B annually.

Can information technology  reduce the burden of cognitive impairment?  Inspired by early examples of "smart spaces" such as the Aware Home at Georgia Institute of Technology, the Gator Tech Smart House at the University of Florida, and Elite Care's Oatfield Estates in Milwaukie, OR, we seek to greatly increase the scope and range of systems that offer cognitive assistance.  Our goal is to free such systems from the confines of a purpose-built smart space, while simultaneously enriching the user experience and assistive value to the user.  In other words, we aim to create cognitive assistive systems that can function ``in the wild'' with sufficient functionality, performance and usability to be valuable at any time and place.   Aided by such systems, commonplace activities such as going to a shopping mall, attending a baseball game, or transacting business downtown should become attainable goals for people in need of modest cognitive assistance.    Here is a hypothetical scenario that is suggestive of the future we hope to create:

Ron is a young veteran who was wounded in Afghanistan and is slowly recovering from traumatic brain injury.  He faces an uncertain future, with a lack of close family nearby and with limited financial resources for professional caregivers.  He has suffered a sharp decline in his mental acuity and is often unable to remember the names of friends and relatives.  He also frequently forgets to do simple daily tasks.  Even modest improvements in his cognitive ability would greatly improve his quality of life, while also reducing the attention demanded from caregivers.  This would allow him to live independently in dignity and comfort, as a productive member of his community.

Fortunately, a new experimental technology may provide Ron with cognitive assistance.  At the heart of this technology is a lightweight wearable computer built into the frame of Ron's eyeglasses. Integrated with the eyeglass frame are a camera for scene capture and bone-conduction earphones for audio feedback.  These hardware components offer the essentials of a system to aid cognition when they are combined with software for scene interpretation, face recognition, context awareness and voice synthesis.  When Ron looks at a person for a few seconds, that person's name is whispered in his ear along with additional cues to guide Ron's greeting and interactions; when he looks at his thirsty houseplant, ``water me'' is whispered; when he look at his long-suffering dog, ``take me out'' is whispered.  Ron's magic glasses travel with him, transforming his surroundings into a helpful smart environment.

The hardware for "magic glasses" is almost here.   Google's Project Glass is a good example.    The First Person Vision (FPV) system from the Quality of Life Technology Center at Carnegie Mellon is another example.   Here is a cool YouTube video of the very first wearable cognitive assistance application that we have built.   Check out the papers on the "Publications" link off the home page.