HealthEdvisor is an interactive automated nurse, with the ability to explain medication and discharge instructions to patients. We are currently testing a ``teach-back'' learning paradigm, in which, after teaching patients about their medicines, Edna asks the patients to teach her, in return. Studies with human nurses suggest that teach-back enhances patient attention, comprehension, and recall; we are trying to determine whether similar benefits might be obtained by an interactive computer agent.
LittleBeats provides a remote, unobtrusive method to simultaneously assess physiological (ECG) and behavioral (audio, motion) streams of data on a large scale among young children and their caregivers in their home environments. We're using LittleBeats to study the development of kids in the context of CoVID-19.
In this paper, we showed that it's possible to define phonemes/phones by clustering the hidden units in an end-to-end speech recognizer. This paper showed that it's also possible to use voice conversion to separate the content embedding from speaker ID. In this paper, we showed that it's possible to automatically generate spoken descriptions of images in an unwritten language. We're now trying to combine these three ideas, using cycle-consistent speech recognition and synthesis to identify phonemes/phones in under-resourced languages.
Here are archives of some completed projects.