Human Sensing (Understanding Human Behavior from Sensors)

How monitoring behaviors helps in improving medical diagnosis, evaluating treatment effects and early detection of some medical conditions

Dr. De la Torre sees a promising future for personal agents that improve medical diagnosis and assist in understanding treatment effects through behavioral analysis. Since coming to Carnegie Mellon University he has conducted research on behavioral systems that can diagnose and monitor different physical and mental conditions from a variety of sensors.

Dr. De la Torre leads two main lines of work:

1) Facial Image Analysis: The face is one of the most powerful channels of nonverbal communication. Facial expression provides cues about emotion, intention, alertness, pain, personality, regulates interpersonal behavior, and communicates psychiatric and biomedical status among other functions. The Human Sensing lab develops real-time automatic systems for facial expression analysis, facial recognition, facial feature tracking, and facial attributes. Applications include driver distraction detection, depression assessment from audio-video, pain analysis, video de-identification, and retail sentiment analysis.

2) Wearable sensing: The Human Sensing laboratory is interested in providing quantitative behavioral measurements of different psychical and medical conditions. Currently, they are working on continuously monitoring Parkinson's using wearable accelerometers, stress and fatigue using heart-rate monitors and galvanic skin response.

This sensor analysis is proving to help with diagnosing symptoms for the following health conditions:

Depression: One major limitation to the diagnosis and treatment of mental health disorders, such as depression, is the lack of objective measures to aid in the diagnosis and to assess the effects of therapy. Developing objective diagnostic and follow up methods that complement the work of mental health professionals would greatly improve diagnosis and treatment of depressive disorders. In collaboration with Professor Jeffrey Cohn at the University of Pittsburgh (UPitt), Dr. De la Torre's team  have shown that facial behavior and vocal prosody are highly correlated with a criterion measure of depression severity, the Hamilton Rating Scale. These findings suggest the feasibility of an objective measurement of depression.

Parkinson's: The symptoms of patients with PD such as tremors and dyskinesia fluctuate during the day depending on the timing and dosage of medication intake. The knowledge of these patterns could be very useful for PD treatments including individualization of drug therapy over time and evaluation of the potential candidates for deep brain stimulation. Due to the limited frequency and duration of clinic visits, a detailed assessment of medication effects is difficult for healthcare providers to quantify and often relies on self-reports by the patients. To address this problem, Dr. De la Torre's team have developed a wearable monitoring system composed of five wearable tri-axial accelerometers (wrists, ankles, and waist) capable of logging data continuously for 20 hours. The wearable system tracks medication cycles throughout the day at home and provides valuable clinical feedback. This work is done in collaboration with Professor Hodgins at CMU.

Stress and Fatigue: Assessing stress in natural environment is beneficial for understanding the emotional behavior of human beings. Existing studies have shown that psychosocial stress can be assessed via self-report measurement, and such measurement is associated with biological and behavioral indices relevant to health. Self-report measurement, however, requires a significant response cost on the participant, making frequent collection of such measurement impractical. If the stress assessment can be conducted on a continuous basis without requiring self-reports, it will significantly expand the usage of the assessment and benefit the research on stress and health.  In collaboration with Joao Paulo Cunha from the University of Porto (Portugal) they are developing smart techniques for detection of stress and fatigue in fireman.

Dr. De la Torre's other laboratory is the Component Analysis Laboratory, where they develop techniques to efficiently and robustly learn models from large amounts of high dimensional static and time-series data. The techniques that they develop are targeted to capture subtle human behavior.


Dr. De la Torre completed his Ph. D. in Spain and began his career with a permanent position lecturing in La Salle School of Engineering in Barcelona in 1997. However, his passion for research never waned and he wanted to advance his career, so he became a Postdoc at Carnegie Mellon University (CMU) in 2003. During his Ph.D. he worked on teaching computers to see, when he began working on the Caremedia Project at CMU he saw that they were providing  meaningful and manageable information resources to enable more complete and accurate assessment, diagnosis, treatment, and evaluation of behavioral problems for the elderly. During his Ph.D. he did mostly theoretical work, but the Caremedia project, made him realized that all this theory could be applied to improve the quality of life of elderly people, people with disabilities and in general patients with different medical conditions. Currently, he is directing the Component Analysis Laboratory and the Human Sensing Laboratory.

Dr. De la Torre has educated over 130 students in the last ten years. He feels as a mentor and fellow researcher it is important to share his enthusiasm for research and machine learning algorithms with students. Currently, he leads a group of eight Ph.D. students, one project scientist, three M.S. students and about 12 visiting scholars.  Over the last nine years he has supervised the research activity of seven postdoctoral fellows, nine Ph.D. students, nine M.S. students, and 105 visiting scholars (70 undergraduate students, 26 graduate visiting students, and nine visiting professors) in the interdisciplinary areas of computer vision and machine learning, see Sixteen of the visiting scholars were women. Nine undergraduate students won a highly competitive Spanish scholarship (e.g., Fulbright, La Caixa, Caja Madrid) to earn a M.S. in the United States. For most of the undergraduate visiting scholars this was their first research experience. 29 undergraduates (out of 65) published their first paper under Dr. De la Torre's mentorship, and 36 are pursuing their graduate studies in top American and European universities.


Matrix Completion for Weakly-supervised Multi-label Image Classification


Sequential Max-Margin Event Detectors


Spatio-temporal Matching for Human Detection in Video


Complex Non-Rigid Motion 3D Reconstruction by Union of Subspaces


Structured Prediction for Event Detection


Max-Margin Early Event Detectors


Unifying Nuclear Norm and Bilinear Factorization Approaches for Low-rank Matrix Decomposition



Best Student Paper Award, 2012

Coolest Faculty of the Year, 2008

50 Finest of Pittsburgh, 2008