Results for single-accelerometer movelet approach
For resting activities (lying and standing), all accelerometers provide accurate predictions. For normalWalk combined and fastWalk combined, the wrist-worn accelerometers perform as well as the hip-worn accelerometers. The right wrist-worn and left wrist-worn accelerometers outperform hip-worn accelerometers in predicting normalWalk Swing, normalWalk noSwing, fastWalk Swing, fastWalk noSwing, and chairStand. Write and cards belong to the group of upper body activities while sitting. Accelerometers worn at three different positions yield very accurate predictions for writing (see the red, blue, and green boxplots in Figure 1 corresponding to write). Right wrist accelerometers falsely predict on average 10.2% of cards to be write. For cards, left wrist-worn accelerometers provide higher median prediction accuracy than right wrist-worn accelerometers, and both outperform the hip-worn accelerometers. Hip-worn accelerometers do not record the subtle movements of the hands and often incorrectly classified cards as dough (5.4% average across participants) and foldTowel (5.3% average across participants). On average, 12.8%, 13.1%, and 9.4% of dressing is falsely classified as foldTowel based on the hip-worn and right- and left wrist-worn accelerometers, respectively. For activities dough, washDish, vacuum, dressing, foldTowel, and shop, all three accelerometers show lower median prediction accuracy and larger variability across participants.
Results for integrative movelets approaches
For the activities lying, normalWalk combined, write, stand, normalWalk Swing, fastWalk Swing, and cards, the expanded movelets approach yields the highest prediction accuracy with the least variability across participants. A substantial increase in prediction accuracy for cards is observed for the expanded movelets approach. To provide a visual representation, Figure 2 shows the matching processes of two participants for whom the expanded movelets approach provides better prediction for cards than all the single-accelerometer movelet approach on its own. The movelets voting approach is inferior to either one of the single-accelerometer movelet approach or the expanded movelets approaches for all the activities with the exception of dough, washDish, vacuum, and shop. The movelets decision tree provides relatively good performance for lying, write, and stand. For other activities, it tends to underperform.
We conducted a four-fold cross-validation to further evaluate the proposed methods. More precisely, every time, we used another part of the data for training and testing and have investigated the sensitivity of the proposed approach. For activities lying, walking, stand, chairStand, and write, the within-subject variability of the prediction accuracy is much smaller than the between-subject variability. For most of the household activities, the within-subject variability of prediction accuracy is larger and the average prediction accuracy over folds is also generally low; this is due to the ambiguity of submovements that are shared among multiple household activities. Results suggest that for activities lying, walking, stand, chairStand, and write, the movelet-based approaches provide highly consistent results when using different training sets.
Discussion on movelets approaches
This work provides movelet-based prediction of activity type at the second-level temporal resolution using triaxial accelerometers placed at the right hip and left and right wrists. Compared with feature extraction-based methods, the movelet-based methods have two advantages. First, instead of extracting features, the movelet algorithm preserves the original signal that can be visually inspected. The matching process between known and unknown movelets mimics the natural human pattern recognition, which makes the process transparent. If prediction fails in a 1-s interval, human visual inspection can usually reveal the reason. Second, movelets can achieve high prediction accuracy at the second level even using very limited training data (several seconds per activity type). This is the first approach that can achieve such performance although remaining fully transparent and easy to understand.
To illustrate how these advantages work in practice, we present several examples here. First, it was found that wrist-worn accelerometers predict walking better than hip-worn accelerometers. This is unexpected and counterintuitive, because these movements are fundamentally performed by producing lower body acceleration. As a second example, the results showed that the accelerometers worn at the left wrist outperformed the one worn at the right wrist in recognizing cards, which is also unexpected. To investigate these, Figure 3 displays the matching processes for normalWalk, fastWalk, and cards. Activity labels are coded in different colors; the annotated labels and predicted labels are plotted in parallel accompanying the original signals. Given a time point, if the annotated label and the predicted label are of the same color, then the activity type is correctly predicted.
Consider the matching process for walking with and without arm swing as measured at the hip (first row, left panel) and right wrist (second row, left panel) in Figure 3. As the person transitions from normal walk without arm swing to normal walk with arm swing, the time series associated with hip movement do not display visually observable changes. In contrast, the wrist accelerometry indicates a strong change. Most interestingly, the blue accelerometry curve shifts to a much higher level than before, probably because of the change in the angle of the accelerometer. Such strong angle changes can be easily observed and detected using movelets and should explain how information is being combined. The single-accelerometer movelet approach is confused between the two types of movements when using the hip data only. In contrast, the predicted labels based on wrist data are almost perfectly accurate. A similar story holds for fast walking with or without arm swing (row 3 and 4 panels). Now examine the right panels in the first and second rows of Figure 3. It is shown that the right wrist accelerometer yields more variable signal than the left wrist one, which makes it harder to distinguish the right wrist accelerometer signals from other upper body activities. As a third example, the prediction accuracy for the upper limb activities while standing was lower and yielded larger variability across participants. This probably happens because of the high level of overlap in movement and ambiguity of some submovements across labeled activities. At the same time, the raw signals of these activities are also similar with visual inspection of the matching process.
Three integrative movelets approaches were proposed to incorporate information from multiple accelerometers. The expanded movelets approach yields the best overall performance among the three integrative movelets approaches. Movelets voting is conceptually straightforward. Because the integration of information occurs after using single-accelerometer movelet approach for each accelerometer, it can simultaneously process acceleration information from different sources and save computing time. At the same time, integration with majority votes of categorical activities labels loses some of the rich information embedded in the original signals. The expanded movelets approach merges all available information and provides the flexibility of weighting different information sources. This method yields exceptional prediction in well-controlled environments, although it may be more prone to errors when one of the devices malfunctions or moves to a different position on the body.
The weights on the sources can be assigned depending on activity types. Determining optimal weights for different sources is not covered in this article, but it will be investigated in the future. The movelets decision tree approach integrates information adaptively. An important assumption is that the top level of the decision tree is well designed to provide coarse discrimination between activity groups, whereas lower level decisions are well designed to make within-group predictions. In the movelets decision tree approach, the prediction error in the first level is propagated into the second level. Thus, designing the tree hierarchy is rather delicate and, likely, application specific. To increase prediction accuracy from the simultaneous use of multiple accelerometers, a careful selection of integrative algorithms is required.
The order of magnitudes of local average accelerations among the three axes is crucial for detecting and differentiating various postures; this happens because the order is a proxy of the orientation of the device relative to earth gravity. For example, our results showed that high accuracy was achieved for lying and standing. The visual inspection of the matching process reveals that the main difference between the accelerometry signals for lying and standing is that the local average of individual time series is different in magnitude and rank. While standing still, the gravity would appear as acceleration mainly along the AnteriorPosterior axis. Gravity affects differently each axis, and the size of the effect depends fundamentally on the angles the axes of the accelerometer form with the gravity direction. Although far from being a perfect proxy for the position, this is enough to differentiate between standing and lying. This is a case where the variability of time series along their long-term averages is of secondary importance, whereas the discrimination between the two resting positions is done by the shift in the relative magnitude of the mean functions.
Our findings have potential implication on the accelerometers’ placement decisions in epidemiological studies. First, both accelerometers worn at the dominant hand and nondominant hand can predict lying, standing, normal walking, and fast walking as well as the hip-worn accelerometers. This provides support for using wrist-worn accelerometers. Second, handedness should also be a consideration in accelerometer placement, which can affect prediction performance. Our results showed that in activities like cards where the dominant hand moves more, the nondominant hand accelerometers yielded higher prediction accuracy. Third, the decision of placement should be customized to the study goals. The results showed that hip-worn accelerometers performed poorly in distinguishing walking with arm swing versus walking without arm swing, whereas wrist-worn accelerometers yielded good results. Consider a scenario when investigators decide to use hip-worn accelerometers and are interested in distinguishing between different types of walking. It will be quite difficult to differentiate between walking normally and walking carrying a small object (no arm swing). Thus, it seems reasonable to simply define a label called normal walking that includes both arm swing and no swing. Alternatively, a wrist accelerometer could be used instead or in addition to the hip one. It was also shown that trunk movements such as walking are easily recognized by all accelerometers, whereas finer movements that occur at the extremities of the human body are predominantly captured by wrist-worn accelerometers.
We have seen a trend that more and more devices are now designed to output raw triaxial data and more and more researchers have started to collect the raw data. There is also increasing interest in evaluating data that are being lost by analyzing the coarse activity data at the hour or day level versus the raw accelerometry data recorded at the subsecond level. The movelets approaches proposed here provide a self-consistent and transparent framework for thinking about and quantifying the data at the subsecond level. The movelets approaches allow the scientist to understand the complex measurement, have access to the entire processing pipeline, and access different levels of data compression via reproducible code and verifiable results.
In this study, all movement predictions were obtained from participants in the research clinic wearing devices installed by trained technicians. The activities are intended to represent activities that happen in the participant’s own environment. As much as one tries to standardize laboratory experiments, the data are likely to provide only a partial snapshot of the heterogeneous activities individuals perform in their own houses. It remains unclear how in-lab data prediction algorithms perform in free-living environments, especially in the absence of labeled in-home data on hundreds of individuals. Also, we have not yet investigated how well methods could be trained on one or multiple participants and then applied to other participants. However, these issues provide opportunities for future work.
This research was funded in part by the Intramural Research Program of the National Institute on Aging. Dr. Crainiceanu was supported by the National Institute of Neurological Disorders and Stroke under Award Numbers R01NS060910 and R01NS085211 and by the National Institute of Mental Health of the National Institutes of Health under Award Number R01MH095836.
This work was also supported by the National Institute of Health through the funded Study of Energy and Aging Pilot (RC2AG036594), Pittsburgh Claude D. Pepper Older American’s Independence Center Research Registry (NIH P30 AG024826), a National Institute on Aging Professional Services Contract (HHSN271201100605P), a University of Pittsburgh Department of Epidemiology Small Grant, and the National Institute on Aging Training Grant (T32AG000181).
This work represents the opinions of the researchers and not necessarily that of the granting organizations.
All the authors have no conflict of interest.
The present study does not constitute endorsement by the American College of Sports Medicine.
1. Atienza AA, King AC. Comparing self-reported versus objectively measured physical activity
behavior: a preliminary investigation of older Filipino American women. Res Q Exerc Sport
. 2005; 76 (3): 358–62.
2. Bai J, Goldsmith J, Caffo BS, Glass T, Crainiceanu CM. Movelets: a dictionary of movement. Electron J Stat
. 2012; 6: 559–78.
3. 3. Bao L, Intille SS. Activity recognition from user-annotated acceleration data. In: Proceedings of the 2nd International Conference on Pervasive Computing
, 2004; 1–17.
4. Breiman L. Bagging predictors. Mach Learn
. 1996; 24 (2): 123–40.
5. Bussmann JB, Martens WL, Tulen JH, Schasfoort FC, van den Berg-Emons HJ, Stam HJ. Measuring daily behavior using ambulatory accelerometry: the activity monitor. Behav Res Methods Instrum Comput
. 2001; 33 (3): 349–56.
6. Choi L, Liu Z, Matthews CE, Buchowski MS. Validation of accelerometer
wear and nonwear time classification algorithm. Med Sci Sports Exerc
. 2011; 43 (2): 357–64.
7. 7. Hills J, Lines J, Baranauskas E, Mapp J, Bagnall A. Time series
classification with shapelets. J Data Mining Knowledge Disc
. 2014; 28 (4): 851–81.
8. Hu B, Chen Y, Zakaria J, Ulanova L, Keogh E. Classification of Multi-Dimensional Streaming Time Series by Weighting each Classifiers Track Record
. Dallas (TX): International Conference on Data Mining (ICDM); 2013.
9. Kozey-Keadle S, Libertine A, Lyden K, Staudenmayer J, Freedson P. Validation of wearable monitors for assessing sedentary behavior. Med Sci Sports Exerc
. 2011; 43 (8): 1561.
10. Krause A, Sieiorek DP, Smailagic A, Farringdon J. Unsupervised, dynamic identification of physiological and activity context in wearable computing. In: Proceedings of the 7th International Symposium on Wearable Computers
. White Plains (NY): IEEE Computer Society; 2003. pp. 88–97.
11. Lam L. Classifier combinations: implementations and theoretical issues. Lect Notes Comput Sci
. 2000; 1847: 77–86.
12. Mannini A, Intille SS, Rosenberger M, Sabatini AM, Haskell W. Activity recognition using a single accelerometer
placed at the wrist or ankle. Med Sci Sports Exerc
. 2013; 45 (11): 2193–203.
13. Pober DM, Staudenmayer J, Raphael C, Freedson PS. Development of novel techniques to classify physical activity
mode using accelerometers. Med Sci Sports Exerc
. 2006; 38 (9): 1626–34.
14. Preece S, Goulermas J, Kenney L, Howard D, Meijer K, Crompton R. Activity identification using body-mounted sensors: a review of classification techniques. Physiol Meas
. 2009; 30 (4): R1–33.
15. Qian H, Mao Y, Xiang W, Wang Z. Recognition of human activities using SVM multi-class classifier. Pattern Recognit Lett
. 2010; 31 (2): 100–11.
16. Rakthanmanon T, Keogh E. Fast shapelets: a scalable algorithm for discovering time series
shapelets. In: Proceedings of the 13th SIAM International Conference on Data Mining (SDM)
, 2013, Austin (TX).
17. Ravi N, Dekar N, Mysore P, Littman ML. Activity recognition from accelerometer
data. In: Proceedings of the Seventeenth Conference on Innovative Applications of Artificial Intelligence
. 2005; 3: 1541–6.
18. Sirard JR, Trost SG, Pfeiffer KA, Dowda M, Pate RR. Calibration and evaluation of an objective measure of physical activity
in pre-school children. J Phys Act Health
. 2005; 2 (3): 345–7.
19. Staudenmayer J, Pober D, Crouter S, Bassett D, Freedson P. An artificial neural network to estimate physical activity
energy expenditure and identify physical activity
type from an accelerometer
. J Appl Physiol
. 2009; 107 (4): 1300–7.
20. Staudenmayer J, Zhu W, Catellier DJ. Statistical considerations in the analysis of accelerometry-based activity monitor data. Med Sci Sports Exerc
. 2012; 44 (1 Suppl): S61–7.
21. Troiano RP, Berrigan D, Dodd KW, Msse LC, Tilert T, McDowell M. Physical activity
in the united states measured by accelerometer
. Med Sci Sports Exerc
. 2008; 40 (1): 181–8.
22. Trost SG, McIver KL, Pate RR. Conducting accelerometer
-based activity assessments in field-based research. Med Sci Sports Exerc
. 2005; 37 (11 Suppl): S531.
23. Trost SG, Wong WK, Pfeier KA, Zheng Y. Artificial neural networks to predict activity type and energy expenditure in youth. Med Sci Sports Exerc
. 2012; 44: 1801–9.
24. Vries De SI, Garre FG, Engbers LH, Hildebrt VH, Buuren SV. Evaluation of neural networks to identify types of activity using accelerometers. Med Sci Sports Exerc
. 2011; 43: 101–7.
25. Welk GJ, Blair SN, Jones S, Thompson RW. A comparative evaluation of three accelerometry-based physical activity
monitors. Med Sci Sports Exerc
. 2000; 32: 489–97.
26. Wolpert DH. Stacked generalization. Neural Netw
. 1992; 5: 241–59.
27. Ye L, Keogh E. Time series
shapelets: a new primitive for data mining. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
, 2009, Paris France.
28. Zhang S, Rowlands AV, Murray P, Hurst TL. Physical activity
classification using the GENEA wrist-worn accelerometer
. Med Sci Sports Exerc
. 2012; 44 (4): 742–8.
29. Zheng Y, Wong WK, Guan X, Trost S. Physical activity
recognition from accelerometer
data using a multi-scale ensemble method. In: Proceedings of the Twenty-Fifth Innovative Applications of Artificial Intelligence Conference
Keywords:© 2014 American College of Sports Medicine
ACCELEROMETER; PHYSICAL ACTIVITY; SIGNAL PROCESSING; PATTERN RECOGNITION; TIME SERIES