Human Motion Capture and Identification for Assistive Systems Design in Rehabilitation. Pubudu N. Pathirana

patients were generally satisfied to wear such a system. However, some patients were concerned about their privacy, as well as how others might think about them particularly in a public area, thereby showing a bit of anxiety and unwillingness to use this system. Furthermore, the strap made patients uncomfortable and difficult to wear when by themselves. Last, but not least, feedback was necessary during monitoring so that patients knew the system was working properly.

Figure 1.9 Locations of five sensors worn by a subject. Source: Cancela et al. [61].

1.4 Model‐based State Estimation and Sensor Fusion

      Kinematic model‐based state estimation can often be used to estimate parameters of interest while combining different information available typically in real‐time applications. Let the N sensor measurements be and the unknown states of the dynamic system be . The dynamic system can be modelled in the form of a state space model,

      (1.2)

      (1.3)

      where A, B and C denote the system matrices and w is the uncertainty input.

      Three filters are commonly used to achieve the goal.

      First of all, a Kalman filter is used extensively as an optimal state estimator under Bayesian assumptions and is extended to cover non‐linear dynamic and measurement models with stochastic, normal or Gaussian distributed noise [29, 210, 231]. This filter was introduced with the purpose of addressing the limitations of other filters in solving the Wiener problems [163]. Various versions of the Kalman filter, ranging from optimal (also called standard) to extended and various unscented versions pertaining to both linear and non‐linear systems have covered a large number of application scenarios. The proposed system is linear with position, velocity and acceleration acting as state variables, enabling the implementation of an optimal Kalman filter for data fusion. Detailed information about the Kalman filter can be found in Chapter 6. Although Kalman filtering is pervasive with the underlying assumptions of Gaussian noise or system uncertainty distributions, when the uncertainties deviate from these assumptions, the performance degradation can be significant and a more generic uncertainty assumption is warranted.

      Secondly, particle filters, which are a form of sequential Monte Carlo sampling, are widely used for state estimation problems with both linear and non‐linear dynamic systems [172, 383, 385]. They relax the single Gaussian assumption in the state and the measurement uncertainties, allowing the handling of more complex noise situations via sampling from multiple probability densities. When this function is a single Gaussian, the particle filter essentially simplifies to the standard Kalman filter. The application of a particle filter in our linear system is described in Chapter 6.

      1.4.1 Summary and challenges

      From the literature, it is clear that a number of sensors have been applied in telerehabilitation. For the evaluation of the approaches introduced in this thesis, we utilised Kinect as the example of affordable OBMCDs to collect data for the following two major reasons.

      First of all, Kinect is a non‐contact motion capture device. Patients do not need to attach any extra item to their bodies. As a result, they may be able to perform rehabilitation exercises or ADLs with more natural poses.

Secondly, compared to conventional RGB cameras, Kinect is able to track 3D positions of 20–25 joints throughout a human body. Therefore, without making much effort in processing images, data collected from a Kinect are much more convenient for further analysis. This is critical for automatically assessing the kinematic performance of patients in a telerehabilitation system without the presence of a professional therapist.

      However, the majority of affordable OBMCDs still suffer from a number of drawbacks. One is that it may not be accurate enough to track small movements, as well as the positions of joints occluded by other body parts. Secondly, its small viewing range limits its application in lower extremity rehabilitation, which, in many cases, involves a large moving area. A solution for these limitations is discussed in Chapter 2.

1.5 Human Motion Encoding in Telerehabilitation

      After capturing patients' movements, it is critical to reduce information so that the key features representing the characteristics of the movements can be selected. Therefore, before conducting an automated performance assessment, it is critical to extract these features by encoding human motions.

      1.5.1 Human motion encoders in action recognition

      Actually, this problem has been extensively studied in the field of human action recognition. For instance, Ren et al. [297] employed the silhouette of a dancer to represent his/her performance by extracting local features to control animated human characters. Wang et al. [372] obtained the contour of a walker from his/her silhouette to represent the walking motion. A spatio‐temporal silhouette representation, the silhouette energy image (SEI), and variability action models were used by Ahmad et al. [19] to represent and classify human actions. In both visual‐based and non‐visual‐based human action recognition of differential features, such as velocity and acceleration, motion statistics, their spectra and a variety of clustering and smoothing methods have been used to identify motion types. A two‐stage dynamic model was established by Kristan et al. [182] to track the centre of gravity of subjects in images. Velocity was employed as one of the features by Yoon et al. [390] to represent the hand movement for the purpose of classification. Further, Panahandeh et al. [272] collected acceleration and rotation data from an inertial measurement unit (IMU) mounted on a pedestrian's chest to classify the activities with a continuous hidden Markov model. Ito [154] estimated human walking motion by monitoring the acceleration of the subject with 3D acceleration sensors. Moreover, angular features, especially the joint angle and angular velocity, have been used to monitor and reconstruct articulated rigid body models corresponding to action states and types. Zhang et al. [397] fused various raw data into angular velocity and orientation of the upper arm to estimate its motion. Donno et al. [97] collected angle and angular velocity data from a goniometer to monitor the motions of human joints. Angle was also utilised by Gu et al. [129] to recognise human motions to instruct a robot. Amft et al. [26] detected the feeding phases by constructing a hidden Markov model with the angle feature from the lower arm rotation. Apart from the above, only a few have considered a similar approach of trajectory shape features such as curvature and torsion. For example, Zhou et al. [401] extracted the trajectories of the upper limb and classified its motion by computing the similarity of these trajectories.

      1.5.2 Human motion encoders in physical telerehabilitation

      Although