Standing Jumps at 120 BPM.

Participants were required to vertically jump sub-maximally from standstill once every four beats (0.5 Hz) twenty times and rest for twenty beats. This resulted in five lots of twenty jumps with ten second breaks in-between over the four minutes. The jump frequency and intervals between jumps was made to ensure that the physical activity remained aerobic. This exercise was chosen to emphasize the mechanical work of the lower limbs to raise the centre of mass.

Arm Swings at 112 BPM.

Participants were required to simultaneously swing both arms horizontally apart, return the arms to the side of the body, swing the arms forward, and then return the arms to side. The duration of each of these movements was one beat (0.53 s; 1.89 Hz). This exercise was chosen to emphasize the mechanical work of the upper limbs.

Squats at 72 BPM.

Participants were required to lower their body by bending their knees and holding their position for eight beats (6.67 s) and then stand upright for another eight beats (6.67 s). This exercise was chosen to test to what extent the energy expenditure of isometric muscle contractions to counter gravity would affect our predictive model.

Jumping Jacks at 76 BPM.

Participants were required to jump and move their feet apart while raising their arms above their head on the first beat, and jump and bring their feet together and lower their arms to side on the next beat (0.63 Hz). This exercise was chosen to have the maximum amount of mechanical work to raise the centre of mass and across all limb segments.

A 15-second moving average was used to smooth the $\stackrel{.}{\mathrm{\text{V}}}{\mathrm{\text{O}}}_2$ measurements. The value of 20.964 (kJ) was used as the energy equivalence of one litre of consumed oxygen [25] and was used to compute a metabolic power (kW). The average of the three minutes of standing was used as the baseline subtraction and the average of the last minute, not including the final data point, was used as the steady-state energy expenditure of the exercise. Metabolic work (kJ) was computed by integrating the metabolic power trace across this last minute of exercise.

Structure of the motion data.

The existing Microsoft Kinect for Windows SDK provides an API and access to the data streams of the camera which produces 30 frames of data per second. The skeleton stream tracks 20 joints by default (Fig 2). The position of each joint is stored as three floating point numbers that represent the XYZ coordinates in metres from the camera.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The 20 body joints of the human skeletal model.

https://doi.org/10.1371/journal.pone.0127113.g002

The data structure of the human skeletal model is a hierarchical tree, where the root of the tree is at the centre of the two hip joints. This joint is referred to as the root joint. A body joint J₁ is said to be the parent joint of a body joint J₂ if both joints are directly connected in the skeletal model and if J₁ is closer to the root joint than is J₂. Alternatively, J₂ is said to be a child joint of J₁. A bone segment is therefore specified by two adjacent joints in the hierarchy, with the tracked information stored in the child joint. The orientation of each bone segment is the transformation in space from its parent’s bone coordinate frame. Thus, there are 19 relative orientations in the kinematic skeleton returned by the Microsoft API (application programming interface) from the SDK. In addition, the absolute orientation and displacement of the player with respect to the camera stored in the root joint are also returned. These parameters are returned as a 4 × 4 rigid transformation matrix. The SDK uses single-precision floating point numbers and a pose recognition algorithm [13] to determine the human pose. Problems such as occlusion and infrared interference may affect the accuracy of the skeletal model parameter values.

The body was modelled as a 14 segment rigid body, with 3 segments for each limb, as well as a trunk and head segment. An important difference between the pose recognition algorithm of Shotton et al. [13] and marker-based motion capture systems is that their algorithm does not enforce individually calibrated constraints on the kinematic skeleton. Thus, properties such as bone segment length can vary between different poses of the same individual. To overcome this limitation we use the adjustment procedure on the lengths of body segments [26] to define the endpoint of a segment rather than using the child joint position directly.

Calibration for the Kinect Cameras.

Based on the Windows API, we developed a software system that combines the motion data from two Kinect cameras. Our software system involves two Kinect cameras placed at 60 degrees apart in front of the participant so as to minimize self-occlusion and interference. We found that the interference between the Kinect cameras is minimized for such a setup as they observe slightly different body parts of the participant. The motion data stored in the data file is in binary format containing the twenty joint positions in XYZ coordinates, a validity character for each joint, twenty bone orientations in YXZ Euler angles and a timestamp for each frame.

To combine the data from the two cameras, the participant is required to stand still with their arms held apart so that they make a straight horizontal line. This assumes that both cameras are at the same height so they differ only by a translation and a rotation about the vertical axis. The distance between the left wrist and right wrist is measured for each Kinect camera. To ensure that the wrist positions estimated by both cameras are accurate for calibration, the software system checks and imposes that the difference between the two distance values is less than 10cm. These wrist positions are then used to determine the relative angle and displacement of the second camera with respect to the first. This allows the joint positions from the second camera to be transformed into the coordinate system of the first and vice versa. If the joint is being tracked by camera 1 but not camera 2 then the value for camera 1 is used and vice versa. If the joint position is valid in both cameras then the average of the two is taken; if it is invalid in both then camera 1’s position is used. These different situations are represented in the validity character in the data file. For the orientations of the bone segments, the observations from camera 1 were always used.

Overview.

Forces generated by individual muscles are generally unknown and are difficult to directly observe in human movement [27]. What is observed is the motion that occurs from the net force acting on the body, and so while the mechanical work done by the individual muscle forces cannot be calculated, the mechanical work done on the body can be calculated from changes in its mechanical energy. Our approach involved using different parts of the mechanical work as separate variables in a multivariate model to predict metabolic energy cost.

Our Model.

We adapt the approach of Willems et al. [28] that includes both external work (the work required to move the body centre of mass, as measured from kinematics) and internal work (the work required to move the body segments relative to the centre of mass). We also adapt their method to model the transfer of mechanical energy between adjacent segments. While Willems et al. [28] only calculated segment energy transfer for the lower limbs, we extend these calculations to the upper limbs. Since muscles expend energy when they lengthen as well as when they shorten, we also separate mechanical work into positive and negative components. This is important since muscles are considered less efficient for positive work (concentric contractions) than negative work (eccentric contractions) [29]. By separating positive and negative work into different variables, the predictive model can learn the relative efficiencies solely from the sample data. Estimations of the body segment properties, such as segment mass, length, centre of mass position, radius of gyration, were calculated from the adjusted Zatsiorsky-Seluyanov’s equations of de Leva [26].

External work.

The energy at any instant of time t associated with the external work of the centre of mass of the whole body (COM_wb) is defined as (1) where M is the total mass of the body, g is the gravitational acceleration, H is the height of COM_wb and V_cg is the linear velocity of the COM_wb relative to the environment.

From the centre of mass position computed from Zatsiorsky-Seluyanov’s equations and the coordinates of the root joint of skeletal model of the participant, an offset position along the spine of the skeletal model can be obtained. At any time instant, we approximate the COM_wb of the participant to be the coordinates of the root joint plus this offset. The linear velocity V_cg is approximated from the finite differences of COM_wb over consecutive time instants. The calculations assess the work of the limb relative to the COM and thus take any changes in the movement of the COM into account. In our software system, variable H corresponds to the vertical component of COM_wb above the floor plane.

Let W_ext(t) = E_COM(t + 1) − E_COM(t). Then

• positive external work is defined to be the sum of positive W_ext(t) values over the exercise recording period;
• negative external work is defined to be the sum of negative W_ext(t) values over the exercise recording period.

Internal work and energy transfer between limb segments.

The energy at any instant of time t associated with the internal work of a segment i is defined as (2) where m_i and ω_i are, respectively, the mass and angular velocity of the i^th segment, V_{r, i} is the relative velocity of the centre of mass of the i^th segment (i.e., COM_i) to COM_wb, and K_i is the radius of gyration from COM_i.

From the 3D coordinates of the two end points of the i^th body segment, the COM_i is taken to be the centroid of these end points. For instance, the coordinates of the shoulder and the elbow joints are used to compute the coordinates of the upper arm segment. Variable m_i described above is computed from Zatsiorsky-Seluyanov’s equations. Similar to the linear velocity V_cg, the linear velocity of segment COM_i can be computed in a similar manner. Taking into account the movement of the whole body, the relative velocity V_{r, i} can be estimated. The angular velocity ω_i of the i^th is computed in a similar manner.

Having obtained all the necessary terms, the internal work E_{int, i} for each i^th segment can be computed for each time instant using Eq (2). The equation assumes a complete transfer of energy between translational and rotational kinetic energy within a segment. Energy transfer is not permitted between the trunk and adjacent limb segments; however, energy transfer is permitted between adjacent segments of the same limb. For example, if the thigh has negative internal work and the shank and foot have positive internal work, then the thigh can only transfer energy to the shank.

Let W_{int, t}(t) = E_{int, i}(t + 1) − E_{int, i}(t) be the internal work value for the i^th segment of a specific limb. Then, after the internal energy transfer has been considered, the positive (or negative) internal work for that limb is defined to be the sum of positive (or negative) internal work values of all the limb’s segments over the exercise recording period. For the positive (or negative) lower limb work, this is simply the sum of the left and right lower limb positive (or negative) internal work. The positive and negative upper limb works are defined in a similar fashion.

Putting all of these together we thus have six variables: positive external work, negative external work, positive lower limb internal work, negative lower limb internal work, positive upper limb internal work, and negative upper limb internal work. The last minute section of each recording was temporally aligned to ensure that the same cycle of the exercise was used across all participants for the mechanical work calculations. For example, the last minute of the squat exercise was aligned (Fig 3) so that the mechanical work of only 4.5 squat cycles was used. These 6 variables form the 6-dimensional (or 6D) feature vector for each data observation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Temporal alignment of the hip centre joints of the participants for the squatting activity.

https://doi.org/10.1371/journal.pone.0127113.g003

An added processing technique to the mechanical work features, namely, an estimation of posture cost, was also implemented. For the squat exercise in particular, a large component of the muscular energy expenditure would be due to the increase in joint torque at the knees and hips to maintain posture, yet no mechanical work would be observed. To overcome this, instead of mechanical work, we used a posture cost when the participant was stationary, i.e., when the velocity of COM_wb was smaller than a threshold for more than a specified number of frames. We used a velocity threshold of 0.05 m/s for a duration of 30 frames (i.e., 1 second). We define the posture cost to be the difference between the default standing cost of 1.5 watts per kilogram (a value commensurate with our observed standing costs and those of others [30]) and a scaled standing cost. The scaled standing cost was calculated as the default standing cost multiplied by the ratio of the sum of the left and right leg lengths to the sum of the left and right ankle-to-hip distances over the duration of the stationary period. Thus, the lower was the person’s centre of mass over a longer stationary period, the higher was the posture cost. This posture cost and the previous 6 variables together form a 7-dimensional (or 7D) feature vector for each data observation.

In addition to the 6D and 7D feature vectors above, we also computed the sum of the 6 variables above to yield a work value composed of all external plus internal work. This gave us a 1-dimensional (or 1D) feature vector for each data observation. This is similar to the approaches used to calculate total mechanical work [18, 31, 32]. We therefore have 3 types of feature vectors to represent the mechanical work of our data observations. With 19 subjects (see Table 1) participating in the experiments and each subject performed the 4 fundamental movements, there were 76 samples of (1D, 6D, and 7D) feature vectors and metabolic energy costs in our data collection.

Regression in machine learning.

Given a training dataset of n observations {x_i∣i = 1, ⋯, n} and corresponding target values {y_i∣i = 1, ⋯, n}, the objective of regression in machine learning is to predict the target value y_* for a new input feature vector x_* Between the observations and the target values there is an unknown relationship, i.e., f(x) = y. The function f, which maps an observation x to a target value y, may be described parametrically if we have sufficient knowledge about the problem domain and the data; otherwise, a non-parametric approach would have to be sought where the form of function f is never explicitly modelled. For more in-depth discussion on regression in machine learning, we refer the reader to Bishop [38] and Prince [39].

Gaussian process regression.

Gaussian process regression [33–35] is a nonparametric regression technique. In GPR, a Gaussian process (GP) places a prior over the space of functions for f without explicitly parameterizing f. A GP is completely specified by a mean function and a covariance function, cov(f(x), f(x′)), which by definition is: $\mathrm{\text{cov}}(f(\mathbf{\text{x}}),f({\mathbf{\text{x}}}^{\prime }))=(f(\mathbf{\text{x}})-{\mu }_f){(f({\mathbf{\text{x}}}^{\prime })-{\mu }_f^{\prime })}^{\top }$ where μ_f is the mean of f(⋅). Our aim is to find a suitable k(x,x′) such that k(x,x′) = cov(f(x), f(x′)). The free parameters involved in the covariance function are known as hyperparameters [40].

In our case, we investigated the cases where each feature vector in {x_i∣i = 1, ⋯, n} is a 1D, 6D, or 7D feature vector and each target in {y_i∣i = 1, ⋯, n} is a metabolic energy cost. The value of n was 76. In our experiments, the feature vectors were normalized so that the mean and standard deviation for each dimension were 0 and 1, respectively. The same normalization was applied to the metabolic energy. For the covariance function, we found that the dot product covariance function was the most suitable: (3)

The only hyperparameter here is ${\sigma }_0^2$, which represents a noise term in the samples.

Given that we are predicting the metabolic energy from noisy samples, we model y = f(x) + ϵ, where ϵ represents a Gaussian noise term (of 0 mean) in the metabolic cost estimation. Thus, putting all the target metabolic energy values into a vector, y = [y₁, y₂, ⋯, y_n]^⊤, we have $\mathrm{\text{cov}}(\mathbf{\text{y}})=K(X,X)+{\sigma }_0^{2}I$, where X denotes the collection of all the feature vectors x’s and K(X, X) is an n × n matrix storing the k(x_i,x_j) values. The metabolic energy for a new observed feature vector can then be predicted using the computed cov(y) matrix and K(x_*, X) vector.

k-Nearest Neighbour.

The k-nearest neighbour algorithm is a simple pattern recognition algorithm that finds the k nearest feature vectors in the training set to a given test vector, where k is a pre-defined positive integer. The locally-weighted regression simply takes the k targets that correspond to these k nearest vectors from the test vector x_* in the training set and weights them according to the inverse of their distances from the test vector. The weighted average of these targets gives a predicted target value y_*. Mathematically, if N(x_*) is the set of the k nearest neighbours of x_* and $\tilde{y}$ is the metabolic energy corresponding to the neighbour $\widetilde{\mathbf{\text{x}}}$ of x_*, then (4) where $w(\widetilde{\mathbf{\text{x}}},{\mathbf{\text{x}}}_{*})=1/d(\widetilde{\mathbf{\text{x}}},{\mathbf{\text{x}}}_{*})$ is the weight for the neighbour $\widetilde{\mathbf{\text{x}}}$ of x_* and d(⋅, ⋅) can be any distance function. Other weighting functions that we tested include $w(\widetilde{\mathbf{\text{x}}},{\mathbf{\text{x}}}_{*})=1/d^2(\widetilde{\mathbf{\text{x}}},{\mathbf{\text{x}}}_{*})$ and $w(\widetilde{\mathbf{\text{x}}},{\mathbf{\text{x}}}_{*})=\exp (-d^2(\widetilde{\mathbf{\text{x}}},{\mathbf{\text{x}}}_{*})/b)$, where the value of b should be appropriately determined to ensure that all the weight values have reasonable magnitudes.

For the k-nearest neighbour regression (KNNR), the only free parameter is the value of k, which determines the cardinality of the set N(x_*). KNNR does not need to have a training stage at all if the value of k is suitably determined by the user for the application. In the literature, various heuristic techniques (e.g., hyperparameter optimization [35]; minimization of the reconstruction error via cross validation) have been suggested to estimate an optimal value for k in the training stage.

Linear regression.

Similar to KNNR where no prior training is required, the linear regression (LINR) algorithm produces the vector α = (α₁, ⋯, α_D), where D denotes the dimensions (1, 6, or 7) of the feature vectors, such that the total regression error is minimized: (5)

In the training stage, the vector α is obtained from the training dataset. In the prediction stage, given a feature vector x_*, the predicted y_* is computed as y_* = α^⊤ x_*.

Cross validation.

Cross validation is a technique for assessing the accuracy of an algorithm when applied to new unknown data. It provides a measure of the generalizability of a statistical model. To obtain statistics (e.g., the average regression error) on the performance of the algorithm, the dataset is parsed a sufficient number of times. In each parse, the dataset is partitioned into a training set and a validation set (or test set). The regression algorithm is trained on the training set and tested on the test set. The regression error on the test set can be computed since the ground truth of the target values y’s are known.

To assess the performance of the GPR, KNNR, and LINR algorithms, we used the leave-one-out cross validation (LOOCV) technique. From the n samples of x_i and y_i, for i = 1, ⋯, n, that we collected, we took each (x_i, y_i) in turn to be the test set and the remaining samples to be the training set. So the training set in each parse was formed by leaving one sample out. This allowed us to obtain n regression errors from which we computed the root mean squared error (RMSE).

Concordance Correlation Coefficient and Paired t-test.

The concordance correlation coefficient proposed by Lin [41] was designed for evaluation of reproducibility, i.e., how well two sets of scalar measurements computed from the same data differ. The concordance correlation coefficient (CCC) focuses on how the two sets of measurements deviate from the 45° line passing through the origin. The concordance correlation coefficient ρ is given by (6) where μ₁, μ₂, ${\sigma }_1^2$, and ${\sigma }_2^2$ are the means and variances of the two sets of measurements, and σ₁₂ is their cross correlation. We use the CCC to assess how the predicted metabolic energy values deviate from the ground truth metabolic energy values in the cross validation process. We then use a paired t-test via the Matlab ttest function to test whether there is a statistically significant difference in the predictions between methods. We set the α-value of the paired t-test to 0.05.