Supervised learning as a variational data assimilation problem

We consider a class of supervised learning problems, in which we are given a set of N_s inputs, denoted by ; and the corresponding set of N_s outputs, denoted by , with and being the domains with respect to the inputs and outputs, respectively. Here, our objective is to learn a certain function , such that h(x_i) match y_i (i = 1, 2, ⋯, N_s) to a good extent. Note that, in general, the outputs y_i may be contaminated by certain noise.

To achieve the above objective, one can solve the SLP as a regularized empirical risk minimization (ERM) problem [30]), as defined below (1) where L is a suitable loss function that measures the distance between y_i and h(x_i), γR(‖h‖) is a regularization term, with γ being the regularization parameter, ‖h‖ the norm of h with respect to a certain metric space, and R the regularization operator. From the perspective of inverse problem theory [31], the regularization term is typically introduced to prevent the estimated function h* from over-fitting the training data, as well as avoid potential numerical issues in the course of solving the minimization problem.

Clearly, without imposing any constraint on the functional h, the regularized ERM problem in Eq (1) is intractable. In practice, it is customary to assume that h belongs to a certain function space, and can be approximated through some parametric model, e.g., in the form of (2) where θ is a set of parameters in the (parametric) functional . Since is parametrized by θ, replacing h by , then the regularized ERM problem in Eq (1) becomes a parameter estimation problem, in the form of (3)

In addition, let us define (4) (5) (6) where in we place the argument θ in front of X^s to emphasize that now θ is the quantity in estimation, and we use a semicolon to separate quantities in estimation (i.e., θ) and those that are given (i.e., X^s). Similar custom will be adopted later for notational convenience.

To facilitate the introduction to our idea, we first consider the situation in which the training inputs x_i follow a certain unimodal distribution. In this case, we choose the functionals L and R in such a way that (7) (8) where and are some pre-chosen weight matrices associated with L and R, respectively, and θ^b stands for a (pre-chosen) initial guess (called “background” hereafter) of θ. Under these settings, the regularized ERM problem in Eq (3) is equivalent to (9)

Comparing Eqs (3) and (9), the scalar factor 1/N_s is dropped in Eq (9), with its impact being absorbed into the regularization parameter γ. From a perspective of data assimilation, Eq (9) constitutes a conventional variational data assimilation (VAR-DA) problem, which can be solved through, e.g., optimal interpolation (OI) or three-dimensional variational (3D-VAR) method [32].

When the training inputs follow a multi-modal distribution, it may be necessary to cluster the training inputs into different groups (so that each group contains unimodal training inputs), and then estimate a set of parameters θ for each group, using an estimation method developed for unimodal cases. In this sense, a parameter-estimation method developed for unimodal cases can serve as the building block of a method for multi-modal cases, similar to the work of [33, 34]. For this reason, in what follows, we focus on presenting an ensemble-based estimation method for unimodal cases. We will discuss how one can adapt the developed method to multi-modal cases, when we come to a concrete SLP problem with multi-modal training inputs.

An ensemble-based approach to solving the supervised learning problem

In analogy to the advance of assimilation approaches from the conventional variational methods [32] to the more recent, ensemble-based methods [11], it is natural for us to develop a certain ensemble-based method to tackle the SLP. To this end, instead of solving Eq (9) to obtain a single set of estimated parameters, we aim to estimate an ensemble of such parameters. By doing so, we will obtain all the intrinsic benefits in using ensemble-based methods, which includes, for instance [6],

• no need to develop a complicated and time-consuming adjoint system (“adjoint free”);
• the capacity to provide a means of uncertainty quantification for the estimated results (“uncertainty quantification”);
• the ability to handle large numbers of state and/or parameter variables (“algorithm scalability”);
• straightforward and fast implementation (“implementation convenience”).

Employing this “ensemblizing” strategy, we reformulate the regularized ERM problem in Eq (9) as an minimum-average-cost (MAC) problem [35], in terms of (10) where N_e is the size of the ensemble of parameters in estimation. Note that each ensemble member θ_j has its own associated background . Typically, the initial values of are generated at random, thus almost surely if j ≠ k. As a result, solving the MAC problem in Eq (10) would result in an ensemble (called analysis ensemble hereafter) of diversified estimates, and this naturally leads to a way of conducting uncertainty quantification for the estimated results.

Following the convention in ensemble-based methods, we choose C_y to be the covariance matrix of the observation noise in the outputs y_i, and C_θ to be the sample covariance matrix with respect to the background ensemble , in the sense that (11) (12) (13)

As shown in [35], with a linearization-based approximation strategy, a solution to the MAC problem in Eq (10) is given by (14) (15) (16) (17)

Eqs (11) through (17) essentially constitute the iterative ensemble smoother (iES) used in [35]. In a practical implementation of the iES update formula Eq (14), one may choose to apply a truncated singular value decomposition (TSVD) to , so that the matrix inversion in Eq (15) can be carried out in a low-dimensional subspace (with the dimension less than N_e). For more information, see [8, 11, 36].

In addition, the update formula Eq (14) often has to be iterated for a number of times to make sure that the estimated parameters would be able to achieve good data match. In such an iteration process, we adopt a “warm restart” strategy, in such a way that an analysis ensemble at one iteration step serves as the background ensemble at the next iteration step. The regularization parameter γ also needs to adapt to the iteration process, and is chosen in such a way to avoid either too big or too small iteration steps. Details on the choice of γ and the associated stopping criteria are elaborated in [8, 35], and are skipped in this work for succinctness.

RBF kernel based functional approximation

After establishing an ensemble-based framework to handle SLP, we go back to discuss the concrete approach to functional approximation in Eq (2). In this regard, there are many methods (see, e.g. [23, 37]), such as generalized linear models (GLM), support vector machines (SVM), and various (shallow or deep) neural networks that one may exploit. In the current work, taking into account various factors like capacity, complexity and cost, we choose to adopt radial-basis-function (RBF) based kernels for functional approximation. The RBF kernel approach was previously proposed in a seminal work of [38] to solve the SLP in Eq (1) in a way similar to a VAR-DA method, and this led to the establishment of RBF networks [39]. Recently, the RBF kernel approach is also adopted by [40] to build computationally efficient surrogate models for history matching.

Specifically, following the RBF kernel approach to functional approximation in [38], we have (18) (19) (20) Note that in Eq (19), we adopt the Gaussian RBF kernel, but other types of kernel functions can also be used, as long as they serve the purpose of functional approximation well. Hereafter, for the convenience of discussion, we may drop the word(s) “Gaussian” and/or “RBF”.

Eq (18) indicates that, the approximation functional is composed of a set of N_cp kernels K, which are associated with different weights c_k, center points (CP) , and scale parameters β_k that influence the spreads of the kernels. In the current work, for simplicity, we pre-choose N_cp and , such that is parametrized by a set θ of parameters c_k and β_k, as indicated in Eq (20) (for ease of visualization, we use “|” to separate different groups of parameters in Eq (20)).

Eqs (18) through (20) are for univariate problems. To extend the kernel approach to multivariate problems (e.g., ), one may consider the following form: (21) (22) (23) (24) In reservoir history matching problems, m can be interpreted as the number of different types of petrophysical parameters (e.g., permeability, porosity and so on) associated with each reservoir gridblock, hence typically it may not be very large.

Eq (23) considers generic anisotropic scale parameters β_k that may have different values β_k,ℓ along different axes x_ℓ (ℓ = 1, 2, ⋯, m). In addition, the factor 1/m in Eq (23) is adopted to mitigate the issue of arithmetic underflow, which may arise in case that becomes sufficiently large. Under the above settings, the total number (cardinality) of parameters in θ is thus (m + 1) × N_cp, as indicated in Eq (24). Therefore, the cardinality of θ is controlled by the number N_cp of center points, while the dimension m of x is typically fixed.

In comparison to the previous work of [38, 40], one feature of our proposed Kernel approach to functional approximation is that the scale parameters β_k not only adapt to different center points , but also vary along different coordinate axes. This kind of flexibility may be considered desirable in the context of machine learning, as it leads to additional parameters that may help improve the expressive power (or capacity) of a learning model to match the training data, and also reduce generalization errors [37].

In addition, in many existing publications, the scale parameters are often manually chosen. In contrast, the ensemble-based approach (Eqs (14) through (17)) renders an efficient and derivate-free framework to estimate multiple sets of such parameters, hence also provides a natural means of uncertainty quantification for the estimation results.

Problem statement

We consider a data assimilation problem, in which the noisy observational data (observations) are obtained through the following observation system (25) where represents a set of true model variables, the true forward simulator, and ϵ additive observation noise, which is assumed to follow a Gaussian distribution with zero mean and covariance matrix C_d. For better comprehension, here we have deliberately avoided notational overlapping with those in the proceeding section as far as possible, since such distinctions would be useful for our discussions later.

In the current work, we assume that, for all , one has , where f : R → R is a scalar function. Thus, an immediate implication of this assumption is that the size of observations is also equal to m_z, i.e., . We note that, the assumption we made here aims to mimic the situation in seismic history matching problems (or other similar geophysical inversion problems which involve spatially distributed, image-like geophysical data), but with certain simplifications to facilitate computations and discussions later. Under this setting, one may treat the scalar function f as an analogy to a rock physics model, which maps petrophysical and/or dynamical parameters (the inputs) to certain seismic attributes (the outputs), such as acoustic impedance, distributed over reservoir gridblocks.

As a data assimilation problem, our objective is to estimate a set z of model variables, conditioned on the observations d^o and some initial guess (background) of z^b, in such a way that z is as “close” to z^tr as possible. In a typical setting, we have access to a certain forward simulator g that maps z to some simulated (or predicted) observations d^sim, i.e., (26) with for a scalar function g : R → R. This simulator, g, is often imperfect, and may not be exactly identical to the true forward simulator f. In the next subsection, we address the issue of imperfection by integrating ensemble-based kernel approach to functional approximation into an ensemble-based data assimilation framework.

Integrating ensemble-based kernel learning into data assimilation

Based on Eqs (25) and (26), we have (27) (28) where r represents a functional of residuals that measure the differences between real observations d^o and the simulations g(z). As in the preceding subsection, we have , with for ℓ = 1, 2, ⋯, m_z.

Following the idea of kernel approach to functional approximation (Eqs (18) through (24)), we can approximate by (29) where is composed of a set of kernels with their parameters contained in η, in the form of (30) with the operator 〈•, •〉 being defined in Eq (23); represents a set of center points , and stands for the corresponding set of observations associated with Z^cp. Likewise, we define .

It is worth noting an essential difference between SLP and data assimilation problems. In SLP (cf. Eqs (9) and (10)), one has multiple “matched” input-output pairs, X^s and Y^s, respectively, as the training data; In data assimilation problems, however, typically we only have access to a single realization of the outputs (observations) d^o at a given time instance and a given spatial location, whereas our purpose is to infer possible inputs z given d^o. Often, due to the limited capacity of the assimilation algorithm, d^o and z do not constitute a “matched” pair, or in other words, z would typically not be identical to the true model variables z^tr that generate the observations d^o. Because of this inconsistency and the sample frequency of observations (at a given time instance and a given spatial location), data assimilation problems with imperfect forward simulators tend to be more challenging than SLP, as we will see later.

The aforementioned difference between SLP and data assimilation motivates us to take a slightly different form in Eq (30) for kernel-based functional approximation, in comparison to those in SLP (Eqs (18) through (24)). Specifically, in Eq (30), we choose to augment both the model variables z_ℓ and the corresponding residuals , and use the augmented vectors as the inputs to the kernel functions. In comparison to the settings in SLP, using in kernel functions allows us to tune additional scale parameters in data assimilation, which may be desirable in terms of flexibility. On the other hand, though, this also requires us to specify a set of observations D^o,cp associated with Z^cp. In general, the choice of Z^cp and D^o,cp may be case-dependent. For instance, if one has a set of pairs from the hard data (e.g., those obtained from core analysis or well log data), then they can be included. In a case study later, we will give a specific implementation example on the choices of Z^cp and D^o,cp.

With kernel-based functional approximation to the residuals, similar to [35] (also see Eq (10)), the data assimilation problem with an imperfect forward simulator can then be addressed by solving the following optimization problem: (31) with (32) (33) where is a joint vector that augments model variables z and parameters η associated with the set of kernels; is the sample error covariance matrix with respect to an ensemble , similar to that in Eq (11); and corresponds to the effective forward simulator.

As in Eqs (10) and (31) also constitutes an MAC problem. As a result, Eqs (11) through (17) provide an approximate solution to the data assimilation problem with an imperfect forward simulator, provided that one replaces , θ and C_y therein by , and C_d, respectively.

When there is no imperfection in the forward simulator (or when one believes so), one may choose not to introduce any correction mechanism. In this case, the parameter part η of (cf. Eq (32)) can be simply taken out. Based on this observation, it is clear that adopting ensemble-based kernel approach to accounting for imperfection in the forward simulator does not significantly change our ensemble-based data assimilation algorithm. Instead, with a modified forward simulator in Eq (33), it only requires some minor changes of the algorithm, by inserting a residual term into the original forward simulator, and then combining parameters associated with the kernel functions and the original model variables to form augmented vectors in data assimilation.

As will be shown later, even with a perfect forward simulator, it might be still beneficial to include a mechanism of model-error correction (i.e., the η term) for the improvement of data assimilation performance. The rationale behind this notion is that, similar to machine learning problems, the presence of η increases the dimension of , so that the assimilation algorithm would have more degrees of freedom to exploit for the search of better results.

Results with respect to unimodal inputs

In the experiment, we generate a set of 10, 000 input samples drawn from the univariate Gaussian distribution N(−5, 1), and the corresponding noisy residuals (defined as the differences between the noisy outputs y^o and the simulations y^sim). We randomly divide the set of input-residual pairs into two subsets: one with 8, 000 (80%) of such pairs as the training dataset, whereas the rest 2, 000 (20%) of such pairs as the cross-validation (CV) dataset. The training dataset is used to estimate the parameters associated with the selected kernel functions, whereas the CV dataset is not involved in learning these parameters. In a typical setting, the CV dataset can be adopted to select hyperparameter(s) in a learning algorithm. In this particular case, though, we do not have hyperparameter(s) to tune. Therefore, we simply use the CV dataset to inspect the performance of the learned parameters after the learning process is finished. Fig 2 shows the histograms of the inputs and noisy residuals in the training and CV datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Histograms of the unimodal inputs and noisy residuals (as labels), with respect to the training and cross-validation (CV) datasets, respectively.

https://doi.org/10.1371/journal.pone.0219247.g002

To employ the kernel approach to approximating the residual function r(x) (Eqs (18) through (20)), we need to specify a number of N_cp center points (k = 1, 2, ⋯, N_cp). In principle, it is possible to consider both N_cp and as additional parameters that may be optimized through certain criteria. However, this will make the resulting learning algorithm become much more complicated. As a result, in the current work, we pre-choose N_cp and manually.

Bearing this in mind, in the experiment below, we let N_cp = 200, and be the points that evenly span the half-closed interval [−6, 6). We have also tested other cases with N_cp = 2000, which turned out to lead to results similar to what we will present below. Consequently, for brevity, below we focus on the cases with N_cp = 200, with which the number of parameters (including the weights c_k and the scale parameters β_k, cf. Eq (20)) is thus 2 × 200 = 400.

An additional remark is that, in comparison to the histograms in Fig 2, it is clear that the interval [−6, 6) and the input ranges of both training and CV datasets do not fully cover each other. We choose such a setting to examine the impact of data coverage on the performance of the learning algorithm.

Now we discuss how to initialize the ensembles of kernel parameters, weights c_k and scale parameters β_k. For convenience of discussion, let us denote the ensembles with respect to the initial weights and the initial scale parameters by and , respectively. In the current work, we let N_e = 100 unless otherwise stated, and be initialized as follows: (40) (41) where σ^ti is the STD of the training inputs, and ξ_k,j are random samples drawn from the normal distribution N(0, 1) for each center point and each ensemble member.

For a given ensemble member (i.e., a fixed j value), we then initialize (k = 1, 2, ⋯, N_cp) as follows. We first randomly select a pair of input-label from the training dataset, denoted by , where is the label (noisy residual) in the training dataset that corresponds to the training input . We then insert the pair into Eq (18), by replacing x, and β_k therein by , and , respectively. At this stage, our goal is to find a set of weights (k = 1, 2, ⋯, N_cp) that approximately solve the the following equation: (42) which can be re-written as the following vector-based equation (43) (44) (45) An approximate solution to Eq (43) can be obtained by solving the following equation instead (46) where α is a positive scalar, and I is an N_cp × N_cp identity matrix. The term αI, essentially stemming from a Tikonov regularization term introduced to solve Eq (43) as a regularized inverse problem [31], helps to improve the numerical stability of the final solution (47) Following the implementation of the iES in [35], in the current work, we let with ξ_j ∼ N(0, 1). It is clear that the solution in Eq (47) does not solve Eq (43) exactly. This, however, is desired, since in general the label may be noisy, and an inexact solution to Eq (43) avoids the problem of over-fitting the training data. Applying Eq (47) to N_e different pairs of (j = 1, 2, ⋯, N_e), we get an initial ensemble of N_e different parameter vectors .

Fig 3 shows the box plots of data mismatch values generated by 100 different sets of kernel (weight and scale) parameters at different iteration steps, where mismatch values are calculated using (a) training and (b) CV datasets, respectively. Note that in the course of learning, only training dataset is used to update kernel parameters. Therefore, it is not surprising to see that the reduction of training-data mismatch tends to be more significant than the reduction of CV-data mismatch. The more important observation in this case, however, is that, even though the CV dataset is not involved in training the kernel parameters, its corresponding data mismatch tends to decrease as the training (iteration) process goes on, which implies that the whole training process appears useful and there is no need to stop the iteration earlier.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Box plots of data mismatch at different iteration steps, with respect to the (a) training and (b) CV datasets in case of unimodal inputs.

https://doi.org/10.1371/journal.pone.0219247.g003

Fig 4 depicts the error-bar plots (in terms of ensemble mean ± ensemble std) of kernel parameters, namely, scales (β) and weights (c), associated with 200 center points that are evenly distributed over the interval [−6, 6). For scale parameters (Panel (a)), the relative changes from initial (in blue) to final (in red) values appear not so significant for all center points. In contrast, for weight parameters (Panel (b)), more substantial changes are spotted for center points located in, e.g., the interval [−6, −2], whereas outside this interval, the changes tends to be less significant again. This does not appear to be surprising, if we take into account the coverage of training inputs (see the upper left panel of Fig 2), and the fact that a Gaussian kernel function decays exponentially to zero as the distance between a training input and the center point associated with the kernel function increases.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Error-bar plots in case of unimodal training inputs, in the form of ensemble mean ± ensemble std, with respect to the initial (in blue) and final (in red) ensembles of scale (Panel (a)) and weight (Panel (b)) parameters, respectively, associated with 200 center points that are evenly distributed over the interval [−6, 6).

https://doi.org/10.1371/journal.pone.0219247.g004

For further demonstration, in Panels (a) and (c) of Fig 5, we also compare the reference curve (red) over the input interval [−10, 10], the corresponding biased curve (green), and ensembles of corrected curves (blue), in the form of the biased curve plus ensembles of residual terms. The reference and biased curves are the same as those in Fig 1, whereas the ensembles of residual terms are calculated using Eqs (18) through (20), in which the kernel parameters correspond to either the initial or the final ensembles of scale and weight parameters, respectively. In addition, for better visualization, in panels (b) and (d), we plot the mean corrected curves (cyan).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Red (reference curve) and green (biased curve) curves in all panels of the current figure are the same as those in Fig 1.

Panels (a) and (c) show the initial and final ensembles of predictions (with respect to the case of unimodal training inputs), obtained by adding to the biased curve the corresponding ensembles of residual terms, which are computed using Eqs (18) through (20), wherein the kernel parameters correspond to the initial and final ensembles of scale and weight parameters, respectively. Panels (b) and (d) also report the means of the initial and final ensembles of predictions, respectively.

https://doi.org/10.1371/journal.pone.0219247.g005

Fig 5(a) and 5(b) indicate that, compared to the biased curve, the way for us to initialize the initial ensembles of kernel parameters tends to improve the prediction accuracies over the interval (e.g., [−6, −4]) on which the training inputs largely concentrate (called concentration interval hereafter). In addition, the resulting ensemble of corrected predictions provides a means of conducting uncertainty quantification for the predictions. Recall that we adopt only 100 (as the ensemble size) random training input-output (label) pairs to initialize the ensembles of kernel parameters. By learning (or updating) kernel parameters through more training data, it appears that the performances of both prediction and uncertainty quantification are improved over the concentration interval, as can be seen in Fig 5(c) and 5(d).

Fig 5 also shows that, for the intervals (e.g., [−2, 2]) over which there are sparse or even no training data, the corrected predictions may be less accurate than the biased predictions themselves. In this case, a natural way to improve the performance of supervised learning is to acquire more training data over different regions. Such an investigation will be carried out in the next sub-section.

Results with respect to multi-modal inputs

To generate more training data, here we consider a scenario with multi-modal training inputs. We will first identify a challenge for the ensemble-based learning algorithm to handle multi-modal training inputs, and then investigate a strategy that helps overcome this problem.

In the experiment, we generate a set of 10, 000 input samples from the distribution N(−5, 1), 10, 000 input samples from the distribution N(0, 1) and 10, 000 input samples from the distribution N(5, 1), and the corresponding noisy residuals. We then randomly split the resulting 30, 000 input-residual pairs into one training dataset (with 24, 000 data points) and one CV dataset (with 6, 000 data points). Fig 6 shows the histograms of the inputs and noisy residuals in the training and CV datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Histograms of the multi-modal inputs and noisy residuals, with respect to the training and CV datasets, respectively.

https://doi.org/10.1371/journal.pone.0219247.g006

In the sequel, we first illustrate what will happen if one directly applies the ensemble-based learning algorithm to the training data with multi-modal inputs. In the experiment, we still adopt 200 center points that are evenly distributed over the interval [−6, 6). As in the previous sub-section, the ensemble-based learning algorithm is directly adopted to update 400 kernel parameters, namely, the scale and weight parameters associated with each center point. However, it turns out that, in the presence of multi-modal inputs, a straightforward application of the ensemble-based learning algorithm may not achieve satisfactory performance. This point is demonstrated in Fig 7. With more training data than in the previous sub-section, the accuracies of corrected predictions over certain input intervals, e.g., [−6, −4], are actually worsened, as is evident if one compares panels (c) and (d) of Figs 5 and 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. As in Fig 5, but for the case with multi-modal inputs, for which no multi-modal learning strategy (MMLS) is adopted.

For better visualization, in Panel (d) we re-plot the reference, the biased and the mean corrected curves over the input interval [−2, 2] in a separate, zoomed-in subplot.

https://doi.org/10.1371/journal.pone.0219247.g007

The under-performance of plain, ensemble-based algorithms in handling multi-modal variables is also discussed in the literature, see, for example, [33, 34, 44–46]. To deal with this problem, we equip the ensemble-based algorithm with an multi-modal learning strategy (MMLS). Concretely, similar to [33, 34, 44, 46], we adopt a Gaussian mixture model (GMM) to fit the probability density function (pdf) of multi-modal variables, which naturally leads to a number of clustered subsets of multi-modal variables (and their corresponding noisy labels). Next, we use each cluster of training data to initialize (and then update) an ensemble of kernel parameters that are associated with the center points (note that different clusters of training data share the same set of center points). The corrected predictions are then taken as biased outputs plus certain residual terms, whereas the latter are calculated as the weighted averages of the residuals predicted using the kernel parameters in each cluster.

More specifically, suppose that the multi-modal inputs are clustered into N_cl mutually exclusive subsets, and in each subset, the pdf of the inputs is modelled by a certain Gaussian pdf. In other words, the pdf p(x) of training inputs is approximated by a GMM, in the form of (48) where w_s is the weight associated with the sth cluster, and is the corresponding Gaussian pdf, parametrized by the mean μ_s and STD σ_s. For each cluster, say, the sth one, we generate an initial ensemble of kernel parameters for the set of center points, in the same way as in the preceding sub-subsection. The ensemble is then updated to , using the training data associated with the cluster. With the above quantities, we are then able to generate corrected predictions for new inputs. For instance, given an input x′, we first calculate the probability P_s of x′ with respect to each cluster, through (49) Then, we can calculate an ensemble of corrected predictions, in the form of biased prediction g(x′) plus predicted residual (j = 1, 2, ⋯, N_e), where is given by (50) with , and a functional consisting of a set of kernel functions (cf. Eq (18) or (21)) parametrized by .

In terms of parametrization strategy adopted in SLP, a noticeable feature in case of multi-modal training inputs is that, each cluster of training data will have its own ensemble of kernel parameters associated with the same set of center points. Following the discussion in the text after Eq (24), for m-dimensional inputs, the total number of kernel parameters then becomes (m + 1) × N_cp × N_cl, larger than that in case of unimodal training inputs. This may thus be considered as an additional way to improve the capacity of a learning model.

Results with N_cl = 3.

In the first experiment, we investigate the case where the number of clusters is the same as the number of modes in the training inputs, i.e., N_cl = 3. We use the MATLAB function “fitgmdist” to estimate the parameters like weight (w), mean (μ) and variance (σ²) (cf. Eq (48)) associated with each Gaussian component. Table 1 summarizes the number of training data points, as well as the values of the aforementioned parameters, associated with each component (cluster). This indicates that the GMM is fitted quite well, in light of how these data are generated. Table 1 also provides each Gaussian component a label (e.g., “C1”), which will be adopted in the discussions below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Number of training data points associated with each GMM component (cluster), and the corresponding parameters estimated using the training data.

https://doi.org/10.1371/journal.pone.0219247.t001

With the aforementioned settings, in principle one can update the kernel parameters associated with each cluster in parallel, although in the current work, such updates are conducted in a sequential manner, namely, C1 → C2 → C3. Fig 8 shows the box plots of data mismatch, with respect to training (left column) and CV (right column) datasets, respectively. For the training dataset, we report data mismatch at different iteration steps cluster by cluster. For instance, in Panel (a) of Fig 8, data mismatch is calculated using the differences between the training outputs in C1, and the predicted outputs with respect to the training inputs in C1. Under this setting, one can see that the ensemble-based learning algorithm progressively reduces data mismatch within each cluster.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Box plots of data mismatch at different iteration steps, with respect to the (a) training and (b) CV datasets in case of multi-modal inputs.

https://doi.org/10.1371/journal.pone.0219247.g008

For the CV dataset, we do not pre-cluster the data points into different clusters. To compute data mismatch with respect to the CV dataset, we use all the CV data points (6000 in total). For better comprehension, Eqs (49) and (50) are referred in our discussion below. Given a CV input x^cv and an ensemble of kernel parameters θ_j,s (j = 1, 2, ⋯ , N_e) for a certain cluster s, we compute an ensemble of predicted outputs (cf. Eq (50)), as well as a probability P_s(x^cv) with respect to the GMM (cf. Eq (49)). Data mismatch with respect to the cluster s is then calculated using the differences between the CV output weighted by P_s(x^cv), and the predicted output that is also weighted by P_s(x^cv). In this way, we are able to cross-validate the impacts of supervised learning within individual clusters. As reported in Panels (b), (d) and (f) of Fig 8, data mismatch of the CV dataset with respect to all clusters tends to decrease through the iterations, indicating that the learning process goes reasonably well.

Similar to Fig 4, in Fig 9 we also plot the initial (in blue) and final (in red) ensembles of scale (left column) and weight (right column) parameters associated with different clusters. For scale parameters, compared to the case with unimodal training inputs (cf. Fig 4(a)), there appear to be more substantial differences between initial and final values in all three clusters (cf. Fig 9(a), 9(c) and 9(e)). For weight parameters, similar to the case with unimodal training inputs (cf. Fig 4(b)), significant changes from initial to final values can also be spotted in the areas surrounding the mode of each Gaussian component.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Similar to Fig 4, but for multi-modal training inputs.

For visualization, we plot scale (left column) and weight (right column) parameters associated with different clusters separately.

https://doi.org/10.1371/journal.pone.0219247.g009

Similar to Figs 7 and 10 shows the results after the MMLS is adopted to train kernel parameters cluster by cluster. As one can see, with the MMLS, the initial ensemble in Fig 10(a) exhibits multimodality, which is not the case for the initial ensemble in Fig 7(a), where the MMLS is not employed. On top of the multi-modal initial ensemble, the ensemble-based training algorithm in general tends to improve the predictions, by updating the kernel parameters sequentially through the use of training data in individual clusters.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Similar to Fig 7, but for the case in which the multi-modal learning strategy (MMLS) is adopted.

In the experiment, the number N_cl of clusters is 3, the same as the number of modes in the training inputs. Note that the learning process is carried out cluster by cluster.

https://doi.org/10.1371/journal.pone.0219247.g010

The impact of the number N_cl of clusters.

The previous results indicate that, when the MMLS is adopted and the number of clusters is the same as the number of modes in the training inputs, one can improve the performance of predictions using the learned kernel parameters. Here, we also examine what will happen, when the MMLS is adopted, but the number of clusters is not necessarily the same as the number of modes in the training inputs.

Fig 11 presents some of the final prediction results (after using all training data to learn kernel parameters) from an experiment, in which we adopt different numbers N_cl of clusters (e.g., 2, 4, 6 and 8) to fit the GMM using the same training inputs (with 3 modes) as in the previous experiment. Combining the results in Figs 7, 10 and 11, it appears that, if the number of the clusters is less than the number of modes in the training inputs, then the learned ensemble of models tends to have insufficient capacities to perform relatively well in the prediction tests. However, when the number of the clusters becomes no less than the number of modes, then the capacities of the learned models tend to improve. In this particular case, it seems that, if N_cl is slightly larger than the number of modes (e.g., N_cl = 6), then one might actually achieve better prediction accuracies over certain intervals, in comparison to the choice of N_cl = 3. Of course, given a fixed number of training data, on average the number of training data per cluster will reduce as N_cl increases. Therefore, if N_cl becomes too large (e.g., N_cl = 8), the prediction accuracies may be instead worsened as the number of training data within each cluster decreases. This insight will be useful for us to handle data assimilation problems in the presence of forward-simulator imperfection, yielding improved flexibility and assimilation performance, as will be shown in the next section.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Similar to Fig 10, but for the final prediction results after all the training data in different clusters are used to learn kernel parameters.

Presented here are the results with respect to of the choices of using 2, 4, 6 and 8 clusters to fit the GMM (from top to bottom), respectively.

https://doi.org/10.1371/journal.pone.0219247.g011

Results in the perfect scenario (PS)

In the PS, we conduct a comparison study involving two experiments. In one of them, no MEC is adopted since the forward simulator is known to be perfect. In the other, kernel-based MEC is introduced to data assimilation, even though the forward simulator is perfect (in many places, we will simply say MEC when there is no confusion). Except for this difference, the other settings in these two experiments are identical. We note that, in the relevant experiment, MEC is conducted by combining Eqs (33) and (50), whereas in Eq (50) the number N_cl of cluster is set to 1 in the current experiments, as we know that the initial ensemble is generated used a Gaussian simulation method (hence unimodal). We will examine the impact of N_cl on data assimilation in the IS.

In comparison to the real observations in Fig 13(a), the mean of simulated observations with respect to the initial ensemble (Fig 13(b)) appears substantially different in many regions. As a result, the data mismatch values of the initial ensemble are relatively large, as reported in Table 2. Through data assimilation using the iES, data mismatch values of the updated ensembles tend to decrease as the iteration process proceeds, whether MEC is introduced or not, as one can see in Fig 14(a) and 14(b). Accordingly, after data assimilation, the means of simulated observations with respect to the final ensembles (with or without MEC), as shown in Fig 13(c) and 13(d), respectively, resemble the observations better than that with respect to the initial ensemble.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Box plots of data mismatch (top) and RMSE (bottom) with respect to the ensembles at different iteration steps in the perfect scenario.

Results in Panels (a) and (c) correspond to the case without MEC adopted in data assimilation, whereas those in Panels (b) and (d) to the case with MEC. Unless otherwise stated, data mismatch in the experiment with MEC is always calculated using the modified forward simulator with a residual term, as in Eq (33). Note that in Panels (a) and (c), the iES terminates at the iteration step 7, due to the stopping criterion that the average data mismatch at this step is less than four times the number of observations (which is 4 × 12, 000 in this case) for the first time.

https://doi.org/10.1371/journal.pone.0219247.g014

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Means and STDs of data mismatch and RMSE with respect to the initial ensemble, and the final ensembles with or without model-error correction (MEC), in the perfect scenario.

https://doi.org/10.1371/journal.pone.0219247.t002

In both experiments, the maximum iteration step of the iES is set to 10. In the experiment without MEC introduced, however, the iES stops at the iteration step 7, due to an alternative stopping criterion that is triggered to terminate the iES, when the average data mismatch is lower than four times the number of observations (which is 4 × 12, 000) for the first time. This early-stopping phenomenon indicates a higher risk of over-fitting observations, should the iteration process have continued after iteration step 7. On the other hand, in the experiment with MEC, since there are more parameters adopted in data assimilation, intuitively one might expect that the problem of over-fitting observations can be even more severe. Surprisingly, it turns out that over-fitting actually appears avoided, while the iteration stops at the maximum step. As a result, the final mean data-mismatch value in the experiment with MEC is higher than that in the experiment without MEC, as reported in Table 2. One possible explanation of the ability to avoid over-fitting may be related to the effect of localization [7], which is rendered by the MEC mechanism, as will be discussed later.

For quality check, in Fig 14(c) and 14(d) we also show the box plots of RMSEs of the ensembles of model variables at different iteration steps. When no MEC is introduced, the RMSEs tend to decrease at the first five iteration steps, and then bounce back to somewhat higher values at the last two iteration steps. This kind of “U-turn” behavior was also found in the earlier work of [6], and can be mitigated or avoided by introducing a procedure of sparse data representation [6], or localization [7, 9], to the iES (an investigation on this issue, however, is beyond the scope of the current work). In contrast, with MEC introduced to the iES, the “U-turn” behavior seems vanished. Furthermore, the final mean RMSE value in the experiment with MEC is lower than that in the experiment without MEC, as one can see in Table 2. In Fig 12(c) and 12(d), we show the mean of the final ensembles obtained in the experiments with or with MEC. Clearly, the mean models of the final ensembles appear more similar to the reference model than the mean model of the initial ensemble in Fig 12(b). The mean model of the final ensemble without MEC tends to do better than that with MEC in the regions on the left-hand side (e.g., for x ≤ 50), but worse in the rest of the regions.

One can also observe an interesting phenomenon by comparing the spreads of box plots in Fig 14, or the calculated ensemble STDs of data mismatch and RMSE in Table 2. Recall that, in the experiments, no localization is introduced to the iES. As a result, it may not be surprising to see that, in the experiment without MEC, ensemble collapse seems to take place. In contrast, in the experiment with MEC, ensemble collapse does not appear to be a problem, or at least is mitigated. This seems to suggest that the kernel-based MEC mechanism can (partially) lead to the same effect on preventing ensemble collapse as localization does. A possible explanation to this phenomenon may be that, as aforementioned, since we use Gaussian RBF as the kernel function, the kernel parameters (scale and weight) associated with a certain center point would exhibit localized impacts, and mainly influence model variables that are sufficiently close to that center point.

As aforementioned, in SLP, typically one has many (matched) input-output pairs as the training data. In contrast, in data assimilation problems, we use a single realization (or one-shot) of the observations (at a given time instance and a given spatial location) to infer possible model variables. As a result, in SLP, one often has the luxury to split a dataset into two parts, one for training (and cross-validation) and one for test; whereas in data assimilation with MEC, this kind of luxury typically does not exist. This makes MEC a particularly challenging problem. Indeed, apart from the potential inconsistencies between the observations and the estimated model variables, there are only one-shot observations used for residual functional estimation, which makes it difficult for the updated forward simulator to generalize to other unseen training data (e.g., new input-output pairs), as our experiments indicate (results not shown).

Bearing the above challenges in mind, when evaluating the performance of MEC, we do not particularly focus on inspecting the generalization ability of the updated forward simulator (after all, the goal of data assimilation is to estimate the ground truth corresponding to real observations). Instead, we adopt the following cross-validation procedure, namely, for a given ensemble of model variables in the experiment with MEC, we compare the corresponding data mismatch values, when the residual term is used or not used in the forward simulator (cf. Eq (33)). Such a comparison aims to examine whether the introduction of the residual term to the forward simulator helps match real observations better or not.

Following this notion, Fig 15 shows the box plots of data mismatch differences at different iteration steps, with respect to the experiment with MEC. At a given iteration step (hence a given ensemble of model variables), these differences are obtained by subtracting data mismatch values which are calculated with the residual term in the modified forward simulator in Eq (33) (as in Fig 14(b)), from the corresponding data mismatching values which are calculated without including the residual term in Eq (33). Positive difference values in the box plots thus imply that the presence of the residual term in Eq (33) is useful for helping match real observations better, and vice versa. From this perspective, Fig 15 suggests that, with the initial ensemble of kernel parameters, the effect of including the residual term in Eq (33) at iteration step 0 is mixed, and there are substantial numbers of difference values residing on both sides of zero (although overall the number of positive values does seem to dominate). After one iteration (at iteration step 1), the model qualities are improved in terms of RMSE (cf. Fig 14(d)), meanwhile the number of positive difference values also increases. However, as models are further improved, the number of positive difference values does not necessarily always dominate, as one can spot in the box plot at iteration step 3. Nevertheless, as the iteration process continues, this kind of “over-correction” diminishes. Eventually, the number of positive difference values dominates at the final stage, while the RMSEs of estimated models tend to gradually reduce.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. Box plots of data mismatch differences at different iteration steps, for the experiment with MEC in the perfect scenario.

At a given iteration step, these differences are derived using data matching values that are calculated with the residual term excluded from Eq (33), minus data matching values that are computed with the residual term included in Eq (33), with respect to the corresponding ensemble of model variables at that iteration step. Therefore, positive data mismatch differences indicate that including the residual term helps match real observations better. For better visualization, we show the box plots from iteration steps 2 to 10 in a separate, zoomed-in subplot in the upper right corner.

https://doi.org/10.1371/journal.pone.0219247.g015

Based on the experiment results in the PS, we conclude that, in this particular case study, even though the forward simulator is perfect, it appears still beneficial to integrate kernel-based MEC into data assimilation for performance improvements.

Results in the imperfect scenario (IS)

Results with N_cl = 1.

In parallel to the results in the PS, we first report the results with N_cl = 1 in the IS. In this case, we also compare the assimilation performance with respect to one experiment where there is no MEC introduced, and another experiment where kernel-based MEC is adopted, with the number of cluster N_cl = 1. The initial ensemble of kernel parameters is generated in the way as in the case study of SLP.

Table 3 reports both data mismatch and RMSE (in terms of mean ± STD) for the initial ensemble, and the final ensembles obtained when MEC is or is not adopted. For the purpose of comparison, we adopt the same initial ensemble as in the PS. From Table 3, one can again see that the use of MEC helps reduce mean values of both data mismatch and RMSE, while retaining higher ensemble spreads in the final ensemble, in comparison to the choice in which MEC is not used. In addition, by comparing Tables 2 and 3, one also spots the impact of imperfection on data assimilation: In the presence of imperfection, the performance of data assimilation is worsened, with mean values of both data mismatch and RMSE in the IS becoming larger than those in the PS.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Means and STDs of data mismatch and RMSE with respect to the initial ensemble, and the final ensembles with or without model-error correction (MEC), in the imperfect scenario.

https://doi.org/10.1371/journal.pone.0219247.t003

The subsequent results in Figs 16–19 are shown in analogy to their counterparts, Figs 12–15, in the PS, respectively. A comparison between these figures are largely consistent with our observations stated in the preceding paragraph. In particular, the box plots of data mismatch differences at different iteration steps, as shown in Fig 19, also indicate that kernel-based MEC is useful for improving data match to real observations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 16. As in Fig 12, but for the experiment results in the imperfect scenario (N_cl = 1).

https://doi.org/10.1371/journal.pone.0219247.g016

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 17. As in Fig 13, but for the experiment results in the imperfect scenario (N_cl = 1).

https://doi.org/10.1371/journal.pone.0219247.g017

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 18. As in Fig 14, but for the experiment results in the imperfect scenario (N_cl = 1).

https://doi.org/10.1371/journal.pone.0219247.g018

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 19. As in Fig 15, but for the experiment results in the imperfect scenario (N_cl = 1).

https://doi.org/10.1371/journal.pone.0219247.g019

Overall, the experiment results presented here confirm again that, in this particular case study, kernel-based MEC helps improve the performance of data assimilation in the presence of imperfection in the forward simulator.

Comparison to an alternative MEC mechanism.

An alternative idea for MEC in data assimilation would be that, in Eq (30), instead of adopting kernel-based functional approximation, one may simply approximate the residual term by an unknown bias term, similar to the strategy adopted in, e.g., [15]. It would then be of interest to see how this alternative MEC method performs, in comparison to kernel-based MEC. For reference later, we call this alternative method bias-based MEC.

In the experiment, we also choose to integrate this bias-based MEC into ensemble-based data assimilation. To initialize an ensemble of biases, we first compute an ensemble of residuals between real observations and simulated observations with respect to the initial ensemble. We then calculate the mean and covariance of the residual, and use these statistics to draw an (initial) ensemble of biases, in a way similar to that we adopted to generate the initial ensemble of model variables. After that, similar to the setting in Eq (31), we augment both model variables and biases, and use the iES to update them in the course of data assimilation.

Fig 20 summarizes the experiment results with respect to bias-based MEC. In comparison to the results with kernel-based MEC in Figs 16–19, it is clear that bias-based MEC tends to result in higher data mismatch and RMSE. In terms of mean ± STD, the data mismatch values of the final ensemble for bias-based MEC are (7.9847 × 10⁶) ± 80.1288, and the corresponding RMSEs are 1.9767 ± (2.7584 × 10⁻⁴). Relative to the mean values, the tiny STDs of the final data mismatch and RMSEs suggest that ensemble collapse is a severe issue in the experiment with bias-based MEC.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 20. Experiment results with bias-based MEC in the imperfect scenario.

https://doi.org/10.1371/journal.pone.0219247.g020

The relative under-performance of bias-based MEC might be partially attributed to the simplifying assumptions, e.g., whiteness, stationarity, and normality [15], regarding simulator imperfection. To see this, Fig 21 shows the histogram of the mean of the residuals with respect to the initial ensemble. As one can see there, the distribution of the mean residuals does not seem to resemble a normal distribution well.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 21. Histogram of the mean of the residuals with respect to the initial ensemble.

https://doi.org/10.1371/journal.pone.0219247.g021

The impact of the number N_cl of clusters.

As in SLP, when using kernel-based functional approximation for MEC, one can also choose to first group model variables into different clusters, and then estimate an ensemble of kernel parameters for each cluster. The final residual functional is taken as the weighted average of the individual (kernel-based) approximation functional estimated from each cluster, similar to the idea described in Eq (50). Note that, in this case study, we know that the initial ensemble of model variables is generated using fast Gaussian simulation [4, 8]. Therefore, in principle, either the joint or the marginal distribution of the model variables is unimodal, and intuitively there would be no need to consider an multi-modal-based approximation strategy. Nevertheless, as we will show below, the multi-modal strategy may help improve the performance of data assimilation.

Fig 22 reports the box plots of RMSEs with respect to the final ensembles that are obtained in data assimilation using different N_cl values. In the experiment, N_cl takes its value from the set {1, 2, ⋯ , 10}. As one can see in Fig 22, except for the case with N_cl = 2, all other choices tend to result in lower RMSEs, in comparison to the choice of N_cl = 1. This thus suggests that, similar to the results in SLP (cf. Fig 11), one may obtain better assimilation performance by using a relatively large value for N_cl that exceeds the actual number of mode(s) in the distribution of model variables. On the other hand, though, the optimal choice of the value of N_cl remains to be an open problem in the current work.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 22. Box plots of RMSEs of the final ensembles, obtained with different numbers N_cl of clusters.

https://doi.org/10.1371/journal.pone.0219247.g022