Participants EEG experiment.

Nine participants (6 male, 3 female), aged 24–44 years (mean age 30), all but one right-handed, volunteered to take part in the experiment. All reported having normal hearing and no history of neurological disorder. The subjects differed with respect to their musical education and practice: two of them reported intensive musical training of more than 15 years and on more than one instrument, five of them modest amounts of musical training (mean: 7 years) and two of them no musical training beyond obligatory lessons at school. Subjects completed a questionnaire about their musical activities and preferences. Participants gave written consent and the study was performed in accordance with the Declaration of Helsinki. The study protocol was approved by the Ethics Committee of the Charité University Medicine Berlin.

Participants behavioral experiment.

In a separate experiment 14 new participants (9 male, 5 female, 12 right-handed, 2 left-handed) volunteered to take part in the behavioral experiment. Their musical experience ranged from no musical training beyond obligatory courses at school to more than 15 years of musical training (mean 6 years).

EEG experiment.

The present study aims at approximating listening situations that resemble those of everyday life. Consequently, participants were not given any special task and were just asked to listen in a relaxed manner and with closed eyes during the presentation of the musical stimuli.

The main experiment was split into three blocks. In each block the nine stimuli were presented in a different order that was designed such that two piano pieces, two violin pieces or two non-musical stimuli never occurred in direct succession.

Behavioral experiment.

In the behavioral experiment subjects were given a short introduction to the concept of tension in music. Then, they were instructed as following:

“In the following you listen to eight excerpts, divided by short breaks. Six of them are musical pieces, two are non-musical. Your task is to indicate continuously with the joystick how you experience the evolution of tension of each piece of music. Please start each piece with the joystick at zero position. If you experience an increase in tension in the music, move the joystick up. If you experience a decrease in tension, release the joystick towards the zero position. You will have the opportunity to practice this before the beginning of the experiment. These tension ratings reflect your individual experience. Therefore it is not possible to do this right or wrong.”

After one practice trial with a different music stimulus, each music stimulus was presented once while joystick movements were recorded. Since the stimulus Chord sequence, due to its simplicity and repetitiveness, was not assumed to give rise to the perception of tension in the listeners it was not part of the behavioral experiment.

Audio analysis.

For each stimulus the power slope was determined by segmenting the audio signal into 50% overlapping time frames of 50 ms width and then calculating the average power of each window. Subsequently, the resulting time course was smoothed using a Gaussian filter of three samples width and the first derivative was taken. Then, the extracted power slope was resampled to match the sampling frequency of the EEG.

We chose a set of nine musical features that cover a broad spectrum of timbral, tonal, and rhythmic categories of music. Sound intensity, which can be considered as an approximate measure of loudness, is a stimulus feature that influences a variety of brain responses [28], [2], [29]. Sharpness, defined as the mean positive first derivative of the waveform power [30], has been found to be an important cue for cortical tracking of the speech envelope [31], [30]. Furthermore, with Spectral centroid, Spectral entropy and Spectral flux we included a set of three spectral features that describe pitch- and timbre-related aspects of sounds. The fluctuation spectrum of an audio signal contains the periodicities contained in a sound wave’s envelope. For musical sounds with rhythmic regularity peaks in the fluctuation spectrum correspond to beat-related frequencies. The fluctuation spectrum can be further characterized by the Fluctuation centroid that indicates where the ‘center of mass’ of the spectrum is and by Fluctuation entropy which is a measure of rhythmic complexity. Pulse clarity is a composite feature that indicates how easily listeners perceive the underlying rhythmic or metrical pulsation of a piece of music. It has been introduced and perceptually validated in [32] and since then has been used in numerous studies [33], [34], [35], [36], [37], [38]. Key clarity is a feature that estimates the salience of key.

Sound intensity and Sharpness were calculated in Matlab (The MathWorks Inc., Natick, Massachusetts). All other features were extracted using the MIRToolbox [39]. Sound intensity and the three spectral features were calculated for time frames of 50 ms overlapping by 50%. Sharpness, Fluctuation centroid and Fluctuation entropy, Pulse clarity and Key clarity were determined for time frames of 3s with a 33% overlap. Subsequently, all features were re-sampled to match the sampling rate of the time-resolved CACor of 3.3 Hz. Additionally, for each stimulus a global description was obtained by taking the mean of each music feature in order to derive a rough estimation of the specific characteristics of each stimulus.

Preprocessing of tension ratings.

In a first step the 14 continuous tension ratings for each stimulus were examined with respect to their homogeneity within the group. This was done by calculating all pairwise correlation coefficients between the single subjects’ ratings. Next, the percentage of significantly correlated pairs was determined with a permutation testing approach where surrogate versions of all 14 tension ratings were generated (see Section Methods, Subsection ‘Significance of correlation‘). With percentages of significantly correlated pairs of tension ratings ranging between 32% and 38% for the non-musical stimuli and Williams and between 47% and 76% for the other music stimuli calculating mean tension ratings for further analysis seemed appropriate. When examining the relationship between tension ratings and stimulus parameters it has to be considered that typically ratings lag behind the stimulus. We utilize the influence of sound intensity on tension ratings that has been reported in the literature [22], [40], [21] to determine the optimal time lag between tension ratings and stimulus. We calculate the cross-correlation between the Grand Average z-score-transformed tension ratings and sound intensity for time lags from 0 to 3s in steps of 10 ms and identify an optimal lag for each stimulus. The resulting time lags ranged from 760 to 1110 ms for the music stimuli, for the non-musical stimuli cross-correlation sequences did not reveal a clear peak. Therefore we set the time lag to 1000 ms. All results related to tension ratings in the following were obtained after correcting for these time lags and re-sampling to the sampling rate of the time-resolved CACor of 3.33 Hz.

In addition to group averages, collections of tension ratings can also be examined with respect to the degree of inter-subject coordination a stimulus exerts on the ratings. Following the framework of activity analysis [41] we determined for each of the 50%-overlapping 1000 ms time frames the percentage of increasing tension ratings (out of the 14 ratings of all subjects) and that of decreasing ratings. Subsequently, we evaluated whether these proportions are significant by applying a permutation-based testing framework (for details see [42]). This resulted in a set of time windows with significantly coordinated rating activity for each stimulus. This can be used to determine a global Coordination Score for each stimulus that indicates how much significantly coordinated rating activity (either increasing or decreasing) occurs in the course of the stimulus. This measure allows comparing stimuli with respect to their ‘ability’ to coordinate behavioral responses.

EEG analysis: preprocessing and ERP analysis.

Prior to the regression analysis generic preprocessing was performed on the EEG data. The EEG data was lowpass-filtered using a Chebyshev filter (with passbands and stopbands of 42 Hz and 49 Hz, respectively) and downsampled to 100 Hz. After that, a 1-Hz highpass-filter was applied. The EEG signal we used for further analysis, therefore, is a broadband signal in the range of 1 to 42 Hz. In principle, one could assume that the EEG features of interest in this analysis may be contained in the low frequency bands, e.g., in the theta and alpha band. Our preliminary analyses, however, showed that the best match between EEG projection and audio power slope can be obtained using the broadband signal.

Since electrodes A1 and A2 were not contained in the head model used in the later analysis (see Section ‘EEG analysis: Calculating Cortico-Acoustic Correlation‘) they were not considered in the analysis. In order to remove signal components of non-neural origin, such as eye, muscle or movement artifacts while preserving the overall temporal structure of the music-related EEG responses we separated the 61-channel EEG data into independent components using the TDSEP algorithm (Temporal Decorrelation source SEParation, [43], with time lags of tau = 0, …990 ms). ICA components that were considered as purely or predominantly driven by artifacts based on visual inspection of power spectrum, time course and topography (see also [44] and [45]; the Supplementary Material of [44] contains example plots of typical components of various types of artifacts.) were discarded and the remaining components were projected back into the original sensor space.

The stimulus Chord Sequence is an isochronous sequence of chords that only differ in frequency content. This represented an opportunity to apply a conventional ERP analysis approach for comparison with the results of the proposed method. The continuous EEG data was segmented into epochs of 300 ms length, starting at tone onsets, and averaged. Since the pre-processing procedure included already highpass-filtering, a baseline correction was not applied. This technique cannot be applied to any other of the stimuli, owing to the high variability of onset characteristics in naturalistic sounds.

EEG analysis: Calculating Cortico-Acoustic Correlation.

The analysis aims to obtain a representation from the ongoing EEG that reflects brain responses to the onset ‘landscape’ of a music stimulus. It can be divided into four modules for (1) preprocessing EEG and audio data, (2) calculating spatio-temporal regression filters for optimally extracting EEG features, (3) applying the derived filters to new data in order to extract EEG projections, and (4) transforming the spatio-temporal filters into a representation suitable for neurophysiological interpretation. Finally, the synchronization between the extracted EEG projections in (3) and the audio stimulus is examined at several levels of temporal resolution and the results are related to behavioral results and stimulus characteristics. Fig 2 summarizes steps (1) to (4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. EEG feature extraction.

(1) In the first step of the analysis the 61-channel EEG signal (after generic preprocessing, see Methods) is temporally embedded and the power slope of the audio signal is extracted. In the training step (2) the embedded EEG features are regressed onto the audio power slope (Ridge Regression). After that (3) the resulting spatio-temporal filter (regression weight matrix) reducing the multichannel EEG to a one-dimensional projection is applied to a new presentation of the same stimulus. The regression filter can be transformed (4a) into a spatio-temporal pattern that indicates the distribution of information which is relevant for the reconstruction of the audio power slope. This spatio-temporal pattern, in turn, can be (4b) decomposed into components (derived with the MUSIC-algorithm) which have a scalp topography and a temporal signature. The EEG projections obtained in (3) subsequently are examined with respect to Cortico-Acoustic correlation (CACor).

https://doi.org/10.1371/journal.pone.0141281.g002

ERP responses are phase-locked to rapid intensity changes that indicate tone onsets in the music stimulus. In order to extract a component from the ongoing EEG that optimally reflects onset-related brain responses we apply Linear Ridge Regression [46] to maximize the correlation between EEG and the audio power slope of the stimulus, an acoustic feature that represents the intensity changes that are expected to trigger ERP responses.

1. Pre-processing of EEG data: temporal embedding
   Since it is not clear a priori by how much the EEG response lags behind the presented stimulus, we perform a temporal embedding of the EEG signal, as proposed in [47], which allows to deal with relationships between signals with unknown delay: to the EEG data matrix X of m observations x n channels additional dimensions that are copies of X, time-shifted by 1,…, 30 data points are added as ‘artificial’ channels, resulting in a data matrix of the dimensionality m x 31. This allows to capture brain responses within a latency of 0 to 300 ms. The audio power slope is extracted as described above.
2. Training: Calculation of spatio-temporal regression filters
   We train Linear Ridge Regression models to optimize the correlation between the temporally embedded EEG of single subjects and the power slope of the audio signal. This is done in a leave-one-recording-out cross-validation approach. For a given stimulus and subject a spatio-temporal filter is trained on the concatenated EEG of two presentations (training set) and applied to the EEG of the remaining third presentation (test set). This is repeated three times in all possible combinations, so that each presentation is once the test set. The temporal embedding performed in step 1 blows up the dimensionality of the EEG signal from 61 to 61 x 31 ‘channels’ and strongly requires regularization. Therefore, instead of using the standard Ordinary-Least-Squares solution for Regression, we apply Ridge Regression where regularization is introduced by adding a ‘ridge’ to the empirical covariance matrix according to an analytically determined shrinkage parameter [48]. This is a common technique to stabilize the estimation of covariance matrices [49].
3. Application of regression filters to new data
   The cross-validation procedure described above resulted in a one-dimensional EEG projection for each stimulus presentation, and, accordingly, in 27 EEG projections for one stimulus (for nine subjects and three presentations each). This set of EEG projections served as basis for examining the relation between brain signal and onset structure at several levels.
4. Transformation and decomposition of spatio-temporal filters
   The spatio-temporal filters that result from step 1 are matrices of the dimensionality n channels x t time lags (here: 61 channels x 31 time lags = 1984 dimensions) which correspond to a sequence of scalp topographies. However, these maps are not appropriate for neurophysiological interpretation; see [50] for a detailed discussion. Instead, the filter maps need to be transformed into patterns (see transformation (4a) in Fig 2) which specify how the activity of the respective sources in brain is projected on the scalp (see [51] for details on the underlying forward model of EEG generation and [50] for the details of the actual transformation). This allows to examine how the information that is used to reconstruct the power slope of the stimulus is distributed in space (on the scalp) and time (relative to the stimulus). An example of such a spatio-temporal pattern is given in the upper panel of S1 Fig. Interpreting the matrix representation of this information is not very intuitive. Therefore, it is desirable to distill a representation from the spatio-temporal regression patterns that is reduced in dimensionality and that has a form that allows for a better comparison with conventional ERP components, e.g., consisting of one or more 'components' with a spatial pattern and a time course.

In the following, we describe such a technique that is based on Singular Value Decomposition (SVD) which is a standard method for expressing the characteristics of a matrix by means of its eigenvectors and -values. Eigenvectors are orthogonal to each other. It is, however, questionable that neural sources and their physiological patterns should obey an orthogonality constraint. Therefore, we apply a variant of the the MUltiple SIgnal Classification (‘MUSIC’) algorithm [52] to extract physiologically plausible estimates of neural components that have produced the subspaces which contain the characteristic features of the spatio-temporal pattern matrices, but are not necessary orthogonal. This transformation corresponds to (4b) in Fig 2.

The 61 x 31 dimensional matrices were factorized using Singular Value Decomposition (SVD) and subsequently reduced to a lower-dimensional subspace covering a predefined portion (98%) of the variance. To the spatial SVD components the MUSIC algorithm is applied in order to extract a new set of (non-orthogonal) patterns. The variant of the MUSIC algorithm used here finds a set of dipoles that, according to pre-defined standard head model and a least-squares metric, match the observed patterns best. These dipoles can be projected out to results in a set of (non-orthogonal) spatial MUSIC components, in our case 2–4 components per subject and stimulus. Furthermore, corresponding time courses can be determined based on the original SVD factorization. In summary, the MUSIC procedure achieves a dimension reduction and decomposition of the spatio-temporal regression patterns into a set of dipole-source related scalp topographies with corresponding time courses. The lower panel of S1 Fig shows three spatial and temporal MUSIC components that were derived from a spatio-temporal matrix (upper panel) by means of the MUSIC algorithm.

Comparing these decompositions between subjects is difficult, since no canonical representation can be derived. Visual inspection of this large collection of scalp patterns suggested the presence of a scalp pattern that was consistent between subjects and stimuli and resembled the scalp topographies of the onset-related ERPs for Chord sequence. In the decomposition this spatial component occurred either as a first, second or third component. Since the present analysis is interested in brain responses to tone onsets we extracted a reference pattern from the onset-related ERPs obtained by classical ERP analysis for Chord sequence. This was done by averaging the scalp topography of all subjects within a 20ms time window that corresponds to the maximum amplitude of each subject’s onset ERP at channel Fz. The time windows were determined manually and ranged between 130 ms and 280 ms after tone onset. For the results of the ERP analysis and the reference pattern see Section ‘Interpretation of spatio-temporal patterns’. Subsequently, the distance between all spatial components and the reference pattern was calculated and the most similar pattern for each subject and stimulus was selected for further comparison. In 43% (38%, 18%) of the components the selected pattern occurred in the first (second, third) MUSIC component.

EEG Analysis: Applying Cortico-Acoustic Correlation.

In a first step we calculate the global correlation coefficient between EEG projection and the audio power slope, termed Cortico-Acoustic Correlation (CACor): We determine the CACor coefficient for each single presentation (nine stimuli, nine subjects, three presentations per stimulus and subject) and assess its significance (details as described below). This aims at assessing whether the onset structure of the stimulus is significantly reflected in the brain signal. In S2 Fig this step of analysis corresponds to the ‘presentation level’. In addition, the Grand Average of all 27 EEG projections per stimulus is calculated and a group level CACor coefficient is derived for each stimulus.

Subsequently, the relationship between CACor, behavioral results and stimulus properties is compared between stimuli (see column ‘Between stimuli’). Based on the significance of CACor coefficients for each single presentation a global CACor score is calculated for each stimulus that specifies in how many of the 27 presentations a significant CACor is present. These can be summarized in a CACor score profile for the nine stimuli. In a similar fashion, a Coordination score profile is constructed from the Coordination scores derived in the behavioral experiment (see above). Additionally, for each of the nine acoustic/musical properties that were obtained in the audio analysis a profile that describes the magnitude of the respective stimulus feature for all stimuli was constructed. The pairwise correlation between CACor score profile, Coordination score profile and music feature profiles is calculated.

In a second step the dynamics of the brain-stimulus synchronization within each stimulus is examined by calculating a group-level CACor in a time-resolved manner (see also S2 Fig rightmost column): For each stimulus the 27 EEG projections (three for each subject) are averaged and segmented into 90% overlapping time frames of 3 s duration. Subsequently, CACor is calculated for each time frame, resulting in a time course of correlation coefficients that has a sampling rate of 3.3 Hz. Correlation coefficients between this time course and the time courses of the nine acoustic/higher-level music features are determined. In an analogue manner, the relation between mean tension ratings and music features is determined. S2 Fig gives a complete overview about data types, features and correlation coefficients that are calculated in the present analysis.

Significance of correlation.

In the present analysis we encounter two statistical problems that are typical for correlation-based analysis of time series data related to naturalistic music stimuli:

First, both, the EEG signal and the audio power slopes (and also other features derived from the music stimuli) contain serial correlation, i.e., subsequent samples are not independent of each other. Thus, the assumptions underlying standard tests for significance of correlation are not satisfied. This can be accounted for in different ways: Pyper et al. [53] propose a method that estimates the effective degrees of freedom based on the cross-correlation between the two respective time courses and, then, calculate p-values based on the corrected degrees of freedom. In the music domain this has been applied in [33]. A different strategy can be pursued by applying randomized permutation tests with surrogate data as proposed in [54] and described for a music-related analysis in [37]

Notably, in the present context, both methods have some specific drawbacks. Pyper et al.’s method ‘punishes’ cross-correlation between two time series in order to avoid inflated correlation coefficients—a technique that also reduces the sensitivity. Permutation tests, in principle, can be regarded as more sensitive, but permutation tests in combination with temporal embedding may be problematic. The opportunity to account for a delayed relationship between both signals that is given by the temporal embedding can, dependent on the relation between window length of embedding and structure of the target function, severely affect the sensitivity of the permutation test.

In order to gain more insight with respect to this aspect, we assessed the significance of CACor at single subject and single presentation level using both methods. For Pyper et al.’s method the maximal time lag of autocorrelation that was taken into account was 2 s.

As a second statistical problem a complex auditory stimulus comprises a multitude of features that are not independent of each other, but are correlated to different degrees. For the present selection of stimuli the correlation coefficients ranged between r = -0.74 and r = 0.93. To account for this correlation, the relation between CACor time courses/tension ratings and music features was determined using the partial correlation coefficient [55]. This post-hoc approach is a way to exert statistical control over variables in a setting where experimental control on the different aspects that are to be investigated is ruled out by design and has been applied in a similar context before [37]. For assessing the significance of partial correlation coefficients, however, the permutation testing approach is not optimal, since it is not possible to generate surrogate data with the same intrinsic relations as the extracted music features. Therefore, Pyper et al.’s method (maximal lag 10 s) is applied to correct for autocorrelation in order to estimate the significance of partial correlation coefficients. The resulting p-values were corrected for multiple comparisons (false discovery rate (FDR), q < 0.05).

Cortico-Acoustic Correlation for single presentations.

The table depicted in Fig 4 gives CACor coefficients for each single subject and each stimulus presentation. The bottom line of each table contains the correlation coefficient for the Grand Average EEG projection (average of the 27 EEG projections for each stimulus). Note that correlation coefficients cannot be compared between stimuli, since duration and autocorrelation properties differ between stimuli. Shaded cells indicate significant correlation at the level of alpha = 0.05. Significance was determined by applying permutation tests and subsequently correcting for the number of 27 presentations per stimulus (see Methods, Subsection ‘Significance of Correlation‘). Stimuli were ordered according to the total number of presentations with significant CACor (called CACor score in the following). Since these were derived in a cross-validation approach (see Methods, Subsection ‘EEG analysis: Calculating Cortico-Acoustic Correlation‘) significant correlation coefficients can be regarded as reflecting a genuine influence of the stimulus on brain responses that generalize across several presentations of a stimulus. To give an impression of the extracted EEG projections and their relation to the audio power slope three examples are shown in Fig 5. Fig 6A summarizes the corresponding CACor scores into a CACor score profile for the set of nine stimuli. For comparison, both, the CACor score profile that was derived by applying permutation tests (dark blue bars), and that derived by applying Pyper et al.’s method [53] to assess the significance of correlation in signals containing serial autocorrelation (light blue bars) are given. Although there are differences in the absolute CACor scores for both methods, the ranking of the stimuli does not change. Importantly, the comparison shows that the zero scores for Orchestra and Jungle indicate an absence of significant CACor for these stimuli and are not introduced by the permutation test approach. In the following the profile derived by permutation tests is used for further analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Table of CACor coefficients.

CACor coefficients for each single subject and each stimulus presentation; GA: Grand Average (N = 27). Pink shading indicates significant positive correlation between EEG projection and audio power slope. Significance was determined by applying permutation tests and subsequently performing Bonferroni-correction for N = 27 presentations per stimulus.

https://doi.org/10.1371/journal.pone.0141281.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. EEG projections.

Three examples that show 3s-long segments of an extracted EEG projection (red) for a single stimulus presentation and a single subject and the respective audio power slope (blue) of Chords, Chopin Etude and Jungle noises. Note that in the optimization procedure a time lag between stimulus and brain response is integrated in the spatio-temporal filter, and that, consequently, the resulting EEG projections shown here are not delayed with respect to the audio power slope. The correlation coefficients indicate the magnitude of correlation for the shown segment of 3 s.

https://doi.org/10.1371/journal.pone.0141281.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. CACor score profile and Coordination score profile.

The CACor score profile for the set of nine stimuli summarizes in how many of the 27 presentations significant Cortico-Acoustic Correlation was detected. Significance of correlation was determined (1) in a permutation-testing approach (darkblue bars) and (2) with Pyper et al.’s method (light blue bars) to estimate the effective degrees of freedom (see Methods). b) Behavioral Coordination score profile. All profiles are sorted according to the descending CACor score.

https://doi.org/10.1371/journal.pone.0141281.g006

Stimuli differ strongly with respect to the consistent presence of significant CACor for the 27 presentations with a CACor score of 24/27 (89%) presentations for Chord sequence, 18/27 (67%) presentations for Chopin, 13/27 (48%) presentation for Rachmaninov, 11/27 (41%) presentations for Vivaldi, Spring, 9/27 (33%) presentations for Vivaldi, Summer 4/27 (15%) presentations for Bach, 2/27 (7%) presentations for Williams and 0/27 (0%)presentations for Orchestra and Jungle. In the Grand Average CACor is significant for Chord sequence and all music pieces, but not for Orchestra and Jungle.

Subjects differed with respect to the total number of presentations with significant correlation, scoring between 4/27 presentations (S1, S9) and 15/27 presentations (S5). With 25/81 presentations with significant correlation for first presentation, 27/81 for the second presentation and third presentation no influence of repetition of the stimuli was found.

CACor, tension ratings, and stimulus features.

Fig 6B compares the behavioral Coordination score profile for the nine stimuli with the CACor score profile. Note that the regular stimulus chord sequence was not included in the behavioral experiment. However, in the CACor score profile blank bars for Orchestra and jungle denote a score of zero. Table 1 gives correlation coefficients (Spearman’s rho) that quantify the correlation of (a) CACor score profile and (b) Coordination score profile with music feature profiles that relate to nine stimulus features. Each of these is represented by scores that indicate the average magnitude of a specific acoustic/higher-level musical property for the nine stimuli. S3 Fig gives a detailed overview how stimuli differ with respect to each of the nine acoustic/higher-level musical properties. The bottom line of Table 1 contains the correlation between CACor score profile and Coordination score profile. The CACor score profile is significantly positively correlated with the Sharpness profile and significantly negatively correlated with the Spectral centroid profile. Furthermore, moderate, but non-significant negative correlation of the CACor score profile with the Fluctuation entropy profile is present. None of the correlation coefficients between Coordination score profile and music feature profiles reached significance. CACor score profiles and Coordination score profiles are significantly correlated with r = 0.9, p = 0.005.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Relation between CACor scores and music features.

Spearman’s correlation coefficient (a) between CACor score profile and music feature profiles for nine acoustic/higher-level music features, (b) between Coordination score profile and music feature profiles. Bottom line: Correlation between CACor scores and Coordination scores. Asterisks indicate significant correlation.

https://doi.org/10.1371/journal.pone.0141281.t001

Dynamics of CACor.

We examined how changes in group-level CACor (correlation between the Grand Average EEG projections and audio power slope) relate to changes in stimulus features and in mean tension ratings during the course of a stimulus. Note that in this part of the analysis correlation coefficients indicate whether CACor and a particular property of the music stimulus co-vary, but do not necessarily inform about significance of global CACor. The full set of partial correlation coefficients (see Section ‘ EEG Analysis: Applying Cortico-Acoustic Correlation’ for details of the calculation) is given in S2 Table. Of these, only a small number of correlation coefficients were found significant, most consistently we found a significant positive correlation of mean tension ratings with Sharpness and a negative correlation with Spectral Entropy.

Out of the nine (eight for the tension ratings, respectively) stimuli the romantic piano pieces (Chopin and Rachmaninov) show the clearest relations between CACor and music features, followed by both Vivaldi stimuli and, then, by one of the non-music stimuli. A similar trend is visible for the relation between tension ratings and music features.

Neurophysiological interpretation.

The proposed regression-based method not only derives a condensed and optimized representation of the multi-channel EEG that subsequently can be examined at several time scales. It also allows to learn about the temporal and spatial distribution of information in the EEG signal that is relevant for the optimization process.

For each subject and stimulus spatio-temporal patterns derived from the regression filters were decomposed into several components (called MUSIC components according to the decomposition algorithm) with a temporal and spatial dimension each (see Methods, Subsection ‘EEG analysis: Calculating Cortico-Acoustic Correlation‘). To obtain a common basis for comparison in this large set of scalp topographies we selected the MUSIC component with the scalp topography that is most similar to a reference topography that was derived by averaging EEG data for all tone onsets of the stimulus Chord sequence. This onset-related ERP (see Fig 7) was fronto-centrally distributed with a negative deflection at appr. 100 ms after tone onset, followed by a positive peak between 150 and 200 ms after tone onset and, thus, represents the typical N1-P2 complex that is elicited as a reaction to sound onset [57], [58]. The fact that for all stimuli the MUSIC component subspace (that represents 98% of the variance of the information ‘relevant’ for optimizing the EEG projection) contained a similar scalp pattern links our results to canonical onset-related ERP components and contributes to their neurophysiological interpretability

The scalp pattern of the present MUSIC component, in particular, resembles Schaefer et al.’s results where a fronto-central component occurred consistently in the decomposed grand average ERP response of participants who listened to fragments of naturalistic music. This group-level ERP had a time course that was correlated with the stimulus envelope with a time lag of 100 ms. Kaneshiro et al. [59] found a similarly distributed fronto-central component as most reliably occurring component in the EEG of listeners who were presented with naturalistic vocal music.

In contrast, the temporal dimension of the selected MUSIC components is much more variable across stimuli (see Fig 7). It is important to recognize that the time courses of MUSIC components differ from averaged ERPs (even though they are on the same time scale). They indicate the weighting of the corresponding scalp pattern over time and, thus, resemble a convolution model or FIR filter rather than an ERP time course. Thus, these time courses describe how information is integrated over time in the optimization process. Typically, such representations are viewed as an estimate of the impulse response of the auditory system [6]. Nonetheless, time lags with large weights in principle can be compared to latencies of canonical ERP components. In summary, our results suggest that in stimuli where a sizeable level of CACor is detected, the spatio-temporal patterns contain neural components that are neurophysiologically plausible and reminiscent of an N1-P2 complex. The variability of their temporal dimension suggests that the shape and latency of the N1-P2 response is influenced by stimulus properties, a finding which agrees with the sensitivity of the N1-P2 complex to several auditory features [2], [3].

Within-stimulus dynamics of CACor, music features and tension ratings.

Considering the variable surface [62] of a naturalistic stimulus, it can be assumed that stimuli not only differ with respect to global measures of CACor, but also that the synchronization between brain signal and stimulus varies during the course of the stimulus. In order to learn how the dynamics of different acoustic/higher-level music features influence this synchronization we compared time-resolved CACor and the time courses of the nine acoustic/ higher-level music features for each stimulus. Our results amount to a small but distinct set of significant correlation coefficients. In particular, Sharpness, which was one of the main influences on CACor scores found in the between-stimulus comparison of CACor, and Spectral Entropy seem to modulate local within-stimulus CACor. This, again, points to an important role of ‘auditory edges’ and spectral properties in the cortical processing of note onsets. It demonstrates that even at this fine-grained time scale a direct relation of these properties to brain-stimulus synchronization can be detected.

Interestingly, Sharpness was the one feature that co-varied with tension ratings most consistently (see S2 Table), while Sound intensity, Spectral Entropy, and Fluctuation Entropy influenced tension ratings only in one stimulus each. This result adds new aspects to previous findings that identified loudness (which is closely related to Sound Intensity) as a main factor for the experience of tension [26] [25] [42] [63] [64]. In [21] harmonic and melodic structure were demonstrated to influence felt tension in addition, whereas their relative contribution depended on the specific stimulus. Our results show that tension is an even more multi-faceted phenomenon since Sharpness can be characterized as ‘acoustic’ (rather than related to higher-level musical structure) and rhythm-related.

A rather general notion of 'tension' has been associated with conflict, instability, or uncertainty [65]. Along these lines, a tentative explanation of the present influence of Sharpness on the experience of musical tension may be formulated: A change from a high level of Sharpness which is characterized by the presence of distinct salient ‘auditory edges’ to a passage where musical events are less clearly defined can be thought to produce uncertainty in the listener and may lead to an increase in experienced tension.

From the perspective of tension as a possible link between predictive processes and emotion [65] [66] our findings may also be further interpreted as reflecting different degrees of predictability of future events: Salient auditory edges (that in our stimuli often go together with clear rhythm) promote the formation of rhythmic and metric predictions. However, the interplay between expectation, fulfilment or surprise that is thought to lead to emotional responses to music is complex and not yet fully understood [21]. Still, this theory may provide a tentative explanation how a stimulus property that can be considered as a ‘merely’ acoustic one contributes to an emotion-related aspect of experiencing music.

The set of nine stimuli in this exploratory analysis contained a wide range of different sounds. Of these, two full length romantic piano pieces represented the stimuli for those influences of stimulus structure, both, on brain-stimulus synchronization and tension ratings was detected most consistently. Acoustically, this can be related to the fact that the piano, as a struck string instrument, has a characteristic attack ‘thump’ [67] and therefore is likely to provide salient onsets that are reflected well in the brain response. Remarkably, the two piano pieces (within our set of stimuli) represent the most complex musical surfaces as (typically for this period) they contain long, lyrical melodies, wide skips, chromaticism, strong contrasts, expressive performance and interpretive freedom (rubato). These highly complex pieces reveal a relation between stimulus structure and physiological measures and behavioral measures most clearly. To a lesser extent, this relation is visible in two movements of a baroque violin concerto, that feature a rhythmic regular pulse and a contrast between ‘solo’ passages of the solo violin and ‘tutti’ passages of the orchestra. Taken together, this suggests that structural variance/richness and strong contrasts are aspects of music that give rise to a clear physiological and behavioral reflection of structural elements of music, a finding that represents a strong argument for experimental paradigms using full-length naturalistic music stimuli.

Conclusion.

The present results demonstrate that sequences of note onsets which constitute original complex music can be reconstructed from the listener’s EEG using spatio-temporal regression-derived filters, leading to a significant Cortico-Acoustic Correlation at the level of single subjects and single presentations. The characteristics of the extracted brain signatures represent a link to canonical onset-related ERPs, such as the N1-P2 complex, that can be extracted by averaging techniques. Stimuli that consistently led to significant Cortico-Acoustic Correlation had high level of Sharpness, were rhythmically simple, or were dominated by low-frequency content. This points to an important role of the presence of a simple salient beat and percussive elements in physiological reactions to music. The same stimuli produced strongly coordinated ratings in group of 14 listeners, a finding that provides a tentative link from Cortico-Acoustic Correlation to conscious experience of music. At a temporally more resolved level this relationship between CACor and stimulus properties was confirmed, while tension ratings seemed to reflect the same properties differently. These results show that the proposed approach enables to investigate the processing of brain responses to tone onsets in complex sounds with enhanced sensitivity and, thus, may represent a valuable tool in a range of applications where naturalistic listening scenarios are of interest.