Subjects

We studied 20 subjects recruited from the local community (mean age 34.6 ±10.66 years) with no history of neurological or psychiatric illnesses. All subjects signed an informed consent to participate in this study. The Institutional Review Board at the University of Göttingen approved this study.

Image acquisition

All subjects underwent a first MRI in a 3T Siemens Magnetom TIM Trio (Siemens Healthcare, Erlangen, Germany) equipped with an 8-channel head coil for signal reception (Invivo, Gainesville, Florida) located at the University of Göttingen, yielding: 1) two 2 MP-RAGE scans with IPAT 2 and averaged after linear co-registration (3D MP-RAGE, TR = 2250ms, TE = 3.2ms, 256×256matrix, 256×256mm FOV, parallel imaging GRAPPA = 2); and 2) diffusion single-shot EPI scan (30-directions with b = 1000 s/mm², TR = 10000ms, TE = 93ms, 128×128 matrix parallel imaging GRAPPA = 2, FOV = 243x243 mm, isotropic 1.9 mm voxel size).

All subjects underwent a second scanning session in the same scanner yielding similar images. The average time between Time 1 and Time 2 in scanner A was 126.4 ±102.8 days (range 12–442).

Finally, all subjects underwent a third scanning session, this time in a different physical unit of the same type of MRI scanner (3T Siemens TIM Trio), equipped with a different head coil (12-channel) employing the same imaging sequences. This scanner was also located at the University of Göttingen. The average time between Time 1 Scanner A and Time 1 in scanner B was 158.4 ±103.6 days (range 21–465).

Image processing

All DICOM images were converted to NIfTI format, and diffusion gradients were extracted using the software mriconvert (http://lcni.uoregon.edu/~jolinda/MRIConvert). The package FMRIB Software Library (FSL)’s Diffusion Toolkit (FDT) was utilized for preprocessing of diffusion-weighted images (DWIs) and for diffusion tensor estimation[15]. The DWIs underwent eddy current correction through affine transformation of each DWI to the base b = 0 s/mm² T2-weighted image.

Gray matter ROIs were obtained from an automatic cortical and subcortical segmentation of the T1-weighted images employing FreeSurfer [16, 17] (http://surfer.nmr.mgh.harvard.edu/) with parcellation into anatomical ROIs according to the Lausanne anatomical atlas, distributed as part of the Connectome Mapping Toolkit (http://www.connectome.ch), yielding 83 ROIs in the subjects’ native T1-weighted space (a total of 42 regions with all regions existing in both hemispheres except for brainstem (1 ROI). Essentially 41 regions are in both hemispheres.) (S1 Table). The ROIS were transformed from the native T1 space into each subject’s DTI space using an affine transformation obtained with FSL’s FLIRT.

Connectivity matrices

For each patient, we performed an estimation of brain connectivity through two commonly used approaches: deterministic tractography and probabilistic tractography. These methods harness advancements from popular and robust neuroimaging tools commonly used by the scientific community.

Probabilistic tractography was estimated by applying FDT’s probabilistic method of fiber tracking [18]. FDT’s BEDPOST was used to build default distributions of diffusion parameters at each voxel, and probabilistic tractography was obtained using FDT’s probtrackx with default parameters, namely 5000 individual pathways drawn through the probability distributions on principle fiber direction, curvature threshold set at 0.2, 200 maximum steps, step length 0.5mm and distance correction (default settings). The connectivity between ROIs was defined as the number of streamlines arriving in one ROI when another ROI was seeded and vice-versa. Seeding and streamline counting was performed in the voxels within the ROI that were located in the boundary between gray and white matter (white matter surface of Freesurfer segmentation). Specifically, the weighted connectivity between ROIs i and j was defined as the number of probabilistic streamlines arriving at the boundary between j and white matter when the voxels in the boundary between i and white matter were seeded, averaged with the number of probabilistic streamlines obtained from the reverse process, i.e, seeding from j boundary, arriving at the I boundary. The calculation of the probabilistic streamlines was corrected based on the distance travelled by the streamline connecting i and j (“distance correction” built into probtrackX). A connectivity matrix A was defined for each subject, where the weighted link Aij corresponded to the number of streamlines connecting i and j divided by the sum of the surface of areas of ROIs i and j.

Deterministic tractography was reconstructed with the software Diffusion Toolkit[19] by seeding all brain voxels. Two types of threshold were used as stopping criterion for fiber tracking. The first one was image mask, i.e., a threshold based on the distribution of the voxel-wise signal intensity observed in the subject’s b0 image, acquired with all diffusion gradients turned off, not sensitive to diffusion direction. This process was performed using the automatic calculation of threshold built into Diffusion Toolkit. The second stopping criterion was the angle threshold, whereby streamlines encountering an angle greater than 45 degrees were stopped. These are default settings.

As expected, the average number of streamlines tracked per subject was lower with deterministic tractography (3.92 x10⁵) compared with probabilistic tractography (4.65 x10¹¹). The process of calculation of ROI connectivity from deterministic tractography is different than the process of probabilistic tractography. With probabilistic tractography, different ROIs are seeded and the number of streamlines arriving at other ROIs is counted. With deterministic tractography, seeding occurs across all white matter voxels and the resulting fibers are then assessed one by one, to evaluate whether the fiber extreme-points connect different ROIs. It is possible that some fibers do not connect ROIs. Thus, the weighted connectivity between ROIs i and j was calculated by the number of deterministic fibers with one extreme in the boundary between ROI i and white matter, and the other extreme in the boundary between ROI j and white matter, or vice versa. The connectivity matrix was normalized based on the distance travelled by each fiber connecting i and j, and the surface of areas of ROIs i and j, according to the formula, proposed by Hagmann et al [1]: w(e) = (2/(Si + Sj)) ∑l/l(f), where Si and Sj represent the surface areas of ROIs i and j, l represents the fiber linking i and j, and l(f) is the length travelled by l.

Importantly, the seed ROIs used in the construction of probabilistic connectivity matrices were the same seed ROIs used in the construction of deterministic tractography matrices. The ROIs were obtained from segmentation of the T1 weighted images, which were acquired in the same session as the DTI images were acquired.

S1 Fig provides an overview of the pre-processing steps, demonstrating, for one representative subject, native T1 images with anatomical ROIs, the non-diffusion images, deterministic tractography streamlines and connectivity matrices.

All raw connectome matrices evaluated in this study are available as S1 Data.

Network properties based on graph theory

For each subject’s connectivity matrix, we calculated graph theory (GT) measures yielding the following parameters: nodal degree (the number of links connecting the node); nodal betweenness centrality (the fraction of all possible network links that involve that node), nodal clustering coefficient (the number of nodes that are neighbors to that node, which are also neighbors of each other); and nodal strength (degree multiplied by the weight of each connection). All GT measures were calculated from non-directed weighted matrices using the brain connectivity toolbox (https://sites.google.com/site/bctnet/).

Reproducibility

We assessed the reproducibility of connectome mapping by evaluating the intra-class correlation coefficient (ICC) between different time point measurements. The ICC was calculated for the absolute agreement between measurements[20]. Assuming that each MRI session is a different “rater”, we evaluated the ICC for each link across all subjects across different MRI scanning sessions. For example, the connectivity between ROIs i and j was assessed for each subject. Hence, for the link i and j, 20 measurements were obtained for each scanning session, and the ICC was calculated across scanning sessions. We assessed the ICC within the same scanner in two different time points and across different scanners. The ICC was calculated using in-house developed scripts based on the methods “Intraclass Correlation Coefficient (ICC)” by Arash Salarian (https://www.mathworks.com/matlabcentral/fileexchange/22099) for Matlab.

We evaluated whether link-wise reproducibility was influenced by distance between ROIs, the overall volume of ROIs and the ratio between ROI volumes. These relationships were assessed through correlations between link-wise ICCs and anatomical features (totaling 12 correlations, 6 for deterministic tractography measures, with 3 for images from the same scanner and 3 for images from different scanners, and 6 for probabilistic tractography measures).

To investigate if links with high reproducibility from deterministic tractography also demonstrated high reproducibility with probabilistic tractography, we performed correlations between measures (2 correlations in total).

To evaluate whether probabilistic tractography yielded higher link-wise ICCs compared with deterministic tractography, we assessed differences in distribution of ICCs through 2 t-tests (one comparing one method versus another using images from the same scanner, and another comparing methods with images from different scanners). We also evaluated whether links with a higher number of tracked fibers across individuals yielded a higher ICC, and whether links with a higher variability in number across subjects (i.e., the standard deviation divided by mean) was associated with a lower ICC. This was assessed through multiple correlations (4 in total per tractography method).

Finally, we evaluated the reliability of GT measures by investigating the ICC across nodal GT measurements. For each subject, a GT measure was obtained for each ROI (for example, degree for ROI i). Similarly, assuming that each scanning session is a “rater”, the ICC was calculated for each node, for each GT measure, and compared across different time points. This step encompassed 14 correlation analyses

Overall, the level of statistical significance was corrected based on the total number of multiple comparisons described above (38 = 12+2+2+8+14). A Bonferroni adjusted p = 0.0013 (i.e., 0.05/38) was defined as the threshold for statistical significance.

MRI scanner effect

We observed a higher reproducibility of connectome mapping when subjects were re-scanned in the same scanner. A larger number of links exhibited a higher ICC when comparing data obtained from the same scanner. Nonetheless, links within large white matter pathways, such as the corpus callosum, superior and inferior longitudinal bundles maintained a high ICC within and across scanners.

Method of calculating connectivity

Probabilistic tractography was associated with a higher number of links with a higher ICC. However it is important to note that for some specific links (as described in S1 Table), the ICC obtained from both tractography methods was equivalent. There was a significant correlation between link-wise ICC from both methodologies, suggesting a consistency in anatomical reproducibility from both tractography approaches.

Deterministic tractography lead to more sparse connectivity matrices (i.e., links with weight equal to, or close to zero). This leads to the speculation that the overall higher reproducibility of probabilistic tractography may be due to those links that are not consistently tracked with deterministic tractography, but resolved with probabilistic methods. Indeed, deterministic tractography reproducibility was also associated with link-weight dispersion (i.e., the variability in the number of fibers per individual). Links with higher inter-subject variability in number of fibers were associated with a lower ICC. This observation suggests that tracks that are more consistently tracked across subjects are also more reproducible.

Importantly, seeding is inherently different between deterministic and probabilistic tractography and it may influence the levels of reproducibility of each method. The purpose of this manuscript is to illustrate the levels of reproducibility of commonly used approaches to reconstruct the structural connectome, rather than a comparison of DTI methods.

Anatomical link-wise characteristics

Overall, we observed that measurement reproducibility was higher for fibers connecting ROIs with their centers of mass within a shorter Euclidean distance. Measurement reproducibility was also higher for fibers connecting larger ROIs. These are intuitive observations, since connections between ROIs located distant from each other imply that the connecting link travels through a longer distance, with possible anatomical intricacies of the fiber pathways. Fiber tracking is therefore more likely to be interrupted as function of the number of connecting steps in the tracking process[2]. Similarly, larger ROIs are likely to generate a higher number of fibers and therefore increase the likelihood of successful tracking. Interestingly, reproducibility of structural connectivity as a function of the size of the ROIs was maintained even for when fiber tracking was corrected based on the surface or volume of the connecting ROIs.

Graph theory metrics

Interestingly, we observed a fairly high ICC for nodal graph theory measures when assessing different scanning sessions (both within and across different scanners) for both tractography approaches. These results suggest that the conformation of the neural network remains fairly stable across scanning sessions, in spite of regional variability in fiber tracking. Specifically, the nodal influence on the network, in relationship with the geometry of the nodal connectivity to the network, as statistically equivalent reproducible and equivalent between measurements. These results are encouraging for future connectivity studies employing graph theory based approaches, as they suggest that the overall neural architecture was largely consistent across scanning sessions and methodological approaches.

General discussion

Given that most neuroimaging studies will employ the same MRI scanner, we believe that our findings related to the reproducibility of data from the same scanner may be our most relevant observation. We noted that reproducibility is overall higher with probabilistic tractography, but also significantly high with deterministic tractography for selected links. Our results suggest that probabilistic and deterministic tractography yield reproducible observations, but deterministic tractography reproducibility is more restricted to specific tracts.

These observations highlight the importance of taking into account the effect of methodological issues when interpreting structural connectivity data. The structural connectome measurement is subjected to the sensitivity and reproducibility of the methodology employed, and biological findings should be interpreted in this context.

Our results expand on previous observation regarding the reproducibility of DTI MRI. Magnotta et al. [21] assessed the multicenter reliability of diffusion tensor imaging, when interpreting voxel-based scalars of diffusion properties. They noted an intra-subject coefficient of variation less than 1%, while the inter-site coefficient of variation increased to ~1%-3%. These results are in accordance with the results from Vollmar et al. [22], who noted an intra-site ICC = 0.90–0.99 and inter-site ICC = 0.82–0.99 in fractional anisotropy measurements. These studies suggest that scalar measures of DTI are fairly reproducible. Our observations are also consistent with previous studies regarding reproducibility of DTI fiber tracking. Heiervang et al. [23] observed that 12 direction data are sufficient for reproducibly defining the core of large bundles but may be less sensitive to smaller pathways. They also observed that measures of MD and FA long tracts were reproducible, with inter-session coefficient of variation below 5%. The high reproducibility of scalar diffusion metrics was also reported by Besseling et al [24]. The findings from Danielian et al. [25] are also particularly relevant: by assessing specific white matter pathways, they demonstrated that same-scan fiber tracking evaluations showed good geometric alignment and reliable diffusion property measurements.

A recent study by Buchanan et al. [26] evaluating the test-retest reliability of connectome measure demonstrated that within-subject differences were smaller than between-subject differences. Our results are largely consistent with their observations, and we expand their conclusion by demonstrating the influence on reproducibility of regional link-wise anatomical localization, measures from different scanners and graph theoretical measurements of network architecture. Interestingly our results also confirm the recent observations that network geometry are reproducible across scanning sessions, as demonstrated by Duda et al. [27] and by Cheng et al. [28], whilst we further demonstrate that tractography approaches do not significantly disrupt this reproducibility.

Interestingly, in an earlier study, Vaessen et al. demonstrated a high reproducibility of small world metrics obtained from connectomes reconstructed from two different sessions in a small group of subjects (n = 6) [29].

We believe that our observations from this present study provide important additional insight into the reproducibility of structural connectivity for three new innovative aspects. First, we evaluated the reproducibility of whole connectome mapping. The concept of the brain connectome is a promising development in neuroscience since several physiological and pathological processes of the human brain are postulated to arise from architectural organization of neural networks[30–33]. Hence, the brain connectome will become an important tool in the investigation of mechanisms underlying health and disease of the nervous system. The understanding of technical limitations and anatomical location of higher reproducibility can greatly improve the interpretation of the findings from connectome-based studies. Second, we compare different tractography approaches. Probabilistic tractography constitutes a robust expansion of fiber tracking, with the possibility to resolve crossing fibers [18]. Nonetheless, it is a time consuming method. By contrasting the reproducibility of probabilistic versus deterministic tractography, we provide further information to guide the decision of one approach versus another in future studies. Third, we compared within and between scanner measures. Even though most studies using the connectome may opt to use data from one scanner, a superb utilization of the connectome is related to longitudinal studies, where, realistically, issues regarding scanner upgrades or replacements may be common. Moreover, multi-site studies are getting increasingly common to generate large datasets or to systematically study rare diseases. Our results can guide the interpretation of data obtained from single or multiple scanners.

Study limitations

The results from this study should be interpreted in the context of its limitations. First, we did not directly address the relationship between fiber curvature and reproducibility, including the way affine registration may threaten reproducibility among areas prone to distortion. Furthermore, ROIs were converted from the native T1 space into the diffusion space using a linear transformation. This “within-subject” approach reduces more significant distortions associated with individual variability of sulci and gyri positioning. However, diffusion images are subject to different types of distortions compared with T1 images and the affine registration may be insufficient to completely correct for those.

We also did not address the influence of other forms of fiber tracking, such as diffusional kurtosis tractography [34, 35] and high angular resolution diffusion imaging, including DSI [36]. We believe that the interaction between these methods and fiber curvature, fiber crossing and complex link pathways would be a highly relevant topic for future studies. Also of importance for future studies is the effect of scanner drift over time. It must be noted that the scan/re-scan period was variable across subjects, and that slight changes in image registration could occur due to hardware variability over time, especially for longer time scales. Importantly, routine maintenance of the scanners must ensure reliable registration throughout time. The maximum scan/re-scan in our study was nonetheless similar to other studies looking at the longitudinal reproducibility of tractography (e.g. [25]).

Second, our connectome data did not include cerebellar or intra-brain stem pathways. Resolution of standard DTI is ill suited to resolve the complex brainstem architecture in a sufficient way. Thus, we opted to use a single brainstem region that was fully integrated into the reconstruction process allowing fibers to start and terminate there. This will ensure that fibers like the cerebral peduncle are captured although not anatomically resolved. Cerebellar regions were not used given the same anatomical constraints and less well established atlases.

Third, the reproducibility measurement employed throughout the manuscript, i.e., the ICC, is attenuated if assumptions of normality are violated[37]. Thus, the ICC can underestimate reproducibility. This limitation may be instrumental in the context of the structural connectome, since link weights that do not conform to a normal distribution are those that are relatively unstable across many subjects (being therefore close to zero). These links are not homogeneously resolved across all subjects and the ICC identifies them as less reproducible. Nonetheless, it is important to notice that the reproducibility described here may in fact be mildly underestimated.

Fourth, we used the number of streamline to represent connectivity strength. However, DTI streamlines are mathematical constructs that may not necessarily represent connectivity strength [38]. In this context, our observations regarding the reproducibility of graph theory measures may be more relevant, since they disclose the reliability of the structural conformation rather than link wise weight.

Fifth, gray matter ROI segmentation was performed based on an anatomically defined atlas [17, 39]. Nonetheless, the division of the gray matter into ROIs is a semi-arbitrary process, which may not directly represent functional or histological boundaries. Thus, gray matter ROIs may be influenced by anatomical variability and may be inferior to connectivity based parcellation schemes that respect individual patterns of connectivity.

Conclusions

In summary, in this study we described the reproducibility of structural connectome mapping as a function of DTI tractography approach, anatomical properties of structural links and MRI scanners. Our results demonstrate that connectome mapping is a largely reproducible technique, particularly as it related to the geometry of network architecture (measured by graph theory) but special attention should be devoted to methodological and anatomical aspects associated with lower reproducibility.