1 Introduction

Multiple statistical tests are routinely applied in genetic association studies and the corresponding P-values are reported. Journals require that P-values should be adjusted for multiple testing to protect against spurious findings. Nevertheless, findings often do not replicate in subsequent studies. Various explanations have been suggested for the low replicability of findings in observational studies, including inadequate accounting for multiple testing [1, 2]. In modern genetic studies, the number of statistical tests can be very large. Such discovery studies are often performed in a manner in which some small number of the most promising results are selected for closer investigation in a replication study.

It is now appreciated that a P-value does not reflect uncertainty about validity of a hypothesis. Yet P-values do contain information that can be used to evaluate this uncertainty, and we incorporate that information into our proposed method. A solution to the dilemma which findings are false and which are genuine can be obtained via conversion of P-values to Bayesian probabilities that a finding is genuine. A simple Bayesian solution has been proposed previously, termed the False Positive Report Probability (FPRP) [3]. In this approach, tailored to genetic association P-values, a plausible effect size, for example an odds ratio and the prior probability of the null hypothesis are proposed by a researcher, and “the probability of no true association” is determined for any finding with the P-value that is smaller than a preset threshold.

It has been suggested that the FPRP approach has two main deficiencies. First, as acknowledged by its authors, FPRP is not the probability that a particular finding is false, because it is based on the tail distributions rather than on the respective densities. The FPRP approach advocated plugging in an observed P-value in place of a fixed threshold. The result can only be interpreted as “the lowest FPRP for which the finding meets a preset criterion for noteworthiness” [4]. In his critique of the FPRP, Lucke [5] wrote that “the FPRP can promote false positive results”, due to a peculiar property of the FPRP that it cannot exceed the proposed prior probability. Secondly, the usage of a single “typical” value of the odds ratio fails to acknowledge that in reality different genuine signals carry different effect sizes and a proper calculation should take into account the entire distribution of possible effect sizes. Lucke was not optimistic regarding performance of methods such as FPRP built using this simplification [5]. However, the extent of imprecision introduced by the usage of a single value remains unclear. Lastly, we note that in the FPRP approach, all variants are divided into two classes, the first class containing those that are truly associated, and the second class containing variants with the effect size that is precisely equal to zero. The second class is assumed to contain majority of the variants and corresponds to the sharp null hypothesis, H₀. While such approach is common due to its convenience, the non-associated set is likely to contain many variants with effect sizes sufficiently small to be either inconsequential to a researcher or undetectable given a particular sample size. It is more realistic to define the null hypothesis in such a way as to allow inclusion of variants with negligibly small effects.

The FPRP method is simple to apply and it has been referenced extensively since its introduction in 2004. Similarly simple methods are needed to convert a particular P-value to the “probability of no true association”, so that the result can be interpreted as the probability about this particular report. One such method has been proposed by Wakefield, called the “Bayesian false-discovery probability” (BFDP) [6]. In BFDP, the prior distribution of effect sizes, specified in terms of the log relative risk, is assumed to be normal. Strength of association is modeled via the variance of that distribution. BFDP has a sound Bayesian foundation, but the focus on the relative risk and the usage of the normal prior distribution for the effect sizes might be somewhat restrictive.

Here we introduce POFIG, “Probability that a Finding is Genuine”—a solution inspired by the idea behind the FPRP approach. Our method shares generality and simplicity of FPRP in that it utilizes P-values that may result from various association statistics. POFIG alleviates the main shortcomings identified with the FPRP approach. The usage of a single value taken to be the mean of the underlying effect size distribution does introduce a bias that can be substantial for very small P-values. However, a modification based on just three typical values (small, medium, and large) with their respective frequencies provides a much better approximation to the complete specification of the effect size distribution. Typically, most signals would carry small effect sizes, some signals would fall into the middle group around the mean of the effect size distribution, and signals of large magnitude would represent only a small proportion of the distribution. In addition, the “null bin” can also be defined to capture magnitudes of effect sizes that are close to zero (Fig 1). Instead of testing the usual null hypothesis that a variant has precisely zero effect, one can test a hypothesis whether the effect associated with a variant is sufficiently close to zero, i.e., falls into the null bin. Statistical methods are starting to appear that allow to characterize the effect size distribution in a tabulated way. Park et al. proposed a method to extract tabulated effect size values with their abundances from genome-wide association studies of complex traits [7]. Our method is designed to utilize such tables of estimated or plausible effect sizes. Correct specification of the effect size distribution becomes especially important when top hits are selected from a study with very many tests. When the distribution of effect sizes is specified correctly, posterior probabilities of association are completely unaffected by multiple testing and by the process of selection of the most significant results. In an experiment with multiple tests, one can sort results by an association statistic and take the most extreme one: the POFIG calculations do not depend on the number of tests. POFIG is similarly unaffected by the process where only those multiple testing experiments are retained which contained statistically significant results. Computed probabilities that a finding is genuine are still correct when selectively applied to experiments that contained statistically significant findings.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Approximating the effect size distribution by three values.

Discretization of the Gamma(shape = 1, scale = 1) distribution. The middle bin is centered around γ_medium = μ = scale×shape = 1 and has the width μ.

https://doi.org/10.1371/journal.pone.0124107.g001

2 Results

Our method estimates the posterior probability of the null hypothesis (H₀) that an association is spurious, or the posterior probability of the alternative hypothesis (H_A) that an association is genuine. These probabilities are estimated given the association strength summarized by a P-value (p). POFIG can be defined using the Bayes rule in a traditional way as POFIG = Pr(H_A|p) = 1 − Pr(H₀|p), (1) POFIG depends on the marginal probability density of P-values for truly associated variants, given by (2) In calculating this marginal density one specifies B plausible ranges (i.e, “bins”) of effect sizes, with the effect size value for the bin i given by γ_i, and the number of loci with that effect size (w_i). The index i = 1, …, B tracks the current bin in the summation. If the total number of loci in the genome is K, the prior probability of the null hypothesis is Pr(H₀) = 1 − ∑w_i/K.

Consistent with the FPRP and BFDP approaches, the above formulation presents the null hypothesis as sharp, i.e, as a special class for signals with precisely zero effect size. In an alternative specification of POFIG, the null hypotheses can be defined to encompass variants with effect sizes that are small enough to be considered undetectable. This formulation has a number of advantages. The sharp null hypothesis, H₀:γ = 0 is a convenient statistical concept, however it implies that the effect size distribution has a spike at a single point γ = 0, which is biologically unrealistic. Instead, it is believed that there is a very large number of variants with tiny effect sizes that influence common traits and complex diseases. These effects are of negligible size when a particular variant is considered by itself, but collectively these effects may explain a substantial proportion of heritability [8, 9]. Rather than treating null effects as a special class, we define an extra bin for the effects with the average effect size equal to γ₀, in a narrow interval near zero and the number (or the proportion) of such effects, w₀. The posterior probability of the null hypothesis is evaluated via (3) with j = 0. The posterior probabilities for the rest of the bins (i.e., j = small, medium, large) can be calculated similarly. The weights w_i in Eq (2) are proportional to Pr(H_i) and ${\sum }_{i=0}^B\mathrm{Pr}(H_i)=1$. The posterior mean for the effect size (4) The summation index in Eqs (3, 4) now starts from zero rather than from one to accommodate inclusion of the null bin that is no longer associated with the point null hypothesis (H₀), as in Eq (1). The effect size in Eqs (1–4) is defined as a parameter of the P-value density. As an example, we will use the usual one degree of freedom chi-square SNP association statistic, but the P-value density is defined in the same way for statistics that have distributions other than the chi-square (details are given in Materials and Methods). The P-value density is (5) where g_γi(·) is the one degree of freedom noncentral chi-square density function with the noncentrality γ_i and subscript 0 refers to the central chi-square density. The function $G_0^{-1}(·)$ is the inverse central chi-square cumulative distribution with one degree of freedom. Tabulated effect size values can be given in terms of the odds ratios and then related to noncentralities, assuming a logistic regression model: (6) where D is the squared log of odds ratio, N is one half of the harmonic mean of case and control sample sizes (n₁, n₂, in allele counts), N = n₁ n₂/(n₁ + n₂), and q is the pooled allele frequency. Thus, the effect of an increase in sample size is an increase in the noncentrality and an increased skewness toward zero in the distribution of P-values. ℰ can be interpreted as a standardized effect size: 2ℰ corresponds to the contribution of a SNP to the additive genetic variance of the trait. Park et al. gave a method for estimating the distribution of ℰ that utilizes data from replicated GWAS findings [7]. As noted above, the effect size distribution can be modeled in terms of parameters, such as relative risk or odds ratio, or directly in terms of the noncentralities. Park et al. used a standardized effect distribution which is the same as the distribution defined in terms of noncentralities. When sample sizes are the same for all tested variants in a study, placing the distribution directly on ℰ (equivalently, on γ) gives posterior probabilities that have the same ranks as P-values. On the other hand, specification of the distribution for D will give probabilities with ranks that differ from the ranks of P-values. In SNP association testing, the same noncentrality (γ) may result from a SNP with a large odds ratio and a small allele frequency, as well as from a SNP with a small odds ratio but a large allele frequency. Thus, usage of the standardized effect size implicitly assumes that rare variants tend to have a larger effect size compared to common variants.

Our approach utilizes a weighted sum of B tabulated effect size values (Eqs 1–3). At one extreme we may consider a single “typical” effect size value (B = 1), as in the FPRP approach. At the other extreme, the effect size distribution may be represented by a continuous function. These modifications and the relation between POFIG and FPRP are given in Materials and Methods.

Representation of the effect size distribution by a discrete set of bins offers several advantages. Among them are (1) simplicity of calculations; (2) emergence of methods for estimation of effect size distribution in human genome [7]; (3) straightforward specification by practitioners, e.g., it is easier for a researcher to come up with several plausible effect sizes and their relative abundances than to specify parameters of a continuous distribution; (4) flexibility of the approach: the binned distribution does not need to be cast in a form of a parametric statistical distribution, such as a normal distribution, and thus can reflect more closely a biologically realistic distribution.

The advantages of our approach might be offset if the number of bins (B) is too small to represent the actual effect size distribution for the purpose of estimating posterior probabilities with good precision. It is especially important that the probabilities are well-estimated for top hits of a study, i.e., for those variants that are selected based on the smallest association P-values. Such selection of top hits is known to induce bias known as the “winner’s curse” phenomenon, in which selected P-values tend to be too small and the corresponding estimated effect sizes tend to be too large. Correcting this bias is known to be difficult [10]. Fully Bayesian inference is not susceptible to the winner’s curse [11], as long as the prior distributions are specified correctly. In our case, prior for the effect size distribution needs to reflect the actual prevalence of all possible signal magnitudes in the genome. Our proposal to collapse the effect size distribution into discrete bins may induce imprecision and make the method susceptible to the winner’s curse bias. However, our results demonstrate that even a crude approximation based on three ranges for effect sizes provides good protection against bias due to selection of top strongest signals in a study.

2.1 Simulation results

Simulation experiments were designed for judging precision of our method in evaluating the proportion of true signals among smallest P-values in multiple-testing experiments. We assumed that the effect size is represented by the odds ratios (OR) and that the squared logarithm of OR has an L-shaped distribution. For any true effect, its effect size was sampled from an L-shaped distribution and the corresponding P-value was obtained based on that effect size value. We also allowed for a proportion Pr(H₀) of signals to be completely spurious (false), i.e., to have the OR of exactly one. In a single multiple-testing simulation experiment with K P-values, each P-value carried a flag indicating whether it originated from a true (T) or a false (F) signal. Once K P-values were sorted, we examined the fraction of flags that were equal to T in a set of K smallest P-values. This gives an empirical (i.e., true) proportion of true signals among K smallest P-values for a single simulation experiment. Our POFIG method provides an estimate of this proportion: it is the average of POFIG values for these K smallest P-values. Next, we averaged the true (empirical) and the POFIG-estimated proportions over a large number of simulation experiments. Given this simulation setup, correctness of our method can be judged by closeness of the estimated values to the true values obtained using the knowledge of which P-values are actually generated from true signals. This gives an overview of the simulation setup; more detailed description is given in Material and Methods.

In the simulation experiments, we defined the effect size D in terms of the squared log of odds ratio, and further allowed for the effect size to have different values for different genuine signals, assuming a continuous (gamma) distribution. We assumed the proportion of null signals is known and considered three degrees of the knowledge regarding the effect size distribution:

1. Complete knowledge, i.e., precise fraction of signals with the odds ratio greater than any given magnitude. Note that the knowledge of the distribution does not imply knowledge of the effect size that gave rise to any given P-value. Only summary information given by a probability distribution of various odds ratios is assumed. We refer to this as the exact or the integration method.
2. Knowledge about the abundance of three classes of odds ratios, low, average and high. We refer to this as the discretization or the tabulated values method. In this paper, we approximate the effect size distribution by three bins, capturing the bulk of the effect size distribution and the fourth “null” bin containing close-to-zero effect sizes, as illustrated in Fig 1. Posterior probabilities in these simulations were evaluated by Eq (3), assuming that the null bin includes odds ratios of 1 to 1.001. We also evaluated these probabilities by Eq (1) that assumes the sharp null hypothesis. Results obtained by these two methods were identical (up to the reported precision). Note that the simulations also modeled the null hypothesis as sharp, i.e., the expected proportion Pr(H₀) of all signals in a simulation experiment had exactly zero effect size.
3. Knowledge about only the mean of the distribution. We refer to this as the single-value or the distribution mean method.

In our experiments we focused on estimating POFIG probabilities for the smallest P-values. Selection of the smallest P-values may cause bias in estimation. An extreme case of selection is when the single smallest P-value is selected from a multiple testing experiment, which we call the minP. Additionally, we may envision a process whereby the entire experiment is discarded by a researcher unless the minP is smaller than the multiple testing adjusted significance threshold. Thus, we considered two types of P-value selection (1) selecting the smallest P-values of every experiment and (2) selecting the smallest P-values of only those experiments where statistically significant findings were observed. First, we considered just the single smallest P-value, the minP, to determine whether our method can correctly recover the probability that the minP represents a spurious signal in both of these selection scenarios. Next, we extended the experiments by considering a larger set of the smallest P-values. In these experiments, we calculated the proportion of spurious signals among selected smallest P-values and evaluated how well our method can recover that proportion. We used either the Bonferroni at the level of 0.05/K or Benjamini and Hochberg’s FDR criterion [12] at the level of 10% as the rule for discarding sets of the smallest P-values.

2.1.1 Posterior probability of H₀ for a signal with the smallest P-value with or without a multiple testing correction.

Table 1 summarizes the posterior probabilities that the smallest P-value is a spurious signal in a study with K tests. As expected, the posterior probabilities are unaffected by the “winner’s curse” phenomenon: the true (first number in each cell) and the estimated probabilities are nearly identical for the “exact” method (the second entry in each cell) which assumes precise knowledge of the effect size distribution (the number of simulations was 50,000 for Table 1 and 10,000 for other tables). The three-value approximation works well but becomes conservatively biased with 1 × 10⁶ tests; the bias increases for Bonferroni-selected scans (the “0.05/K” columns) compared to experiments without selection and for experiments with the relatively higher proportion of non-zero signals. The single value approximation (last number in each cell) is surprisingly good for small to moderate number of tests. At 1 × 10⁶ tests, the bias for the smallest P-value becomes excessive (Table 1), but the bias is greatly reduced as P-values with higher ranks are considered (Table 2). Results in Table 1 suggests that the Bonferroni adjustment for multiplicity (α = 0.05/K columns) fails to substantially decrease the probability that the smallest P-value is a spurious finding: although we observe a decrease in the values of posterior probability, the reduction is not as drastic as may have been anticipated, considering the stringency of the Bonferroni threshold, 0.05/K.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Probability that the minimum P-value is a false finding.

Table entries W/X/Y/Z give the true probability (W), followed by the estimated probabilities using the integration method (X); the discretization method (Y); the distribution mean method (Z). Sample sizes (n₁, n₂: number of cases, controls in allele counts): n₁ + n₂ = 800 for K = 10 to 10,000; n₁ + n₂ = 4000 for K = 10⁶.

https://doi.org/10.1371/journal.pone.0124107.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The average of posterior probabilities among the 100 smallest P-values, i.e., the false discovery rate (FDR) among the 100 smallest P-values.

Table entries W/X/Y/Z give the true averaged probability (W), followed by the estimated averaged probabilities using the integration method (X); the discretization method (Y); the distribution mean method (Z). Sample sizes (n₁, n₂: number of cases, controls in allele counts): n₁ + n₂ = 800 for K = 10 to 10,000; n₁ + n₂ = 14000 for K = 10⁶.

https://doi.org/10.1371/journal.pone.0124107.t002

2.1.2 Proportion of false signals among the 5 smallest P-values with and without an application of the FDR criterion.

Similar conclusions can be drawn from Table 3 where Benjamini and Hochberg’s FDR criterion [12] (B&H’s FDR) was applied at the level of 10%. In experiments with the application of B&H’s FDR, the five smallest P-values were discarded unless all five satisfied B&H’s FDR criterion. Proportions of spurious signals are very similar for the results with and without the application of B&H’s FDR criterion.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. The average of posterior probabilities among the 5 smallest P-values, i.e., the false discovery rate (FDR) among the 5 smallest P-values.

Table entries W/X/Y/Z give the true probability (W), followed by the estimated probabilities using the integration method (X); the discretization method (Y); the distribution mean method (Z). Sample sizes (n₁, n₂: number of cases, controls in allele counts): n₁ + n₂ = 800.

https://doi.org/10.1371/journal.pone.0124107.t003

2.1.3 Proportion of false signals among the 100 smallest P-values.

Table 2 summarizes the proportion of false signals among the set of 100 smallest P-values. That proportion, or the false discovery rate (FDR), is simply the average of posterior probabilities among the 100 smallest P-values. In this case, even the single-value approximation is reasonably close to the true value: as the P-value rank moves away from minP, the bias decreases.

Interestingly, the three tables of simulation results demonstrate the phenomenon that as the number of tests becomes larger, the proportion of spurious signals among a given number of top hits decreases. This can be inferred from inspecting results of experiments with the numbers of tests K increasing from 10 to 10,000, where the sample size and the prior Pr(H₀) were held the same for different values of K.

2.1.4 POFIGs for the smallest P-value from a scan, its replication P-value, and for the combined scan and replication P-value.

In Fig 2, the first graph is a histogram of estimated posterior probabilities associated with the smallest P-value for a simulated discovery scan with 10,000 tests (simulation details are described in the Methods section “Scan, replication and combined probabilities”). Smaller probabilities that the finding is false are associated with the over-presence of genuine signals, indicated by red color, while large probabilities are associated with false signals, indicated by blue color. The mean value of the first histogram does estimate the true proportion of false signals in an unbiased way. However, there is a large area of overlap (purple color), that is, there is no good separation of “red” and “blue” signals. Once all of these signals are evaluated in a four times larger replication sample (middle histogram), we observe an appropriately U-shaped distribution. Now most low and high probabilities correctly capture the genuine and the false signals, respectively, although there is a sizable purple bar at the right indicating mis-classification of some genuine signals as false. We observe that a sufficiently large sample size assures a good correspondence between the probability and the actual status (genuine vs. false) of a signal. The U-shape becomes more prominent in the last histogram that is based on probabilities for the combined scan and replication P-values, due to an increased sample size. These figures clearly demonstrate utility of POFIG for discrimination of genuine and spurious signals. A value of Pr(H₀|P-value) that is close to one reflects high likelihood that the finding is spurious. In contrast to that, large association P-value does not give support toward the hypothesis of no association.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Distribution of estimated posterior probabilities that a P-value is a false finding.

Color refers to whether the actual signal is genuine (red) or false (blue). Purple color denotes the overlap of genuine and false signals. First graph: Estimated probability that the smallest P-value in a scan (minP) is a false finding. Second graph: Estimated probability that the signal from the scan with the smallest P-value is a false finding when evaluated in a replication study. Third graph: Estimated probability that the finding is false based on the combined P-value (the smallest scan P-value combined with the replication P-value).

https://doi.org/10.1371/journal.pone.0124107.g002

2.2 POFIG application to Crohn’s disease P-values

For the analysis of real data, we used SNP association P-values for Crohn’s disease from Barrett et al.[13] We considered newly identified loci from Table 2 of their work, selecting those SNPs for which genes of interest were listed. Utilizing their scan and replication P-values, we applied the POFIG method to provide scan and replication posterior probabilities of no association for selected SNPs. We used the prior for the H₀ equal to 1-142/12877, where 142 is the number of susceptibility loci estimated by Park et al.[7] and 12877 is the total number of loci, estimated using their definition of a locus based on the extent of linkage disequilibrium. We assumed the effect size distribution to be a gamma with the shape parameter 1 and the scale parameter was chosen to give the distribution mean computed from Supplementary Table 3 of Park et al. We also conducted a sensitivity analysis, utilizing different priors and effect size distributions including tabulated effect sizes given in Park et al.[7].

It is common to combine P-values or association statistics between studies, for example, as presented in Barrett et al.’s Table 2. It should be pointed out that although this type of meta-analysis, based on summary statistics can be nearly as efficient as analysis based on pooling raw data [14], possible non-independence of samples and effect size heterogeneity between studies require careful consideration during meta-analysis of GWAS [15].

The scan and replication probabilities can also be combined. These combined probabilities can be obtained by using the posterior probability from the scan as the prior Pr(H₀) for the replication. This is mathematically equivalent to replacement of $f_{\bar{\gamma }}(p)$ in Eq (1) by the product $f_{\bar{\gamma }}(p_1)\times f_{\bar{\gamma }}(p_2)$ where p₁ and p₂ are the scan and the replication P-values, while using the initial prior probability, Pr(H₀), for the scan. Using this approach, combined posteriors probabilities, labeled as “Combined Pr(H₀|P-value)”, were obtained by combining the scan and the replication posterior probabilities that the finding is spurious for the newly discovered SNP associations. The results based on the combined probabilities provide substantially higher chances that the newly identified loci are genuine findings. Barrett et al. reported combined P-values for the scan and the replication statistics. Their values were based on a combination of two-sided statistics. Therefore, in our analyses, both the scan- and the replication posterior probabilities were obtained using two-sided P-values. This approach does not take into account direction of the association (i.e., effect sign). In Materials and Methods we describe a simple way that takes into consideration the correspondence of effect directions between studies (section “Scan, replication and combined probabilities”).

Fig 3 and Table 4 provide combined posterior probabilities for the newly identified loci reported in Barrett et al.[13]. Barrett et al.’s scan, replication, and combined P-values are given in the table for comparison. Figs 4 and 5 and Table 5 illustrate the effect of the prior probability and the assumed effect size distribution on the posterior estimates of Pr(H₀|P-value). Neither of the choices considered in the Table and Figures substantially affected the posterior probability that the association is false. In particular, posterior probabilities using the tabulated distribution are similar to the ones obtained assuming a continuous, L-shaped gamma distribution.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Minus log of posterior probabilities that the finding is spurious.

Minus 1-POFIG for novel SNP associations with Crohn’s disease in Barrett et al. reported on the logarithmic scale.

https://doi.org/10.1371/journal.pone.0124107.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. P-values reported for the newly identified loci for Crohn’s disease in Barrett et al. and the corresponding posterior probabilities of the null hypothesis.

Rows highlighted in bold correspond to the maximum and minimum Scan P-values.

https://doi.org/10.1371/journal.pone.0124107.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Probability that a finding is false (1-POFIG).

The probability that the most significant newly identified loci reported in Barrett et al. is false. The prior probabilities are based on the estimates for the total number of susceptibility SNPs reported in Park et al. The shape parameter 0.634 was chosen to yield the median value of the noncentrality distribution to be twice as small as that of the exponential distribution (Shape = 1) for the same value of the scale.

https://doi.org/10.1371/journal.pone.0124107.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Probability that a finding is false (1-POFIG).

The probability that the least significant newly identified loci reported in Barrett et al. is false. The prior probabilities are based on the estimates for the total number of susceptibility SNPs reported in Park et al. The shape parameter 0.634 was chosen to yield the median value of the noncentrality distribution to be twice as small as that of the exponential distribution (Shape = 1) for the same value of the scale.

https://doi.org/10.1371/journal.pone.0124107.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Posterior Pr(H₀|P-value) for two newly identified loci based on the results reported in Barrett et al. [13].

The three prior probabilities are based on the three estimates for the total number of susceptibility SNPs taken from Park et al’s. supplementary Table 5. [7] The “Tabulated” effect size distribution is based on Park et al’s supplementary Table 3; “scale” in Gamma(1 or 0.634, scale) is chosen to have the distribution mean equal to that of the tabulated distribution.

https://doi.org/10.1371/journal.pone.0124107.t005

Fig 6 illustrates the comparison of the POFIG posterior probabilities of no true associations to the FPRP values. The FPRP estimates tend to be smaller, as they refer to the whole tail of P-values as small as the one observed. The posterior probabilities of H₀ are monotone in the ranking of P-values as the distribution of the noncentralities for this analysis was specified directly (Eq 14), based on the results reported in Park et al.[7]

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Posterior probabilities of the null hypothesis by POFIG and FPRP methods.

The posterior probabilities are calculated for the newly identified Crohn’s disease associations from Barrett et al.’s study [13]. Unmarked genes in the lower left corner: IL12B, LRRK2-MUC19, C11orf30, CDKAL1, CCR6, ITLN1—for red and blue; IL12B, LRRK2-MUC19, C11orf30, CDKAL1, CCR6, ITLN1, JAK2—for green (FPRP).

https://doi.org/10.1371/journal.pone.0124107.g006

3 Discussion

Modern molecular technologies continue to reveal information about genetic architecture of human diseases. It is now possible to utilize available knowledge to make informed decisions regarding parameters needed for the calculation of probability that a P-value represents a genuine association, namely the overall proportion of susceptibility loci and the distribution of their effect sizes. The “False Positive Report Probability” (FPRP) approach [3] utilizing these parameters was proposed previously, however the FPRP was not designed as a method for assessing the probability that a finding (i.e., a “report”) in question is false. Here we introduce a similarly inspired approach, the method for assessing the “PrObability that a FInding is Genuine” (POFIG). Our method eliminates the shortcomings of FPRP, but shares its computational simplicity.

As in the FPRP approach, one can simply assume a “typical” effect size (taken to be the mean of the underlying true distribution). This results in a method that is just as simple computationally as the FPRP. Its bias in the estimation of the posterior probability increases with the number of tests, but decreases with the P-value magnitude or rank. That is, the bias is the largest for the most significant finding in a study. When the mean value is used, estimate of the posterior probability that the finding is spurious appears to be conservative (i.e., it is larger than the true value) and is tolerably small for experiments with as many as 10,000 tests. Truly massive multiple testing experiments such as whole genome sequencing and genome-wide association studies may require a more precise specification of the effect size distribution. Ideally, the whole probability distribution needs to be specified and the population genetics theory suggests that a gamma distribution can be successfully utilized to model the expected L-shaped distribution of effect sizes [16]. A far more convenient way is to assume several tabulated effect size values with the relative abundance of each. Methods and data from which these values can be readily extracted are starting to emerge [7]. We find that this computationally simple approach can greatly reduce the bias and usage of only three bins with low, moderate and large effect size values already provides sufficient precision.

False signals can be handled in two different ways within our approach. Traditionally, false signals are assumed to have zero effect size; this requires specification of the prior probability of the null hypothesis, Pr(H₀). Alternatively, an extra “null bin” is introduced with its respective proportion for signals which effect sizes are within a narrow interval close to zero. The second specification acknowledges an understanding that there are many signals whose effect sizes are tiny but non-zero, avoiding an unrealistic assumption that there is a sizable fraction of signals with exactly zero effect size.

There are various desirable properties of POFIG. When the same genetic variant (hypothesis) is tested in several studies, one can assess the combined posterior probability of false association by sequentially using the 1-POFIG probability from one study as the prior Pr(H₀) for the next. This can be entered in any order and is equivalent to using the initial prior probability, Pr(H₀), while replacing f_γ(p) in Eq (1) by the product f_γ(p₁) × f_γ(p₂) × …, where p_i is the i-th study P-value. Alternatively, one can combine study-specific P-values first and convert the result to the posterior probability of a hypothesis. This approach gives a simple way to account for correspondence in the effect direction between studies (Eqs 20, 21). We may contrast this approach with the practice of combining scan and replication statistics or P-values. Barrett et al. reported scan, replication and combined P-values in their Table 2. Their combined values are obtained by pooling Z-statistics, and very similar values can be obtained by combining their scan and replication P-values by the inverse normal method [17]. These combined P-values are biased under the hypothesis of no association by the fact that the scan P-values were selected among the set of the smallest P-values. To preserve the type-I error rate in a conservative way, these combined P-values would need to be adjusted by the number of tests in the scan. Ideally, classical P-value combination procedures would need to take into account the number of tests as well as ranks of P-values in both the scan and the replication studies [18].

Following our approach, we combined the scan and the replication posterior probabilities for novel SNP associations with Crohn’s disease [13] using our approach (Fig 3). Not only the two probabilities can be combined in a straightforward way, but they can also be averaged across selected SNPs. This average gives the estimated proportion of spurious signals among the SNPs. In the analysis of Barrett’s data, we computed probabilities that a finding is spurious (i.e., false). The average of these probabilities gives the estimated proportion of false signals, or the false discovery rate estimate (FDR), as reported in the last row of Table 4.

Selection issues aside, the scan and combined P-values in Table 4 should be viewed with caution, because the smallest GWAS P-values may themselves be imprecise for various reasons, including residual confounding, genotyping errors, and violations of statistical assumptions in computing P-values from very large values of association statistics. We also assume that the bulk of P-value distribution is appropriately flattened by methods that adjust for systematic biases caused by population stratification or other sources of confounding [19–21]. Our posterior probabilities are extracted from P-values, thus they are similarly affected by these issues.

Our approach compares favorably to classical estimation procedures where effect size estimates for top hits of a study tend to be over-estimated and the corresponding P-values tend to be too small. This bias, known as the winner’s curse phenomenon, has been a formidable challenge. Bowden and Dudbridge concluded that in the classical (frequentist) statistical framework, no unbiased correction is possible without replication data [10]. Combination of selected smallest P-values results in complicated distributions [22–24] and methods for combining results from studies, some of which involve selection of the best results, are lacking. The main appeal of our method is that it is unaffected by selection of either smallest or statistically significant P-values, as we demonstrated by simulation experiments. Estimated posterior probabilities and their averages among selected P-values (FDR estimates) are unbiased. These estimates of posterior probabilities and especially of their averages among the smallest P-values were shown to be robust in the presence of high linkage disequilibrium [25]. Admittedly, high precision in estimation of true probabilities comes at a price of correct specification of the effect size distribution and the overall proportion of susceptibility variants. This underlines importance of emerging statistical research on estimation of these parameters from large scale genetic data.

Table 1 shows that in experiments with and without a multiplicity correction, the smallest P-value is increasingly more likely to represent a genuine signal as the number of tests increases from 10 to 10,000. In experiments with the multiplicity correction, only those experiments were retained for analysis where one or more results were significant. Although the multiplicity adjustment itself does not change the order of P-values, probability for the signal with the smallest P-value to be genuine changes when computed among those retained experiments with significant findings. There is another interpretation of these results. In the columns with the multiplicity correction, simulations can be thought of as modeling a process that leads to the publication bias phenomenon: experiments without significant findings are put in a file drawer, while those with significances are sent to a journal. The fact that the calculated posterior probability correctly estimates the true proportion of spurious results indicates that our method is unaffected by the publication bias phenomenon. The calculation of posterior probabilities is not informed in any way by the significance threshold used or by how many tests were performed.

It is understood within the research community that P-values, even after a multiplicity adjustment still do not reflect chances that the finding is spurious. Nevertheless, multiplicity adjustments are thought to be highly efficient in filtering out spurious results, similarly to how the distillation process turns the wash into vodka. It is evident from Table 1 after comparing probabilities for columns with and without the correction that the multiplicity adjustment even as strict as the Bonferroni does not necessarily serve as a good filter of spurious signals. Application of Benjamini and Hochberg’s FDR criterion (Table 3) changes the true proportion of spurious signals only slightly. When half of the signals are genuine, this proportion drops from 15% to 12%, but in these simulations, only about 1 in every 18 experiments had all five smallest P-values significant by this criterion. These simulations were deliberately designed to be under-powered to detect any single genuine signal. In studies with low power, genuine signals do not come up at the top with high certainty. In these situations, multiple-testing approaches should supposedly shield us from accepting too many spurious signals as genuine. As we see, this is not the case. Moreover, there appears to be increased enrichment of top hits by genuine signals as the number of tests becomes larger.

In summary, P-values, despite their deficiencies, do contain information that can be used for judging validity of findings and we propose POFIG for converting P-values to valid probabilities that a finding is genuine. As we demonstrate, this simple method avoids numerous complications that arise in making decisions based on P-values. POFIG requires an informed input from researchers regarding a likely proportion of genuine signals in their multiple testing experiments and some characterization of the effect size distribution. The later can be replaced by a good guess for the mean of that distribution unless the P-values are extremely small, in which case validity of P-values themselves is likely in doubt.

Software

Microsoft Excel and R programs to convert P-values to posterior probabilities that a finding is genuine are available at https://sites.google.com/site/pofigscript/ or by a request to D.V.Z. The programs assume that P-values result from one degree of freedom chi-square tests, but can be easily modified to accommodate other tests statistics. We welcome requests for help with modifications.

4 Materials and Methods

Suppose our test statistic is Z, with the cumulative distribution function (CDF) under the null hypothesis denoted by G₀(·). Under the alternative hypothesis, H_A, we denote its CDF by G_γ(·). The parameter γ captures deviation from the null hypothesis. P-value is obtained in the usual manner, as P = 1 − G₀(Z). By rewriting this and applying the inverse of the CDF, $G_0^{-1}(·)$, we recover the statistic Z: (7) Under H_A, the P-value has the CDF 1 − G_γ(Z). We can apply 1 − G_γ(·) to the both sides of Eq (7) to write this distribution in terms of the usual P-value computed under H₀. Thus, under H_A, the P-value CDF, which we denote by F_γ(p) is (8) By differentiating this CDF, we obtain the corresponding probability density function, PDF: (9) Here, g₀(·) and g_γ(·) are the PDFs that correspond to the CDFs G₀(·) and G_γ(·).

P-value distribution as defined in this section arises from a broad range of statistical tests. For a normally distributed test statistic, γ is the normal mean, shifted away from zero. For chi-squared F, and t statistics, γ is the noncentrality parameter of the corresponding distribution. In all these cases, zero value of γ corresponds to H₀. Thus, the P-value distribution is quite general with regard to the underlying statistical test. Its applicability is not limited to models with a single predictor; e.g., in regression models with adjustment for covariates, distribution of P-value for a predictor of interest can be modeled in the same way. The proposed approach applies to continuous test statistics with a distribution where the degree of deviation from H₀ is captured by a value of a single parameter (γ). Because this approach requires an explicit specification of the distribution, it cannot be readily applied to P-values that result from permutation-based tests and other tests that are based on resampling. It is at present unclear whether our approach would provide sufficient precision when applied simply as a convenient statistical device to approximate and model an unknown (e.g., resampling) distribution. We note that a beta distribution has been used successfully to mimic P-value distributions in several studies concerned with problems related to those considered here [26–29]. The choice of a beta distribution in these studies was due to flexibility of its shape rather than to any known relation to the actual distribution of P-values. Good performance of approximations based on a beta distribution is encouraging, given that the P-values of common test statistics do not in fact follow that distribution. Thus, the issue of applicability of our approach to P-values that do not necessarily follow an assumed parametric distribution is worthy of future investigation.

To make the exposition more focused, we primarily discuss the binary outcome, a chi-square statistic, and the effect size defined via odds ratios or directly as a value of the noncentrality, but there is no restriction for the outcome to be binary. Lee and Wray provide useful expressions that present the noncentrality parameter in terms of the proportion of variability in the outcome due to a genetic predictor [30]. These expressions cover a variety of designs with different sampling schemes and both continuous and binary outcomes. Yang et al. give formulas to relate noncentralities of population-based quantitative trait and case-control designs [31].