Input Parameters

The “data file directory” contains the readouts from each assay plate, with each plate data point saved in a separate file (accepted file formats are tsv, tab-delimited txt for all operating systems; and additionally, csv, xls and xlsx for Windows users) (Figure 2). Instead of entering the well coordinates and the intensity values as a list, data must be provided in a matrix form in 16-by-24 or 8-by-12 well format for 384- or 96-well plate screens, respectively. The well coordinates are automatically captured, and the well IDs are assigned accordingly. Number of header lines, which should be identical for each data file, must be specified in the “# of header lines in each data file” field on the GUI window. The processed data content is summarized in the “Raw_data_compiled.tsv” file output. Since GUItars is capable of running analysis for screens with or without replicates, the user must define the screening method of interest by selecting either the “single copy” or “replicates” radio button. If there are replicate assay plates with the same RNAi content, the “replicates” option should be selected, and the number of replicates must be specified. An exception to this rule is if the user prefers to evaluate each assay plate independently, keeping in mind that a different SSMD calculation will be performed. For screens with replicates (interplate), the number of assay plate replicates corresponding to each RNAi source plate has to be equal; otherwise, each condition should be analyzed separately.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. User input files required by GUItars.

Three separate input files are required by GUItars: A “data file directory” containing individual files for each plate, an “annotation file” with first two columns containing RNAi source plate ID and assay plate well ID with a single header line, and a “plate ID file” with a single header line. An “annotation file” and a “plate ID” file are mandatory only if the “hit mapping to the RNAi annotation file” option is checked.

https://doi.org/10.1371/journal.pone.0049386.g002

The “hit selection method” drop-down menu contains a list of available analysis methods with GUItars. For user convenience, the list contents and the associated parameter options are interactively updated and become visible for further analysis specifications. For screens without replicates, the list includes percent activity, z-score, robust z-score, SSMD, and robust SSMD options. For screens with replicates (3 or more), the user can choose between t-test and SSMD. The “hit selection cut-off” field is provided for user-defined threshold selection, and the value to be entered should be a reasonable cut-off value for the analysis method that will be used. To guide the user through the SSMD cutoff selection, predefined cutoff values are provided in a drop-down menu based on criteria developed by Zhang [11]. With methods other than percent activity and t-test, the user-defined “hit selection cut-off” is applied to automatically identify hits that either increase or decrease assay signals, corresponding with assay output readings above the (+) cutoff and below the (−) cutoff, respectively. Another drop-down list is also provided with options to perform the analysis either on raw or log-transformed (log₂ or log₁₀) data.

“Hit mapping to the RNAi annotations” is an optional feature that allows the users to choose whether or not matching the assay plate (hit) wells to the RNAi annotations is desired. The following fields will be enabled or disabled based on the user's selection: “plate ID file” and “RNAi annotation file”. The “plate ID file” contains the assay and RNAi source plate ID information in one of the accepted file formats as mentioned above (Figure 2). For accurate mapping purposes, the RNAi source plate IDs must match the plate IDs in the first column of the “RNAi annotation file”. If the hit mapping option is selected, the “plate ID file” will be used as the master guide for data import. The file names within the data file directory must contain the unique assay plate IDs defined in the plate ID file. If the plate ID field is left empty for either the assay or RNAi source plate, it will be replaced with “no plateID” notation, and the rest of the analysis and mapping will be affected. In screens with replicates, hit mapping is required, and the assay plate IDs for the replicate plates must be entered consecutively in the “plate ID file”.

The “RNAi annotation file” lists the RNAi source plate contents/annotations containing a header line in the first line (Figure 2). It is mandatory that the first and second columns of the file contain RNAi source plate IDs and the corresponding assay well IDs, respectively. The rest of the columns can be as many as desired comprising any relevant gene information (e.g., gene ID, accession number).

Output Parameters

In addition to an Excel file generated with multiple tabs containing comprehensive analysis results (Windows only), output data are also written into individual tab-delimited files for user convenience (all operating systems). All the graphical outputs are stored as JPEG images and written in HTML format, which can be viewed on any web browser. The images are also saved in MATLAB figure format (.fig) which can be printed or saved in higher resolution in various formats such as tif, png, eps, and pdf, as needed. A separate GUI tool to open the MATLAB figures is included in the software package available at the project home page (http://sourceforge.net/projects/guitars/) for users without licensed MATLAB software.

A desired directory and a folder name (maximum 25 characters) to save all the output files can be specified in the “output directory” and “output folder name” fields, respectively. The output folder name will be further customized automatically with date and time information to avoid overwriting existing files.

Plate Configuration

For ease of use, well configurations are defined on an interactive plate map provided in the GUI window. In this section, the plate format and the well coordinates should be assigned using the corresponding radio buttons and tabs above the plate configuration panel. The screening plate format can be either in 96-well or 384-well layout, which must match the data format in the “data file”. Based on the user's selection, the plate configuration panel is dynamically updated to the selected plate format. If the “plate quality metrics calculations” option is checked, negative controls and at least one of the positive control well positions must be specified in the plate map.

The user can specify the well positions on the plate map by checking the boxes corresponding to each well and can navigate through different well types using the colored tabs. GUItars is capable of handling three sets of positive controls in plate QC calculations. To avoid conflicts arising from selecting the same well in more than one tab, the selected wells in the active tab will automatically be disabled in the inactive tabs, and the user will be prompted with a warning message when attempting to check a disabled well position, except in certain circumstances (e.g., a negative control well can also be defined as a negative reference well). For ease of use, the “Check All” and “Uncheck All” buttons can be used to perform these actions simultaneously on all wells within the active tab.

As a side note, for a screen in which a high hit rate is not necessarily expected (i.e., a screen not using a focused library), all wells containing the RNAi samples should be designated as negative references. On the other hand, negative controls can be used as negative reference if a focused library or a confirmatory screen is being used as described by Zhang [13]. In GUItars, negative reference wells are used as a primary data source in scoring calculations.

Wells that do not contain any controls or RNAi samples should be designated as blank, so they will then be excluded from the calculations for all plates. If only a subset of the plates contain blank wells, or if certain outliers are desired to be manually excluded from the analysis at plate level, then the data points corresponding to those wells should be replaced with the “NaN” notation in the individual data files. GUItars also includes an algorithm to automatically knock out the outlying data points from the control wells based on user demand. For that, the user should check the “automatically knockout outlier control points” option and fill out the corresponding enabled fields.

The current analysis session can be saved and retrieved via the “Save Protocol” and “Open Existing Protocol” buttons, allowing the users to reuse or share their analysis setup with others. Once all the required fields are completed in the GUI window, analysis can be started by clicking the “Start Analysis” button, and the status of the process can be monitored via the status bar. When the analysis is started, preprocessing of the user input files is performed, and the data sets are checked for completeness. If any unexpected data formatting or analysis issues are encountered, the user is informed by a pop-up warning message, and the program is aborted. GUItars is developed with a robust error capturing mechanism against operator errors with 20 various warning messages to pinpoint the problem and provide an easy-to-use tool.

Plate QC

As a common practice in RNAi screens, logarithmic transformation of the raw data is often performed to achieve a symmetrically distributed data set by normalizing highly skewed distributions [16], [20]. If the user chooses to log-transform the raw data, data distribution histograms of each plate are generated for raw data as well as the transformed data, as seen in Figure 4A. With the aid of histograms, one can decide whether the applied transformation method has met the data distribution expectations and whether the hits identified from any of the plates should be evaluated with skepticism.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Graphical outputs demonstrated on a 12-plate siRNA screen analyzed with the robust SSMD method with GUItars.

(A) Raw data (left) and log₂-transformed data (right) histograms of each plate showing the original data distribution and effect of data transformation (one representative plate is shown). (B) Original scale (left) and rescaled (right) heat maps of each plate helping to capture systematic errors (one representative plate is shown). (C) Column-wise plate-series plot. (D) Screen-wise line plot for average control readings showing a clear separation between negative control and positive controls that is consistent throughout the screen. (E) Screen-wise SSMD score scatter plots with cutoff lines at 1.28 and −1.28 for signal-increasing and signal-decreasing hits, respectively. (F) Hit distribution heat maps for signal-increasing (top) and signal-decreasing (bottom) hits. (G) Screen-wise hit counts for signal-increasing (top) and signal-decreasing (bottom) hits.

https://doi.org/10.1371/journal.pone.0049386.g004

Plate heat maps are provided to help with the inspection of systematic errors, which may commonly include patterns introduced by liquid handling instruments or edge effects associated with incubation quality (Figure 4B). However, in assays with a high signal window, one can misleadingly conclude an inexistence of systematic errors because the color scaling is biased. Therefore, a second set of heat maps is provided with a color scaling ranging between minimum(mean of negative reference)−2*minimum(standard deviation (SD) of negative reference) and maximum(mean of negative reference)+2*maximum(SD of negative reference). As seen in Figure 4B, it is easier to observe systematic patterns in the heat map on the right than in the one on the left, although there are no evident errors observed in the screen shown.

While a heat map is a good tool for visually identifying systematic errors at the individual plate level, a plate-series plot is also an expedient way to review the overall performance of the screen at a glance. With GUItars, plate-series plots are generated in row-wise and column-wise formats as so that the row and column effects can be captured easily. A representative column-wise plot is shown in Figure 4C, and it is confirmed that there are no major systematic errors.

With GUItars, calculations of the comprehensive plate QC parameters are performed based on the user's positive control, negative control, and negative reference well designations (after data transformation, if selected) when the “plate quality metrics” option is checked. Although RNAi screens share certain similarities with small molecule screens, there are many aspects in which they differ. Notably, the Z′-factor [21], a measure of the quality of a screen, is considered acceptable for small molecule screens when it is greater than 0.5. However, for RNAi screens, Z′ is usually less than 0.5. As such, using the Z′-factor, RNAi screens are usually less robust than small molecule screens. As proposed by Zhang [10], SSMD can be a better alternative plate QC metric than the Z′-factor. Zhang has shown that, as a quality measurement in high-throughput assays, SSMD is based on a firm statistical theory and clear probabilistic meaning, as opposed to the Z′-factor, which lacks a statistical basis and entails a relatively arbitrarily chosen pass/fail cutoff. Moreover, Zhang has illustrated that quality assessment by the Z′-factor may be misleading for screens with moderate or strong positive controls, while the SSMD-based QC metric can quantify plate quality for screens with positive controls with various activities [22]. Although the Z′-factor is applicable to assays with single readout, another version of the formula which can incorporate multiple readouts via linear projection of the data is developed by Kümmel et al. [23] for applications such as high content screening. However, it was not integrated into GUItars' plate quality calculations since multivariate applications are beyond the scope of this study. In GUItars, SSMD (method-of-moment estimate), as a primary plate QC metric, is calculated in addition to the Z′-factor and signal window for each positive control independently in a plate-wise manner. A verbose quality metrics summary is written to an output file, which also includes mean, SD, and coefficient of variation values of positive control, negative control, and negative reference wells (Figure 5). SSMD, Z′-factor, and mean values of controls (Figure 4D) are also graphed on separate line plots for easy plate-to-plate comparison.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. Excel readable output file.

Individual tab-delimited output files as well as a comprehensive Excel file are generated with the following information: Plate QC calculations before and after control outlier knockout, scores for all wells classified by well type, scores for hit wells classified by hit type (i.e., signal-increasing or signal-decreasing), and annotated hit list (optional) with corresponding scores.

https://doi.org/10.1371/journal.pone.0049386.g005

The outlier knockout is then performed if it is selected by the user in the analysis setup. The knockout algorithm is designed to disregard the wells whose values are greater or less than the mean ± x SD of all the wells in the particular control set, starting from the most outlier well, where x is the user-defined outlier threshold. The knockout process ends once the maximum number of points to knock out is reached or when there are no points left satisfying the above criteria, whichever occurs first. In the screen shown, we chose to knock out 30% of the points with values more extreme than ±2 SD. All the plate quality metrics and visualizations are regenerated after the control outliers are knocked out so that the user can see the comparable results of the process.

Scoring and Hit Selection

GUItars is primarily designed to enhance and facilitate the application of the SSMD-based scoring and hit selection method on high-throughput RNAi screens. In addition to SSMD and robust SSMD, GUItars can apply other commonly used methods, including normalized percent activity, z-score, and robust z-score for screens without replicates and the paired t-test for screens with replicates, as summarized by Zhang [24]. Hence, the user has the flexibility to choose among various scoring methods from the user interface. The other methods mentioned above, which are incorporated into the currently available open-access software packages, however, are associated with certain statistical drawbacks for the analysis of high-throughput screening data. In a primary screen without replicates, the mean difference and percent activity measures fail to incorporate data variability in the hit selection [2]. On the other hand, the regular z-score and SSMD methods assume that the variability of a sample RNAi is same as the variability in a negative reference group [25]. It is known that the robust versions of the z-score and SSMD are less sensitive to outliers due to the replacement of mean and SD with median and median absolute deviation (MAD) [9], [26], [27]. Therefore, the use of the robust versions of these methods is recommended for high-throughput screen analysis. Although z-score and SSMD are linearly related parameters for screens without replicates, the primary difference arises when the SSMD is applied for screens with replicates, in which case z-score is not a valid criterion to score RNAi samples with various variability [13]. On the other hand, in confirmatory screens with replicates, p-values associated with a z-score or t-test determines whether the null hypothesis, which is the mean of an RNAi sample being equal to the mean of the negative reference group, is accepted or not. Also, sample size has major influence on the resulting p-values due to the formulation of these tests. In contrast, SSMD across replicates provides a direct means of measuring the RNAi effect compared with the negative reference wells without the effect of sample size. In GUItars, we use the uniformly minimal variance unbiased estimate (UMVUE) of SSMD in the sample scoring [11].

In the siRNA screen shown, we applied robust SSMD hit selection at a cutoff of 1.28 on log₂-transformed data. As a result of the analysis, associated scores for all wells in every plate were written into an output file with the corresponding assay plate ID, RNAi plate ID, and assay plate well ID, which is categorized based on the well types (i.e., positive control, negative control, and negative reference) (Figure 5). To visualize the scores and the hit distributions in each plate, individual scatter plots are generated, in which horizontal lines are drawn corresponding to the cutoff used to select hits that either increase or decrease the assay activities. A series plot showing the well scores of all the screened plates is also necessary to explore the plate-to-plate variance within the screen. As seen in a representative graph in Figure 4E, the SSMD values and hit distributions are consistent among all the plates, except in plate 6, which has smaller plate median and MAD values, yielding higher SSMD scores.

In GUItars, the positive control and the negative reference wells are used as the high and low signals in the normalized percent activity calculations whereas only the negative reference wells are used in all the other methods. When the percent activity method is applied, the user has to define which control corresponds to the high signal and whether the hits are selected above or below the selected cutoff. If the t-test method is chosen, the “hit selection cut-off” is treated as the maximum p-value below which the hits are considered significant. Unlike SSMD, by the nature of the t-test statistics, the program outputs cannot classify whether an RNAi hit increases or decreases the assay signal.

The scoring calculations are followed by hit selection based on the cutoff defined by the user excluding the user-defined control and blank wells. The hit well IDs and the hit categories (i.e., increased or decreased signal) are determined above and below the positive and negative cutoff values (in percent activity and t-test, only the positive cutoff is considered), and written into an output file (Figure 5). A hit distribution heat map is utilized as a means to detect the effects of systematic errors in the final hit distribution. It is expected that the hits are distributed randomly among the wells in all plates; therefore, if a particular well is a “hotspot” for hits in most of the plates, that may be an indication of systematic errors. To provide a visual summary of the hit distribution, the frequency of a well being identified as a hit in all plates within a screen is calculated for each well and presented in separate heat maps for signal-increasing and signal-decreasing hits (Figure 4F). Additionally, the number of hits identified from each plate should also be consistent within a screen unless certain plates are specifically designed to lead to higher hit rates. Therefore, a screen-wise line plot of the number of signal-increasing and signal-decreasing hits is selected as another graphical output mode, allowing the user to inspect for the overall hit selection performance (Figure 4G). Thus, a hit distribution map along with a hit counts line plot provides crucial information for determining the reliability of the screen results.

Hit Annotations

If the “hit mapping to the RNAi annotation” option is selected, a hit mapping algorithm is used to match the hit well IDs of the assay plates with the corresponding RNAi plate annotations according to the user input “plate ID” and “RNAi annotation” files. For screens with replicates, one should use extra care to input the same RNAi source plate ID for all corresponding copies of the assay plates. An output file containing the assay plate ID, score, and the category of the hit wells is generated with the matched RNAi source annotations (Figure 5).