Open Access

Research Article

Introduction Top

The PLoS Web site employs a system for the assessment of articles that combines pre- and post-publication peer review [1].

Notes, Comments and Ratings

There are three major components to the post-publication review:

1. Notes: Notes apply to a specific point in the Web version of the article text and should be used to highlight a minor point, to make additions or clarifications, or to identify and link to material, including more extensive discussions, presented elsewhere. There isn’t a word limit on these notes, but when highlighted, the note is truncated after a few lines with a link through to the full note. Anyone, including authors, can reply to a note so that a conversation can develop around a specific point in the article.
2. Comments: Comments are of a more general nature and can also be added to the article, leading to threaded discussions concerning the content, conclusions, and consequences of a specific article. Comments are not attached to any specific part of the article and they can be extended by replying to any comment in the discussion thread. This allows for multiple branching threads of discussion.
3. Ratings: Users can assign individual ratings to articles, which are then aggregated to provide overall ratings and rankings of articles. Users are not allowed to rate more than once, but can revise their rating. For more details, see [2].

Authors of all Notes, Comments, and Ratings will be identified.

Method Top

Who Can Contribute?

All registered users are able to add Notes, Comments, and Ratings to any article. Anyone can register as a user. Users are required to unambiguously identify themselves and to supply a valid e-mail address in order to register. Registration information will be kept strictly confidential by PLoS staff unless otherwise indicated. PLoS reserves the right to suspend the privileges of any registered user. Discussions and ratings are not anonymous. Please view the PLoS Privacy Statement [3] and the Terms of Use [4]summary for more information.

How to Add and View Notes

To make a Note, first make sure you are logged into the Web site. Highlight the text to be annotated, and then click the "Add a note to the text" link in the right-hand navigation menu of the article. The drop-down list should already have “Note” selected. If it is not, then select “Note” from the drop-down list. Enter a title and text for your Note in the appropriate boxes. When you are satisfied with the text, click "Post" to attach the Note to the article. Any correctly formed URLs in the Note text field will automatically become working links.

Notes can be started at any point within the text, but for ease of reading we ask that you do not begin Notes in the middle of words. We advise that longer Notes are first written in a word-processing program that allows for spell checking before they are copied and pasted into the Note field.

Notes are represented by the small blue "clouds" within the online text. The number in the cloud indicates the number of Notes that begin at that point in the text. To see what selection of text is associated with a Note, run the cursor over the cloud, and the text will be transiently highlighted. Click on the cloud to view the title, date, contributor, and first 250 text characters of a Note. To view the complete text of a Note or the citation information of a Note, click the "View/respond to this" link to view the full text.

How to Add and View Comments

To add a general Comment about the article as a whole, first make sure you are logged into the Web site. To create a Comment, click the "Make a general comment" link in the right-hand navigation menu of the article. Enter a title and text for your comment in the appropriate boxes.

We advise that Comments are first written in a word-processing program that allows for spell checking before they are copied and pasted into the Comment field. When you are satisfied with the text, click "Post" to attach the Comment to the article. Any correctly formed URLs in the Comment text field will automatically become working links.

To view all Comments and Notes on an article, along with any replies, click the "View/join ongoing discussions" link in the right-hand navigation menu of the article.

How to Add and View Corrections

Authors and other users can post two kinds of Corrections: Minor Corrections and Formal Corrections. Minor Corrections indicate small errors and clarifications to the article, whereas Formal Corrections are reserved for errors that significantly affect the utility or understanding of the article. Formal Corrections are also sent to PubMed Central and PubMed. In both cases, the decision to indicate a posting as a Minor or Formal Correction is the responsibility of PLoS staff.

To make a Correction, first make sure you are logged into the Web site. Highlight the text to be corrected, and then click the "Add a note to the text" link in the right-hand navigation menu of the article. Select “Correction” from the drop-down list. Enter a title and text for your Correction in the appropriate boxes. When you are satisfied with the text, click "Post" to attach the Correction to the article.

The Correction will initially appear as a Note but will be promptly reviewed by PLoS staff. After the Note is reviewed, it will either stay as a Note or be changed to a Minor Correction or Formal Correction.

Once a Note is changed to a Correction, the Correction is represented by a small red triangle "cloud" within the online text. The number in the cloud indicates the number of Corrections that begin at that point in the text. To see what selection of text is associated with a Correction, run the cursor over the cloud, and the text will be transiently highlighted. Click on the cloud to view the title, date, contributor, and first 250 text characters of a Correction. To view the complete text of a Correction or the citation information of a Correction, click the "View/respond to this" link to view the full text.

Formal Corrections are also listed above the title of the article.. The title of the Formal Correction and a link to the Formal Corrections will appear in this block.

Responding to Notes, Comments, and Corrections

To respond to a Note, Comment or Correction, click the "Respond to this Posting" link in the bottom-right corner of the window. Enter a title and text for your response in the appropriate boxes. We advise that responses first are written in a word-processing program that allows for spell checking before they are copied and pasted into the comment field. When you are satisfied with the text, click "Post" to attach the response to the Note, Comment or Correction. Any correctly formed URLs in the response text field will automatically become working links.

Following this procedure, an unlimited number of branching threads can be anchored to an initial Note, Comment or Correction. If any contribution is removed following moderation, all subsequent responses along its thread will also be removed.

Rating Categories

The PLoS Web site allows users to rate articles for subjective "quality". Scientific work can be measured on a number of scales. To reflect this, research articles can be rated in three separate categories: Insight, Reliability, and Style. Rating is done on a 1 to 5 scale with 1 being the lowest and 5 the highest rating.

1. Insight: This provides a measure of how thought-provoking a user found an article or how much it advances our scientific understanding. The scale for Insight ranges from 1, Bland, a report which provides no more than an incremental advance on the published literature; to 5, Profound, a report which substantially deepens or alters current thinking.
2. Reliability: This provides a measure of how secure a user feels the results and conclusion in a study are. The scale for Reliability ranges from 1, Tenuous, the study is preliminary and will need confirmation; to 5, Unassailable, the results are of high quality, the reasoning is tight, and the conclusions are completely solid.
3. Style: This provides a measure of how well performed and presented a user considers a study to be. The scale for Style ranges from 1, Crude, the technical accomplishment and presentation of the study is adequate at best; to 5, Elegant, the study satisfyingly presents the results of technically accomplished and expertly executed experiments.

Users can rate research articles in any or all categories; the ratings given are combined to give an overall rating from that user. Note that Front Matter articles, such as Editorials, Perspectives, and Reviews, can only be given a single overall rating.

Ratings from all users are combined to give average ratings in each category and a combined overall rating. The number of users who have rated an article is also displayed.

Users can also supply comments to accompany their rating of an article. These can be used to explain the ratings given. Unlike other forms of comment, rating comments cannot be used as the beginning of a discussion thread. The comment field is optional; however, comments cannot be made unless the user also rates the article in at least one category.

Figure 1. PLoS Development Team

The PLoS development team at the release of Ambra 2.0 on November 11, 2011. From left to right: Richard Cave, Alex Kudlick, Scott Sterling, Mike Baehr, Jennifer Song, Eddie Dickey, Bill O'Connor, Joe Osowski, Zoe Mclaughlin

doi:10.1371/journal.pone.0000000.g001

Viewing Ratings and Rating Comments

The overall "Average Rating" for an article is displayed in the right hand navigation menu for all articles. Clicking on "See all categories" reveals the average rating in each category. Also displayed is the number of users who have rated the article; "User Ratings" links to a page containing the individual ratings from each user as well as any comments they have included.

Ratings in all and any category are displayed on articles whether they have been rated or not. However, when compiling lists based on ratings, PLoS will only include articles rated by more than 10 users.

How to Rate

To rate an article first make sure you are logged into the Web site. Then, click the "Rate This Article" link in the right hand navigation menu of the article to be rated. This will open a separate window within which you can select your desired number of rating stars for the study and enter any comments that you may wish to accompany your rating. Once completed, you will be returned to the article. Ratings and rating comments can be modified at any time by following the 'edit my rating' link in the right hand navigation menu.

Good Practice

All contributions must conform to the norms of civilized scientific discussion. Any contributions that do not meet these standards will be removed. Any users who consistently transgress these conventions will have their user privileges removed.

A partial list of discussion standards includes the following:

1. Language that is insulting, inflammatory, or obscene will not be tolerated.
2. Unsupported assertions or statements should be avoided. Comments must be evidence-based, not authority-based.
3. When previously published studies are cited, they must be accurately referenced and, where possible, a DOI and link to a publicly accessible version supplied.
4. Unpublished data should be provided with sufficient methodological detail for those data to be assessed. Alternatively, a permanent Web link to such information should be provided.
5. Arguments based on belief are to be avoided. For example the assertion, "I don't believe the results in Figure 2." must be supported.
6. Discussions should be confined to the demonstrable content of articles and should avoid speculation about the motivations or prejudices of authors.

Requesting Review of Inappropriate Content

To request review of a comment you believe to be inappropriate, or if you suspect that the rating system is being abused, click the "Request review" link in the bottom-right corner of the rating. Indicate why the rating should be reviewed ("Spam," "Offensive," "Inappropriate," or "Other"), enter additional information in the text box, and click "Submit". PLoS staff will investigate the concern, which may involve consulting external experts if necessary. Any comments that PLoS staff deem to be inappropriate will be removed. Any users who consistently posts inappropriate material will have their user privileges removed.

PLoS is the final arbiter of the suitability of content for inclusion in the PLoS Web sites.

Commentary and Ratings F.A.Q. Top

Question 1: I have just read a peer-reviewed article in one of the PLoS journals. What do I do next?

Rate the article - tell us what you think of it by giving it a star ranking (1-5). Post a note and/or a comment onthe article. This will help build on the article's research and start a discussion with your peers.

Question 2: But I am not an expert in the field!

That is perfectly OK - nobody can be an expert on everything. But you have read this article because you were interested in the topic and understood some, most, or all of the article. Thus, you are an educated, informed reader. A comment or question that you post will be worthwhile and of interest to others.

The authors, referees and academic editors of PLoS articles are encouraged to respond to your c omments and questions. They have put a lot of hard work into the manuscript and will appreciate compliments as well as polite criticisms and questions from both experts and non-experts.

Comments that are entirely off-topic, or offensive, or posted by purveyors of pseudo/non/anti-science, will be deleted. If you notice an inappropriate comment, please flag the comment so that we can evaluate it and act accordingly.

Question 3: Q: I think the article is well done, well written and complete. I have nothing to ask, critique or add to it. I also think that the paper adds another piece to the puzzle, is not controversial and does not challenge existing theory. Good, but not exciting. Should I say anything?

Yes. PLoS welcomes such articles. You may choose to rate the article high on Reliability and Style and lower on Insight if that is how you feel. But do post a comment stating your opinion nonetheless. Congratulate the authors on good work and note that you think it is a valuable addition to the existing body of knowledge without being revolutionary.

Question 4: I think the article is challenging the current dogma in ways that make me excited about the new view. But I may have quibbles with some details.

Rate the article, perhaps giving somewhat lower marks on Reliability and Style and higher on Insight, if that is how you feel. Post a comment that shows your excitement about the work, as well as voices your minor concerns.

Question 5: I think the article has a major problem, but I am afraid to challenge a big name in my field.

Your nervousness is understandable. But if you believe that you have identified a real problem with the article and you feel confident , it is likely that other readers will feel the same. Be the first one to comment about it (try to use non-confrontational language such as 'could' not 'should' etc) and read the responses of others who may agree or disagree with you. On the PLoS journals, everyone is equal and everyone is expected to treat others with equal respect. Courage to challenge authorities will gain you a fair reputation among your peers.

Question 6: What is in it for me? Why should I spend my time and expose myself to do this?

Think of it in a long-term perspective. Post-publication review is likely to spread over time and become ubiquitous. Within a year or so, all PLoS journals will have this functionality. Other publishers are likely to follow suit. People who comment frequently and write valuable comments will build reputations over time. If you rate, note and comment on PLoS papers now, you will be one of the early adopters and will be recognized and respected for this in the future.

Within a few months, there will be a new functionality implemented on PLoS articles, whereby users will be able to rate each others' comments. Hopefully, this rating system will be universal over all Open Access journals with post-publication reviews. Frequent quality commenting will help you build up a high rating score from your fellow commentators. In the future, this score will be a number that you could include in your CV and use for promotions, getting jobs, etc. Also, prolific commenting may result in offers for future scientific collaboration or employment. This has already happened to prolific commenters on science blogs and is even more likely to happen to prolific commenters on peer-reviewed articles.

More Information Top

For more information about notes, commenting and rating papers, including the technical details, see the Comment Guidelines and Rating Guidelines.

Then charge ahead and get started - join the community!

Acknowledgments Top

We acknowledge the guidance of Drs H. Varmus, P. Brown, M. Eisen, J. Omura, past members of the Topaz development team and the past/present staff at PLoS[5]. Special thanks to past contributors from the Topaz development team (Stephen Cheng, Eric Brown, Joycelyn Chung, Amit Kapoor, Pradeep Krishnan, Ronald Tschalär, Ed Yoon) and from PLoS (Susanne DeRisi, Dragisa Krsmanovic, Mark Patterson, Nick Peterson, Margaret Shear, Russell Uman, Bora Zivkovic).

References Top

1. Public Library of Science (2008) PLoS Guildelines for Commenting, http://www.plosone.org/static/commentGui​delines.action.
2. Public Library of Science (2008) PLoS Guidelines for Rating, http://www.plosone.org/static/ratingGuid​elines.action.
3. Public Library of Science (2008) PLoS Privacy Statement, http://www.plosone.org/static/privacy.ac​tion.
4. Public Library of Science (2008) PLoS Terms of Use, http://www.plosone.org/static/terms.acti​on.
5. Public Library of Science (2008) PLoS Staff, http://www.plos.org/about/people/staff/.