The authors have declared that no competing interests exist.
Conceived and designed the experiments: AEF MJP. Performed the experiments: AEF. Analyzed the data: AEF MJP. Contributed reagents/materials/analysis tools: AEF MJP. Wrote the paper: AEF.
Citizen science programs are increasingly popular for a variety of reasons, from public education to new opportunities for data collection. The literature published in scientific journals resulting from these projects represents a particular perspective on the process. These articles often conclude with recommendations for increasing “success”. This study compared these recommendations to those elicited during interviews with program coordinators for programs within the United States. From this comparison, success cannot be unilaterally defined and therefore recommendations vary by perspective on success. Program coordinators tended to have more locally-tailored recommendations specific to particular aspects of their program mission.
Science is a process that cannot be completely understood without fully understanding the actors, their interactions with each other and nature, and their context
The recent upswing in both quantity and scale of citizen science projects can be attributed to an increase in enabling technologies, the need for large-scale datasets, and the push for outreach by funding agencies
The science studies literature describes these benefits collectively as resulting from recognizing the value of non-scientific expertise (though caution they are not always achieved). Citizens experience the world in ways outside the traditional scientific method, allowing physical and temporal access to the question at hand that is necessary to provide these benefits and others. The perspectives citizens bring from their individual ways of knowing can offer creative approaches to scientific problem-solving and connect scientific information to policy applications
The focus on process by citizen science is a small-scale example of a larger societal phenomenon recognizing many forms of expertise in science – termed “democratizing science”
The benefits of citizen science and democratizing science originate from emphasizing process – specifically by spreading power derived from expertise among more people, creating a scientifically educated public for future scientific development, and connecting science to everyday life. Citizen scientific co-production of knowledge parallels benefits from co-production of policy, or co-management
Theorizing knowledge co-production and nesting citizen science within the broader movement to democratize science are recent phenomena. Many citizen science programs pre-date scholarship on the matter, hence there are concerns when attempting to achieve both scientific and social benefits in one program
In a fully democratized science, ‘success’ depends not only on the quality of the product but also the process. There is still a demand for the “best available science”, interpreted as high-quality data and building blocks for scientific theory development, especially from the established scientific community
Since citizen science efforts predate the movement for a democratized science and therefore may not employ the broadened definition of success, academic literature might leave out citizen recommendations for improving that success. Specifically, published articles favor the hypothesis-testing model of science that many citizen science projects do not easily fit into, suggesting the need for a more thorough review
We identified leaders of citizen science projects through a Google search using the search terms ‘volunteer’ and ‘monitoring’, followed by systematic random sampling of the search results beginning on a random page, using programs that identified as citizen science within the United States on their webpage. The search was intentionally structured this way to focus on long-term, environment-based citizen science programs. This left out other disciplines’ citizen science projects (like computer-based games for biochemical structures or astronomy identification); of course, we ran the risk of missing programs that did not identify themselves using this particular terminology, but we achieved an adequately sized group of programs for analysis. We contacted the phone number listed for either the program coordinator or primary investigator, with an overall 50% response rate for a total of 19. The non-responses were all from incorrect contact information: disconnected phone lines while not in field season (as determined post-analysis) or people no longer affiliated with the program. The responses may therefore favor programs with year-round staff and larger programs with up-to-date websites.
Structured interviews lasted around 20 minutes each following an interview guide with questions addressing program mission, daily function, and recommendations for other programs to be successful. We coded answers according to code trees directly addressing our research questions, adding sub-codes to each tree as they emerged. We also took notes during the interviews, especially of tangential or explanatory information, for later analysis coded using grounded theory
We identified articles for review through a search in three databases covering education (ERIC), environmental science (Web of Science), and sociology (Sociological Abstracts). Search terms included ‘citizen science’, ‘volunteer’, ‘assessment’, and ‘monitoring’. We reviewed each study that occurred within the United States and coded them according to the following questions (derived from the interview script for ease of comparison); articles answering none were dropped from review, leaving a total of 67:
Did the author consider the study successful as determined by the stated purpose of the study?
Did the author consider the data accurate? At what level?
What are the recommendations of the authors for other investigators hoping to use citizen science?
Program coordinator interviews reached programs that were on average 12 years old, with variation from one to 38 years. Programs size varied, having an average of 279 volunteers ranging from 15 to approximately 5000 (see
Number | Percent | |
Average age of program | 12 | n/a |
Volunteer coordinator | 10 | 53% |
Professional collaboration | 11 | 58% |
Mandatory training | 12 | 63% |
Consistent volunteer commitment | 12 | 63% |
Data publicly available | 18 | 95% |
Published in scientific literature | 11 | 58% |
Total n = 19.
The top recommendations from program coordinators, mentioned by at least a quarter of respondents, are as follows: 1) collaborate with experts, 2) have a consistent methodology, and 3) present data to policymakers (see
Surveys | Percent | Literature | Percent |
Collaborate with experts | 53% | Collaborate with experts | 48% |
Have a consistent methodology | 47% | Present data to policy influencers | 45% |
Present data to policymakers | 39% | Have a consistent methodology | 40% |
Have a standardized training program | 39% |
Over half (57%) of the literature used in the meta-analysis reported results from citizen science programs existing for more than five years. Most of the funding for these programs came from either government (30%) or academia (43%), which is often funded through grants from the National Science Foundation. The stated purpose of these articles, with the exception of two, was to aid the development of a citizen science program in a particular location or habitat. These demographic traits mean that most of the articles discussed programs that had long-term history and long-term goals.
The top recommendations for the literature are as follows: 1) collaborate with experts, 2) present data to policy influencers, 3) have a consistent methodology, and 4) have a standardized training program (see
Although the top recommendations were shared in the literature and surveys, there are a few notable differences. First, the fourth recommendation from the literature – to have a standardized training program – was only mentioned by one survey respondent, who commented that the only reason training was an issue for his program is because they hadn’t instituted standardized training from the inception of the program. Several forest monitoring programs mentioned the need to “get it right the first time”, incorporating the idea that initially setting up the program correctly, on a number of facets, is more important than any one aspect of program function.
The survey respondents created a much longer, more varied list of suggestions than the literature review generated, generally tailored to specific challenges their program had faced. Though these results were initially coded to the same categories as those found in the literature review, this specificity is notable because they are written for new program directors to be successful in similar programs. The literature review recommendations were aimed at a more general audience. For example, the top recommendation of the need to collaborate with experts was to “look at other existing programs”, for “networking, the way to answer specific questions”, or as “crucial to pull resources together effectively”. Many groups also stated they “absolutely would not be able to do anything [without partnerships]” and that meeting the other recommendations followed as a consequence of collaboration. These specifics better describe the nature of the collaboration, which experts are involved, and what products are expected as a result.
The recommendation of needing consistent methodology was often phrased in the context of a particular challenge the primary investigator faced. For example, some doctorate-holding volunteers in a Gulf of Maine monitoring program felt the need to revise existing protocols on the fly; the program coordinator attributed achieving consistent methodology in this case to enforcement and emphasis on existing written protocols. In contrast, the methodology challenge to most stream monitoring programs is the level of taxonomic detail required to follow the protocol; they instituted field handbooks in response. In yet other cases, practice made perfect, where more than 3 hours of training were required before volunteers could measure and officially record data.
The final recommendation from the literature, connecting to policymakers, was more straightforward. Still, there were some groups that approached the citizen science process as objective scientists and felt the data should speak for themselves. In these cases, they wanted to be able to hand over raw data to state or local agencies to include in larger databases used for decision-making. In other cases, the approach was through advocacy, where digested results would be presented at public meetings, annual volunteer meetings with invited policymakers, or included in letters to elected representatives.
Both sources of recommendations direct future programs toward behaviors for success. However, not all programs had the same definition of success. The mission of each program was not purely about creating scientific information, but about some combination of education, restoration, stewardship, and community-building. As with the recommendations for success, the surveys revealed more nuanced answers about mission or purpose. Many respondents hesitated a moment before classifying the success of their program, given the mission statement.
“Success” also depended on the larger socio-ecological context. One program coordinator stated that it is “hard to measure success” when watching floundering lobster nurseries despite their program’s efforts. She knew that they had saved at least six nurseries from development projects, but still had a hard time classifying the last few years of her program as ‘successful’. Others respond to context by changing their mission. A coordinator doubted the relevance of their mission, expressing uncertainty as to “who we should become next… we’re at a crossroads”. This program could not have success before defining their new mission. One grassland program described the work of the group as on an “as we can handle basis”, shifting from one mission to the next as need arises.
Stated program purposes fell into 6 general categories: mapping species distribution, increased stewardship, restoration, baseline data, tracking trends, and “not sure”. Examples of these missions include “create a habitat for the future”, take “just a snapshot”, “actively changing the program based on what the public needs”, and “to get data, but awareness and stewardship”. 84% of coordinators classified their program as having met its purpose with slightly less (68%) stating confidence in their data (or, in many cases, at least some of their data).
Because there are a multitude of purposes, the questions “do you consider yourself successful” and “is your data reliable” do not necessarily address the same thing. Though most coordinators were confident in their data, that confidence came with the caveat that the data may not be professional, but results are certain “from the sheer amount of time out there”. Others recognize that not all their data is reliable but they are “along the right steps” and have some data to share. The extreme ends of the spectrum also existed, creating data that “absolutely and exceeded” certainty or “data, maybe…education, certainly”.
The articles reviewed claimed success most of the time (97%), with their overall goal being scientific in nature. Part of this result may be due to the tendency for scientists to only publish positive results, leaving negative results to the grey literature or a filing cabinet
The concept of community, both inside the program and within the program’s region, was mentioned by many of the respondents in different contexts. When asked about program demographics, many coordinators said a better question to ask is how many people from the community are involved. Especially for programs in small towns, coordinators were proud of the fact that nearly every resident participated in some way in the program. They stated that the community-building process strengthened their program overall. One program held an annual summit, which the coordinator described as putting “context to their work” by presenting efforts to the community and showing volunteers tangible results.
At first glance, literature and interviews yielded similar recommendations – but only in the general sense. Interviews revealed a much more varied list of suggestions leading to less clear “top” suggestions (which were reported here for comparison). The recommendations out of the literature employed blanket statements strategizing scientific successes. Such recommendations need to be filtered through the particular context of a program, be that mission priorities, habitat, volunteer demographics, or something else. The more specific suggestions made by program coordinators emphasized particular parts of the process – collaboration for resources or new research questions rather than legitimacy through ties to established experts or data-checking. Thus, recommendations from the literature may be helpful when planning the broad aspects of a program but not as helpful in the first few struggling years that every program goes through.
Similarity in these broad-scale recommendations is logical because successful process often leads to successful product – in this case, reliable data. The difference between the recommendations from the literature and the surveys lies in recognizing the mechanism of achieving that end and the concomitant benefits. For instance, presenting data to policymakers may at first seem tangential to the scientific goal of a program, but through raising excitement in the program, such action ensures more long-term volunteers that maintain the strength of the program and the integrity of the data
The main difference in the broad-scale recommendations is that the standardized training emphasized in the literature did not emerge as a priority among program coordinators. Standardized training may already be expected in the structure of a volunteer program since multiple people are collecting data to later be compared. Perhaps this is so obvious an expectation that it need not be said in formalized recommendations. Alternatively, the term ‘standardized training’ might imply that volunteers are merely “data monkeys” with little input in their own educational experience. Since volunteers may expect constant interaction with scientists and other volunteers as part of their learning experience – and a common mission of the programs – standardized training may serve only as a minimal educational baseline; therefore it is prioritized in the literature recommendations focused on data, but not by program coordinators who aim for increasing education.
The programmatic mission as determined through the phone surveys varied, making the definition of success different for each program. Only one of the intended program purposes (collecting baseline data) was data-focused, defining success by the quality of data collected. Others were process-focused, such as tracking changes over time or increasing stewardship in participants. The process was more often successful than the product (data) and some of the respondents pointed out that the data is “good enough” or not the main focus of the program. This is in stark contrast with the published studies, many of which discuss citizen science as a method, evaluated against traditional methods by the same metrics of success – data quality.
Even though most of the programs were confident in the reliability of their data, most had also dismissed at least some of the collected data as unreliable. Program coordinators reported having many still-unpublished results, either because they were negative or not reliable enough. Consequently, the publications resulting from these types of programs represent a self-selected group with enough positive results to write about. The pool of literature found for the review likely represents a similar fraction of programs in existence. These programs had some component of their mission focused on creating baseline data. The recommendations from these programs found in the literature therefore refer to specifics that address only one type of mission, which is not representative of the myriad programs practicing.
Success in citizen science programs is defined by the particular mission statement guiding the program, which is more likely to focus on the scientific process than purely on the results. Of course, a strong and successful process is more likely to lead to reliable data, but not necessarily in all cases. The differences in the specificity of recommendations observed between the literature and surveys can be explained by what they were making recommendations for. The literature recommendations represent a subset of projects with successful missions relating to providing data while survey recommendations were more sensitive to other aspects of missions and incorporated more struggling programs. Thus the focus on process should be made explicit when making recommendations, advertising or fundraising for citizen science groups.
The authors would like to acknowledge the program coordinators who took the time to participate in our interviews and the research librarian staff at Cornell University’s Mann Library for help structuring the literature search. The authors would also like to thank Cornell’s Department of Development Sociology for funding and the citizen science team at the Lab of Ornithology for continuing the discussion on citizen science and motivating this publication.