Outsmart Invasive Species Project

We conducted our experimental study in the context of the Outsmart Invasive Species Project (Outsmart). Outsmart is a collaboration between the University of Massachusetts Amherst, the Massachusetts Department of Conservation and Recreation (MA DCR) and the Center for Invasive Species and Ecosystem Health at the University of Georgia. The project aims to strengthen ongoing invasive species monitoring efforts by enlisting help from citizens across New England (Connecticut, Rhode Island, Massachusetts, Vermont, New Hampshire, and Maine), with a particular focus on Massachusetts. Volunteers are asked to identify and report data on invasive plants and insects in their own time and submit data via a free account through the Early Detection and Distribution Mapping System (EDDMapS) website (www.eddmaps.org) or through our smartphone app called “Outsmart Invasive Species” (“Outsmart” for short; http://masswoods.net/outsmart) (Figure 1). The project leverages the increasing number of people equipped with smartphones or digital camera/web technology and aims to expand the scope of invasive monitoring with a particular focus on early detection of new or emergent threats.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Sample screenshot images from the Outsmart App.

https://doi.org/10.1371/journal.pone.0111433.g001

Invasive species monitoring has been identified as a key realm where citizen science can be employed particularly effectively [5]. Invasive species can wreak ecological and economic destruction and once established may be impractical or impossible to eradicate [9]. Thus, as researchers have identified “Monitoring programs aimed at detecting low-density ‘founder’ populations can play a critical role in slowing or even stopping the spread of harmful invasive species by identifying recently established populations that can be targeted for control and/or eradication” ([10], citing [9]). Yet, like other citizen science programs, one hurdle that the Outsmart project faced was how to train participants distributed across New England. To test the effectiveness of remote training, we developed in-person training sessions, training videos, and text/images and conducted a study asking participants to identify five invasive plant species in Massachusetts over the course of the late summer and early fall 2013.

Text Development and Video Production

We began our study by creating a set of training materials. For each species, we developed text and selected pictures that highlighted key features, working with a regional expert, Ted Elliman from the New England Wildflower Society (newenglandwild.org), and imagery available from the University of Georgia's Bugwood Image Database System (http://images.bugwood.org/). The text described distinct characteristics, seasonal changes in appearance, potential look-alikes, and the ecological threat posed by the species. To eliminate potential error from using different language for the different training groups, we aimed to keep the in-person, video, and text scripts as consistent as possible.

We did, however, allow for slight variations to create a smoother presentation of information as text, video, and in-person presentation are fundamentally different mediums. The text version for the Outsmart app contained key characteristics at the top of the page followed by species description, seasonal developments, look-alikes, and ecological threat. The app displays pictures that appear at the top of the smartphone screen with scrolling capability. The user can also click on a picture to zoom-in and pan around on a specific key feature. The text allows users to move around the page at their leisure and easily see the different sections.

The invasive id training videos for use on the app or on the web present information more linearly than text, with less ability to quickly jump to a particular section. In the videos, we began with a shortened description of the ecological threat, used the same language for the key characteristics, used a shortened version of seasonal changes, and used a slightly shorter text for potential look-alikes. The shorter text was deemed to be appropriate because video let us simultaneously present an image while verbally describing it, which reduced the need for a lengthy description. The shorter format was also deemed beneficial as it helped reduce smartphone battery drawdown, which is important if users view the videos while in the field.

The script used for in-person training was based on the video scripts and the instructor was careful not to offer additional special hints or clues. Participants of the in-person sessions were allowed to ask questions, which meant they could receive some additional information.

While aware that any differences in information given to the text/image, video, and in-person groups created variability, we concluded that slight variations were appropriate as they best captured real world applications of the three training methods. Therefore, by keeping training relatively standardized while allowing for the inherent strengths of the three methods to be captured, we reached a design representative of actual application in the field.

Volunteer Recruitment

During the summer of 2013 we worked with two interns funded by The Nature Conservancy to recruit participants at fifteen events (e.g., festivals, fairs, etc.) throughout the region. We also posted recruitment materials on the Outsmart project website and emailed anyone who submitted an invasive species report after July 1 to encourage them to participate in the study. Additionally, students in three Natural Resource Conservation courses at the University of Massachusetts Amherst were recruited to volunteer. All volunteers were directed to complete the online survey in order to participate in the study.

Survey

The survey was designed using Survey Gizmo, a web-based survey design tool (surveygizmo.com) and was pre-tested with colleagues and students. To increase the number of respondents the survey included institutional icons at the top for legitimacy, length of time the survey would take, end-date of the experiment to create urgency, research context to emphasize significance and importance, and disclosures that data would be kept private.

In the survey, participants were asked to self-identify their plant ID experience level as “No experience”, “Beginner”, “Intermediate”, or “Advanced”. Participants who identified as “Beginner” “Intermediate” or “Advanced” were asked an additional two questions. The first question asked them to identify how they learned their plant ID skills. The second asked if they could specifically identify, without aids, the five species used in the experiment (described in Study Design section below). In addition, the survey collected data on gender, age, and education level.

Upon completing the survey, participants were randomly assigned, via an automated process in Survey Gizmo, to text/image, video, or in-person training. Depending on the cohort they were assigned to, participants received different instructions upon completing their survey. All instructions included links to both the Android and iPhone Outsmart application, links to create an EDDMapS account (a prerequisite for using the Outsmart mobile app), instructions on using the app, and contact information for the research team. The instructions for Cohort 1 (in-person training) told participants they would have to attend an in-person training session at one of three established times and they were also encouraged to use embedded text/images and video to identify the five target species. If participants in this group were unable to attend one of the training sessions they were re-assigned to the text/image or video group. The instructions for Cohort 2 participants (app with video training) told them that they could use the embedded training videos in addition to the embedded text/images to identify the five target species. The instructions for Cohort 3 (text/image training only) told participants to use only the app-embedded text/images to identify the five target species.

Study Area

While the Outsmart project has participants across New England, for the purposes of the experiment we limited our study to Massachusetts. Massachusetts is the eighth most forested state in the nation and has a wide variety of topography and forest types ranging from mountainous spruce-fir-northern hardwoods in the western portion of the state to the coastal plains and lowland pitch pine-scrub oak forests to the east [11]. Massachusetts was also one of the first states to be settled by Europeans and has undergone extensive land-use change since that time [12]. These factors, in combination with accidental and intentional introductions of non-native species led to the occurrence of many different species of invasive plants in the state. Massachusetts Division of Fisheries and Wildlife's Natural Heritage Endangered Species Program in partnership with the Massachusetts Invasive Plant Advisory Group (MIPAG) have identified 67 plant species that are either "Invasive," "Likely Invasive," or "Potentially Invasive" [13].

Study Design and Data Collection

Of the many invasive species found throughout Massachusetts, we limited our experimental study to five common species that represents invasive plant species both relatively easy and difficult to identify. Our assessment of a species difficulty was based on conversations with field biologists at the U.S. Fish and Wildlife Service, examples in the literature for some species [8], and a survey of University of Massachusetts students in a plant identification course. The easy to identify species were: Japanese knotweed (Fallopia japonica), autumn olive (Eleagnus umbellata), and multiflora rose (Rosa multiflora). The more difficult species were: glossy buckthorn (Alnus frangula) and exotic honeysuckles (Lonicera spp). Honeysuckles are actually a group of species in the same genera, but are referred to as a single species here for simplicity.

The general inspiration for our study design came from our desire to mimic the experience of actual Outsmart users as closely as possible. After receiving training, we asked participants to use the app from June 1, 2013 through September 30, 2013 and report any suspected occurrences of the five assigned species. We deemed this timeframe to be appropriate as it encouraged as many submissions as possible while limiting potential bias due to seasonal change in plant appearance. Just like a regular Outsmart user, participants in the experiment were allowed to report any sightings, whether they specifically went out looking for the invasive species or if they just happened to notice them while out hiking, walking, or doing other outdoor activities. User submissions were verified as correct or incorrect by Outsmart researchers utilizing the EDDMapS interface.

Data Analysis

We received 534 total submissions from 78 participants: 19 in Cohort 1 (in-person trained), 24 in Cohort 2 (app and video training), and 33 in Cohort 3 (text and image trained group). Although these volunteers were randomly assigned to groups of equal size, the final groups were unbalanced due to participants dropping out of the study. It is likely that the in-person group had the smallest number of participants because of the additional time and effort required to show up for in-person training. One participant was dropped from further analysis because they submitted data from northern Vermont, which was outside our study area of Massachusetts. A second participant was dropped because they only submitted data for non-targeted species for the experiment. One record was dropped from one participant because a picture was not included. We ended up with 529 usable submissions from 76 participants.

When we examined our submissions by date, we discovered that 95% of our data was submitted between September 2 and September 30. This gave us additional confidence that our results would not be biased by seasonal changes in plant appearance. The September-skew may be due to general “participant procrastination” or the fact that students formed a large contingent of our participants. It may be that these students chose to defer their participation in the experiment until they were back at the university.

To analyze the data further, we generated a descriptive statistic for the percent correct, by training type (cohort), for each species of plant. This was generated by aggregating data from all users and then dividing the number of correct submissions for each species by the total number of submissions for that species.

We next analyzed the data at the individual participant-level. For each participant we divided the number of correct submissions by the total number of submissions, which yielded a percent correct score. We then used one-way analysis of variance (ANOVAs) to look for differences across all training types and also between training types [8]. Because the participant's percent correct scores was sensitive to low numbers of submissions we only used data from participants that submitted 5 or more observations when conducting these ANOVAs. This yielded a total of 56 participants: 14 in the in-person training cohort, 17 in the video training cohort, and 25 in the text and image training cohort.

Because our samples included individuals with different plant ID experience levels (beginner, intermediate, and advanced), ages, and education levels we also ran a generalized linear model (GLM) using the raw data on number of correct submissions out of total submissions as our dependent variable [14]. We used a binomial distribution, and weighting the number of submissions so that those who submitted more data were given more weight in the model. This allowed us to confidently use data from all 76 participants without low submission levels skewing the results. We ran a series of GLMs testing for effects and interactions between the plant id experience, training type, age, and education variables. Lastly, to decouple any interactive effects, we ran GLMs for each plant ID experience level (beginner, intermediate, and advanced) to see the effect of training type on percent correct within each levels of experience.

Plant Species

The percent correct by species shows that in general our understanding of the species' difficulty were reasonably accurate (Table 1). An exception may be Glossy Buckthorn and Autumn Olive. Glossy Buckthorn was classified as difficult, and this seems correct for those who received text/image training, but for those who received video on the app or in-person training Autumn Olive actually had lower percent correct scores. This may be because Russian Olive is quite common and similar in appearance to Autumn Olive, which led to more false reports for this species.

Exotic Honeysuckle, as expected, proved to be the most difficult species to identify across all three training types and speaks to the limitations of effectively training citizen scientists when it comes to very difficult to identify species. Although, the fact that video training was on par with in-person training (in fact doing slightly better for Exotic Honeysuckle identification) suggests that this training method may be as effective as in-person training even for difficult species.

Across all species, we find that all training types were fairly effective with even submissions from the text/image training group being correct 81% of the time (Table 1). Our hypothesis generally held true across species that those with app-based video training would do better than text/image training and in-person training would do the best. Encouragingly though, for the feasibility of broadly providing training to many people, submissions from the video-trained cohort actually outperformed the in-person training cohort in 3 of the 5 plant species. These findings need to be considered carefully though as the interactive effect of previous plant ID experience and training type is not taken into account in these descriptive statistics.

Training Types

Looking at the average percent correct of individuals gives us additional confidence that the training types were significantly different. Our expectations were met that the text/images participants would have lower correct plant ID scores than the other two training types (Figure 2). Surprisingly, participants trained with video on the app actually had a slightly higher mean percent correct score than those trained in person, but this was not a statistically significant difference. We expected in-person training to do the best across the board, so this is an exciting finding as it shows that video can be at least as effective for training citizen scientists in plant ID.

Additionally, even though text/image training via the app was not as effective as app-based video or in-person training, the mean plant ID score was still 79% (Figure 2). This suggests it is possible for citizen science projects to use remote training methods and still acquire high quality data. And if well-produced videos are used for the training, the submitted data may be on par with that submitted by those trained in-person. This has encouraging implications for expanding the scope and scale of citizen science projects.

One caveat for the text/image group is that we did not have a way to restrict their access to the app-embedded training videos. Instead we explicitly instructed them to only use text/images and trusted them to comply. We also did not have a way to prevent participants in any of the groups from using outside resources such as guidebooks or online materials. However, as the results largely confirmed our expectations on training type and plant ID ability, it would appear that participants followed our instructions and did not cheat.

Plant ID Experience

Looking at the effects of training type, while controlling for previous plant ID experience, are encouraging. The group classified as beginners are those we are most interested in because they had little to no previous plant ID experience. This group allows us to most clearly see the effect of the different training methods. Here we see that those receiving text/image training had a median percent correct score in the mid 60% range, but there was a large variance (Figure 3). The group trained with video on the app was more homogenous, with a median percent correct score of 100%. Finally, those trained in person had a perfect score of 100% correct plant ID. The sample sizes for all three groups were fairly low, but the results are encouraging and suggest there is value in further study.

The effect of training type on the intermediate group was a bit less clear. In-person training was significantly better than app-based text/image, but app-based video was statistically equivalent to app-based text/image training. The video group had a large variance, which we attribute to a small sample size.

Finally, the advanced plant ID experience group had results that we expected. Regardless of training type they all had median plant ID scores of 100% correct (Figure 3). This is logical considering people that already possessed advanced plant ID experience weren′t influenced by training, since they likely already knew how to identify the targeted species. One interesting aspect of this group however is the large variance in the in-person training cohort, where some people that self-identified as “advanced” actually did quite poorly at plant ID. We are calling this phenomena the “overly confident” or “arrogance” factor, meaning it appears some participants incorrectly thought they already knew how to identify targeted plants and failed to pay attention during in-person training, which led to lower correct identifications.

One caveat to this component of the study is that participant's plant ID classification was derived from a self-classification on our survey. We did not independently test or verify their experience. Yet, as the results largely match our expectations for the groups, we believe that this self-classification was a generally accurate reflection of participants' abilities.

Limitations

While app-based text/image training was quite effective at plant ID and app-based video training was shown to be equivalent to in-person training, it is important to mention the limitations of our study. We only targeted five invasive plant species and our study was confined to the Commonwealth of Massachusetts. It is unclear how transferable these findings may be to other species and geographic areas. While we tried to study a range of difficulty levels, the relative difficulty of Exotic Honeysuckle ID, may suggest that for very difficult species citizen science projects may have trouble training volunteers to correctly identify a species, even with in-person training.