Figures
Abstract
Objective
The purpose of this pilot study is 1) to develop an annotation schema and a training set of annotated notes to support the future development of a natural language processing (NLP) system to automatically extract employment information, and 2) to determine if information about employment status, goals and work-related challenges reported by service members and Veterans with mild traumatic brain injury (mTBI) and post-deployment stress can be identified in the Electronic Health Record (EHR).
Setting
Post-deployment Rehabilitation and Evaluation Program (PREP), an in-patient rehabilitation program for Veterans with TBI at the James A. Haley Veterans' Hospital in Tampa, Florida.
Main Outcome Measures
Documentation of employment status, goals, and work-related challenges reported by service members and recorded in the EHR.
Results
Two hundred notes were examined and unique vocational information was found indicating a variety of self-reported employment challenges. Current employment status and future vocational goals along with information about cognitive, physical, and behavioral symptoms that may affect return-to-work were extracted from the EHR. The annotation schema developed for this study provides an excellent tool upon which NLP studies can be developed.
Conclusions
Information related to employment status and vocational history is stored in text notes in the EHR system. Information stored in text does not lend itself to easy extraction or summarization for research and rehabilitation planning purposes. Development of NLP systems to automatically extract text-based employment information provides data that may improve the understanding and measurement of employment in this important cohort.
Citation: Dillahunt-Aspillaga C, Finch D, Massengale J, Kretzmer T, Luther SL, McCart JA (2014) Using Information from the Electronic Health Record to Improve Measurement of Unemployment in Service Members and Veterans with mTBI and Post-Deployment Stress. PLoS ONE 9(12): e115873. https://doi.org/10.1371/journal.pone.0115873
Editor: Chris T. Bauch, University of Waterloo, Canada
Received: May 18, 2014; Accepted: November 27, 2014; Published: December 26, 2014
This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Data Availability: The authors confirm that, for approved reasons, some access restrictions apply to the data underlying the findings. Public data deposition is not ethical or legal and would compromise patient privacy. Data are housed in the VINCI Veterans Affairs (VA) Informatics and Computing Infrastructure (VINCI), a secure, virtual computing environment developed through a partnership between the VA Office of Information Technology (OI&T) and the Veterans Health Administration's Office of Research and Development (VHA ORD). VINCI is a partner with the Corporate Data Warehouse (CDW) and hosts all data available through CDW. VHA National Data Services (NDS) authorizes research access to patient data. Data are only available for researchers who meet the criteria for access to confidential data. Contact: VINCI@va.gov.
Funding: Funding for this work was provided by VHA QUERI RRP 12-450. The views expressed in this work are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the US Government. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Traumatic Brain Injury (TBI) is referred to as the “signature injury” of modern warfare due to the types of explosives used and the improved survivability of those injured in combat [1], [2]. TBI is defined as “a traumatically induced structural injury and/or physiological disruption of brain function as a result of an external force that is indicated by new onset or worsening of at least one of the following signs or symptoms immediately following the event: (a) any loss of consciousness (LOC); (b) any loss of memory for events immediately before or after the event; (c) any alterations in mental state; (d) neurological deficits that may or may not be transient; and (e) intracranial lesion” [3]. TBI severity is categorized as mild, moderate or severe based on the length of LOC, alteration of consciousness or post-traumatic amnesia (PTA) at the time of injury and not by the severity of symptoms at various points in time after injury [3], [4]. Researchers have a consensus about the definition of moderate to severe TBI, but the definition of mild TBI (mTBI) has been elusive [5], [6]. The VA/DOD adopted the American Congress of Rehabilitation Medicine (ACRM) definition of mTBI that specifies that mTBI does not exceed PTA for greater than 24 hours, Glasgow Coma Scale (GCS) of 13 to 15 after the first 30 minutes; and/or LOC for more than 30 minutes and absence of traumatic lesion when radiographic studies are obtained [3], [4], [7].
Employment has been emphasized as a key component of community reintegration for persons with disabilities [59], [60]. Return to work after sustaining a TBI is a common goal across civilian and veteran populations. Due to the consequential nature of TBI, returning to work after injury is often a challenge [20], [26], [27], [29], [38], [57]. Failure to return to work can lead to poor psychosocial outcomes such as decreased community integration and decreased financial independence [27], [54], [55], [57]. These factors are of particular importance to Veterans with mTBI when they return from duty and attempt to reintegrate into civilian life. Veterans may not be diagnosed with mTBI until they experience difficulties reintegrating into society [22], [58]. The cognitive difficulties resulting from the mTBI such as memory loss, attention deficits, and executive control dysfunctions can affect reintegration and successful employment efforts [20], [22], [26], [27], [29], [32], [35], [57], [61], [62].
In this article, we begin with a brief review of current literature available on the complex challenges Veterans with mTBIs face when participating in rehabilitation and transitioning into the community. Then, we describe the TBI Post-Deployment Rehabilitation and Evaluation program (PREP) at the James A. Haley hospital in Tampa, Florida. Next, we provide a brief overview of information extraction (IE) methods used in mining of patient electronic medical records. Information found on Veteran employment status and vocational history stored in text notes located within the Electronic Health Record (EHR) is presented. In conclusion, we discuss how the pilot study findings may improve our current understanding of the reintegration and employment needs of Veterans with mTBIs. Finally, we discuss how development of future natural language processing systems (NLP) may support easy summarization of employment status, goals, and work-related barriers located in text in the EHR.
Review of the Literature
Since 2000 more than 280,000 incident diagnoses of TBI in all military personnel (not just war-related TBI) have been reported, with a majority of these injuries being classified as mild [8], [52]. Determining the exact incidence and prevalence of deployment-related TBI is challenging due to the fact that mild injuries (mTBI) may not require immediate medical attention and it is possible that these injuries may not be documented directly after the event occurs when acute injury characteristics are most accurately assessed [3].
The rate of TBI in military personnel is higher than in the civilian population [3], [9]. It is estimated that approximately 1 in 6 US service members deployed in Operation Iraqi Freedom (OIF), Operation Enduring Freedom (OEF), or Operation New Dawn (OND) has sustained a TBI, with many of these Veterans experiencing multiple TBIs and complex injuries [3], [10]–[14]. Vast majorities of combat TBIs are blast-related and are often associated with polytrauma, which is defined as two or more injuries to physical regions or organ systems, one of which may be life threatening, resulting in physical, cognitive, psychological, or psychosocial impairments and functional disability [3], [10], [15]–[17]. Blast injuries have been identified as one of the main causes of mTBI in service members who have been deployed [22] and a history of mTBI is common among those who served in OIF/OEF conflicts [14], [26], [63], [64].
In general, TBI typically results in short- and long-term physical, cognitive, emotional, and behavioral consequences that negatively impact an individual's ability to live an independent, integrated, and productive life [18], [19]. Previous research highlights the potential for functional impairment following TBI [10], [19]. Most symptoms of mTBI (e.g., somatic, cognitive impairment, and psychological) will resolve during the first few weeks and months post-injury, although some individuals (5–15%) continue to experience symptoms for long periods of time with some Veterans experiencing cognitive difficulties that impede reintegration into society post-combat [3], [10], [20]–[22], [26], [29]. Complex co-morbidities such as chronic pain, irritability, sleep disturbances, and mental health disorders (e.g., depression, anxiety, PTSD) interfere with work and independent living [16], [20], [23]–[30].
Overall, Veterans with TBI may experience higher rates of unemployment than civilians and have more difficulty obtaining employment compared to those in previous conflicts, while they are often in their prime earning years [10], [31]–[33]. Veterans and service members with TBI may experience high unemployment rates in part because these individuals experience multiple problems when returning to work. The number, complexity, and interaction of problems may contribute to the difficulty in maintaining long-term employment [20], [24], [34], [35], [70]. For example, problems may be due to environmental barriers such as access to reliable transportation or the ability to drive in civilian environments [44]–[46], [56], [67]. Other pre-injury factors, such as premorbid work history, employment status, and income have been associated with more successful return to work [35], [37], [57], [65], [68]–[70].
PREP Program
The Post-deployment Rehabilitation & Evaluation Program (PREP) is a unique in-patient rehabilitation program located at the James A. Haley Veterans' Hospital in Tampa, Florida. This program provides evaluation and treatment of mTBI and post-deployment difficulties including post-deployment stress. In the PREP program, comprehensive evaluations are completed in the in-patient setting rather than extending the evaluation period over a course of several months. The average length of stay is 3–6 weeks, depending upon rehabilitation and mental health goals, although briefer targeted admissions and extended stays are common. Veterans and service members who complete this program undergo specialized evaluations conducted by physiatrists (rehabilitation medicine physicians), physical/vestibular therapists, occupational therapists, speech and language therapists, recreational therapists, vocational rehabilitation counselors, pain specialists, audiologists, vision therapists, social workers, psychiatrists, sleep medicine specialists, neuropsychologists, and psychologists. An interdisciplinary team of clinicians and providers with expertise in mTBI, chronic pain, and post-deployment stress offer integrated and holistic coordinated care across two phases. This includes a one- to three-week comprehensive evaluation to examine physical, cognitive, and mental health symptoms (Phase 1) and an intensive treatment for post-deployment and combat injuries including PTSD and other readjustment issues (Phase 2).
Patients who complete the PREP program report improvements in their cognitive and physical abilities (85% and 78% respectively), emotional functioning (62%), and reductions in headaches, pain, and dizziness [53]. Overall, 43% of program graduates reported improvement in their self-efficacy, which is associated with greater likelihood of meeting the demands of the work environment and improved quality of life [36], [37], [53]. PREP notes were selected for review because they contained rich information on reintegration and employment needs of Veterans and service members with mTBI.
PREP participants interact with a wide variety of clinicians during their stay; therefore providing a rich environment of the kind of information that might be available throughout the Veterans Health Administration (VHA) Electronic Health Record (EHR). The VHA has used an EHR since the 1970s called VistA. This system records important data regarding a Veteran's encounter with each clinician in both structured data tables and text. In the VHA, vast amounts of information are locked away in EHR in the form of progress notes. As of 2013, there were over 2 billion progress notes in the VHA. Many aspects of a patient's rehabilitation trajectory are recorded in progress notes, including diagnosis, rehabilitation strategies, and patient outcomes. These progress notes provide a rich source of rehabilitation-related data, possibly the largest of its kind in the world. Information in these data could provide valuable insights into methods and outcomes of rehabilitation, particularly those related to employment.
Information Extraction
Information extraction (IE) is defined as the extraction of predefined types of information from text [72]. There are four primary methods available to implement an information extraction system, including natural language processing (NLP), pattern matching, rules, and machine learning. The use of rules and pattern-matching exploits basic patterns over a variety of structures, such as text strings, part-of-speech tags, semantic pairs, and dictionary entries [73]. Patterns are easily recognized by humans and can be expressed directly using special purpose representation languages such as regular expressions [74]. Regular expressions are effective when the structure of the text and the tokens are consistent, but tend to be one-off methods tailored to the extraction task. Hand coding of regular expressions can involve a complex and time consuming development effort that requires a priori knowledge of all possible patterns that represent the concept being sought. Machine learning techniques can be an effective method for automated knowledge acquisition [75]. Features extracted from the text, such as parts of speech and sentence length, are fed into a learning machine to assist in tasks downstream like word sense disambiguation. The primary disadvantage for machine learning used for IE is that it requires a labeled dataset for training. The primary means of performing IE is natural language processing (NLP). NLP research focuses on developing computational models for understanding natural language [76]. Using tools built around ontologies (controlled vocabularies) like SNOMED-CT [77] have enabled researchers to automate the capture of information in clinical narratives.
Mining patient electronic medical records can be useful for detecting patterns in patient care [78], patient treatment habits [79] and their results. Statistical text mining has been used to determine if patients suffer from comorbidities related to obesity [80] and smoking [81], as well as detecting fall-related injuries [82]. Regular expressions have been used to extract blood pressure values from progress notes [83]. NLP has been useful for extracting medical information such as principal diagnosis [81] and medication use [84] from clinical narratives. This has led to a better understanding of the conditions patients face and how to treat them [85]. Manual chart review for annotation has been used extensively and when appropriate rigor is applied, the information extracted is very reliable and is often used as the reference standard to evaluate IE Systems.
Current Study
In this study, our first objective was to develop an annotation schema and a training set of annotated notes to support the future development of a natural language process (NLP) system to automatically extract employment information reliably from the EHR. The second objective of our study was to determine if information about employment status, goals, and work-related challenges reported by service members and Veterans with mTBI and post-deployment stress are identifiable in selected notes available in the Electronic Health Record (EHR). While the VA EHR contains information about employment, much of it is stored in text notes and is not easily extracted or summarized. Further, there is not a clear understanding of the reintegration challenges faced by Veterans using Veterans Health Administration (VHA) services and there are no available clinical outcomes readily available through VHA administrative databases [3]. The results of this study are intended to provide a foundation for future prospective studies of the measurement of unemployment of service members and Veterans with mTBI.
Methods
Ethics Statement
The Institutional Review Boards at the Veterans Health Administration's Office of Research and Development (VHA ORD) and the University of South Florida approved this study protocol and granted waivers of individual consent based on the absence of individually identifying data. Individual patient identifiers from the EHR were replaced with ID numbers. The Department of Veterans Affairs provided a waiver of HIPAA authorization for research conducted in this study.
Electronic Health Record Data
In order to investigate employment information in the EHR, we used a convenience sample of service members and Veterans with TBI who participated in the PREP program to examine a variety of sociodemographic and clinical factors that may affect return-to-work (RTW) or other productive activity. In the EHR, progress notes are entered directly by clinicians providing service and stored as text-based records in the database. Progress notes are automatically titled within the system based on the service or provider writing the note (e.g., “Nursing Note,” “Psychology Note,” “Social Work Inpatient Note”).
For this study, we reviewed 16 example notes from all 369 unique note titles available for the cohort and chose six note titles that include the targeted information. Veterans' employment information was extracted from text in the EHR from several note titles including the TBI/Polytrauma Rehabilitation/Reintegration Plan of Care notes. Several elements are included in TBI Care Plans including documentation of patient and family goals and Veteran employment status. Due to previous study findings reporting deficiencies in the TBI Care Plans [3], [51], five additional note titles were reviewed in this study including a) Social Work (SW) Psychosocial, b) Physical Medicine and Rehabilitation Service (PM&RS) Vocational Rehab Consultation, c) PM&RS Vocational Rehabilitation Testing, d) Compensated Work Therapy (CWT) Initial Assessment, and e) OIF/OEF SW Psychosocial notes. These note titles were selected because they contain rich employment information. Note titles reviewed and selected are displayed in Table 1. Specifically, information on employment status, work history, educational history, and future paid and unpaid vocational goals were annotated and extracted from all six of the aforementioned text notes. Patient and clinician commentary about readiness and ability to return to work were also extracted and annotated.
Participants/Sample Selection
The target population for this pilot study was taken from a larger project and included a cohort of service members and Veterans diagnosed with TBI and post-deployment stress that 1) participated in the PREP program between 2009 and 2012 and 2) had at least one TBI Plan of Care note.
Research Design/Procedure
In this retrospective cohort study, data from our cohort were extracted from the VHA EHR. Structured administrative data were used to describe demographic information on age, marital status, gender, and ethnicity of the cohort. Also, a chart review was performed to extract additional information from our cohort including other demographic information; work history; employment status, goals, and vocational interests; the number of symptoms associated with mTBI and post-deployment stress; and clinical and sociodemographic factors associated with employment status.
EHR Chart Review
A chart review was performed on all six aforementioned note titles. In this study, an annotation schema and a training set of annotated notes was generated to support the future development of a natural language processing (NLP) system. Such systems can reliably extract information related to returning to work. Information that can be extracted from the six aforementioned note titles include: vocational status, work history, educational history, driving status, and future paid and unpaid vocational goals The annotation schema used in the chart review was generated and iteratively refined based on insight from a clinical expert and by reviewing an initial batch of approximately 40 notes. Once the annotation schema was finalized, batches of approximately 50 notes were chart reviewed. The process for EHR chart review and annotation is shown in Fig. 1. The annotation software Knowtator [86] was used to chart review all of the notes. Knowtator has a unique advantage over other annotation tools due to its ability to define complex annotation schemas (e.g. schemas which have constrained relationships between entity types) [86].
As part of the chart review, the highest level of education and highest military rank attained were annotated, along with number of unique injuries, location of injury (“in theater” or “stateside”), number of unique deployments, and exposure to blasts (dichotomized as “yes” or “no”). Information about injury severity was collected from the TBI Plan of Care initial note (mild, moderate, or severe) and EHR text information about LOC or PTA was annotated if this information was found (dichotomized as “yes” or “no”). Employment status was classified based on the most current note reviewed. If the service member or Veteran reported being on active duty and unemployed, they were grouped into the active duty category. Reported symptoms were grouped into cognitive, physical, and emotional/behavioral categories as defined at the DOD Force Health Protection and Readiness Conference meeting held in 2007. The presence of specific reports of PTSD, chronic pain, headaches, sensory impairments, and sleep disturbances were also annotated.
Data Analysis
Data analysis for this project was conducted on the VA Informatics and Computing Infrastructure (VINCI), a secure, virtual computing environment developed through a partnership between the VA Office of Information Technology (OI&T) and the Veterans Health Administration's Office of Research and Development (VHA ORD). VINCI is part of the Transformation for the 21st Century Initiative #13 to provide researchers a nation-wide view of high-value VA patient data. While VINCI brings together data sources and provides the analytical environment for performing studies, VHA National Data Services (NDS) authorizes research access to patient data. New research projects are granted access to snapshots of data that can be updated as needed. In addition to secure data storage, VINCI includes a cluster of servers set aside for tasks like analysis, data processing, and extracting information from text. This allows VA researchers to have access to the data and the applications needed to select, transform, and analyze patient data in a central and secure location accessible from the VA intranet.
Results
Descriptive Information for the Sample Population
Veteran Demographics.
There were 177 Veterans and service members who were enrolled in PREP during the study period (inclusion criterion #1). Of those, 128 Veterans also had a TBI Plan of Care note (inclusion criterion #2). In this pilot study, all of the targeted note titles for 60 Veterans/service members, (n = 200 notes) were reviewed and annotated. Targeted note titles reviewed are displayed in Table 1.
The sample consisted of 60 Veterans and service members (cohort) diagnosed with TBI. Basic demographic data is displayed in Table 2. Most of the cohort in this sample was between the ages of 28 and 32 years (42.0%). The majority of the sample was male (96.7%), White (76.7%), and married (50.0%). In addition, 53.0% of the sample reported completing some college courses but were unable to finish their degree.
Veteran and Service Member TBI Injury Characteristics and Symptoms Reported.
Many Veterans reported two or more deployments (41.7%), most sustained mild injuries (65.0%), with the most common cause reported being due to a blast (78.3%) in theatre (89.2%). Nearly half of all the Veterans sampled reported LOC but a much smaller proportion reported PTA (55.0% and 13.3% respectively). Injury characteristics and symptoms reported are shown in Table 3 and indicate that there are many commonly reported areas in which cohort members with TBI experience symptoms. Groupings indicate that physical symptoms were reported most frequently (43.9%), followed by emotional/behavioral symptoms (29.8%), and finally cognitive problems (26.3%). Specifically, out of the 60 cohort members, a majority reported PTSD (88.3%), chronic pain (88.3%), headaches (86.67%), and sleep disturbances (76.67%).
Veteran Military and Employment Descriptions.
Specific military and employment information including rank, military occupation specialty (MOS), work history, employment goals, vocational interests, and driving ability is displayed in Table 4. A sizeable portion of the sample had a military rank of E5 (46.7%) and a military occupational specialty of infantry (36.7%). Further, 75.0% of the cohort reported history of civilian work. Many members of the cohort reported future goals of paid employment (65.0%) and returning to school (60.0%). Driving information was found for each member of the cohort. Over half (68.0%) of these individuals were able to drive. Results from the CareerScope assessment profile and associated interest inventory clusters were found in 32% of the sample.
Discussion
Most individuals with mTBI return to work within a few months post-injury, but those who continue to experience persistent symptoms and functional impairments months' to years' post-injury may encounter limitations in work and other areas of social functioning [20], [21], [29]. Previous research has found that multiple variables need to be considered when studying employment following mTBI in civilian populations as well as military populations [10], [20], [26], [27], [38]. In this pilot study, we explored whether the VHA EHR contains information that could provide insights regarding the levels of unemployment among veterans and service members with mTBI. We confirmed that limited administrative (structured or coded) data in the EHR are available on the employment status of Veterans and service members with mTBI. However, data extracted in this study reveal that information about employment and factors that may affect return-to-work are available in text notes in the EHR. Currently, this information is not readily available for review by clinicians for interdisciplinary rehabilitation planning purposes. Having this information available may assist clinicians concurrently address health-related and vocational needs, which may facilitate successful adjustment and reintegration for Veterans and service members [71].
Through annotation and data extraction, we found that service members and Veterans who participated in the PREP rehabilitation program were mostly Caucasian males between the ages of 28 and 32 who reported being unemployed and noted a variety of symptoms and post-deployment stress. Comparable to many previous research findings [10], [32], [33], [39], we found that many service members and Veterans reported mTBI (65%), PTSD (83%), chronic pain (88%), headaches (87%), sleep disturbances (77%), and unemployment (75%). It is important to note that Veterans and service members in this study completed the in-patient PREP program, which specializes in the evaluation and treatment of complexities associated with mTBI and post-deployment adjustment difficulties. Those who enroll in the PREP must be available for therapies for several hours a day over a period of a few weeks. Due to the qualifications and rigorous nature of the PREP program, many of these individuals may have been unemployed when they enrolled into the program.
The majority of the participants in our sample reported being deployed with a smaller number noting two or more deployments. A large proportion of those deployed reported exposure to blasts, which is an area of research that has yet to be fully elucidated. Multiple complex relationships have been found between multiple deployments and blast exposures [40], [41]. There appears to be a strong correlation between exposure to a blast and an individual's physical, cognitive, and emotional/psychological functioning. Thus, those exposed to a blast may experience various problems that could affect their ability to obtain steady employment.
We also found that many service members and Veterans reported having completed some college coursework, but did not obtain a degree (53%). Many of these individuals had a future goal of returning to school/training (60%) and then entering into the paid workforce (65%). These findings substantiate previous research that military advocates, educators, and researchers are concerned about post-secondary education attainment, and that complex relationships exist between TBI, PTSD, and educational attainment [42], [43]. Although it was beyond the scope of this study, future research should investigate reasons why service members and Veterans may be less likely to complete their degrees.
In this study, we were able to find the driving status of all Veterans and service members in the EHR records we reviewed. In our sample, over half of our participants were able to drive. Previous research has identified access to reliable transportation to be the biggest environmental barrier experienced by persons with TBI [44]. Self-reported driving problems are common with OIF/OEF Veterans [56]. Further, those who do not resume driving are at risk for poor community reintegration including occupational reintegration [45], [46], [67]. It is possible that service members and Veterans in our sample had access to reliable transportation, but adjustment to driving in civilian environments or other comorbid factors associated with TBI [56] may contribute to the high unemployment rates found in this study.
Finally, we found that approximately 19 (32%) of the participants in this study had CareerScope [47] assessment profiles which include interest inventory clusters. The interest inventory measures and identifies careers of interest that correspond to the US Department of Labor's interest areas. Of the twelve broad interest areas, the most frequently reported interest areas were 04-Protective, 02-Scientific, 05-Mechanical, 11-Leading/Influencing, 12-Physical/Performing, and 07-Business Detail. These findings were not surprising since these areas of interest mirror many of the Military Occupational Specialties (MOS) reported in the EHR. Research has shown that a number of factors have been associated with successful return to employment [35], [65] including premorbid employment status [16], [17], [66]. Further, the nature of pre-injury employment has been recognized as a significant pre-injury factor [57].
These findings may indicate that vocational rehabilitation counselors and other rehabilitation service providers need to heighten their awareness of the specific work histories, career interests, and needs of service members and veterans with TBI entering or reentering the civilian workforce or post-secondary education [48]–[50].
Limitations of the Study
Limitations of this pilot study include a small sample size consisting of mostly Caucasian male service members and Veterans who all participated in the PREP program. How these findings generalize to other racial groups and females is unknown. Additionally, because the PREP program is in-patient, many of these service members and Veterans may have been unemployed for extended periods. However, many service members and Veterans who experience mTBI may struggle with unemployment. Additionally, the present study utilized records from a large VA hospital indicating that results may be more representative of the Veteran population due to the high volume of clients at this facility.
Due to the design of this study, it is difficult to determine whether each barrier identified in this study may be a cause, effect, or incidental correlate of employment status. As stated previously, the PREP program is intensive and many of the individuals are unemployed. Perhaps if researchers are able to determine specific declines in functioning following deployment, it can be linked to the injuries they may have received such as mTBI. It is possible that this would allow for a greater proportion of needs to be met and a higher likelihood of return to work for veterans with mTBI. An additional limitation is that only a single annotator performed the chart review. Future studies should include two annotators and an adjudicator to provide additional chart review reliability.
Conclusions/Future Directions
Employment information on a unique sample of service members and Veterans who completed the TBI PREP in-patent rehabilitation program was reviewed in order to 1) develop an annotation schema and a training set of annotated documents to perform information extraction (IE) on text notes available in the VA EHR, and 2) to determine if information about employment status, goals and work-related challenges are identifiable in the EHR. In our study, we found that limited administrative (structured or coded) data in the EHR are available on the employment status of Veterans and service members with mTBI. Further we found that data about employment and factors that may affect return-to-work are identifiable and available in clinical text notes in the EHR.
Findings in our pilot study may improve our understanding of the myriad of employment challenges experienced by these individuals. Study findings add to our current knowledge and understanding about the diverse reintegration challenges experienced by service members and Veterans with mTBI. Overall, we found that service members and Veterans with mTBI experience high rates of unemployment, while many of these individuals have future goals of returning to school and/or paid employment. These findings suggest that service members and Veterans with mTBI experience a number of symptoms that may affect their ability to reintegrate into the workforce or post-secondary education. Concurrently addressing health-related, vocational, and independence needs may help alleviate gaps in treatment planning and service delivery to Veterans experiencing post-deployment stress and barriers to community reintegration [71]. It is important for clinical team members have access to current and accessible information on vocational status, goals, and driving ability. This practical information may be used to guide specialized evaluations and inform treatment planning to help facilitate reintegration.
Larger-scale retrospective and prospective studies are needed to advance our understanding of the unique reintegration needs of a population of service members and Veterans that is not fully elucidated. Natural Language Processing (NLP) may be a system to consider using to automatically extract great amounts of text-based employment information from a large sample, which will assist clinicians, vocational service providers, and researchers. Future research is needed to further our understanding of the reintegration needs and provide information on ways to best support Veterans as they enter or reenter the workforce [71]. Increased awareness of the challenges experienced by service members and Veterans with TBI may aid in an improved understanding of symptoms that are important to target for future interventions that may improve community re-integration outcomes.
Acknowledgments
The authors would like to thank the following individuals for their work, support, and contribution to the development of this manuscript: Lisa Ottomanelli, PhD, Jay Jarman, PhD, Bryan Merritt, MD, Philip Foulis, MD, MPH, and Bridget Hahm, MA, MPH. Finally, the authors would like to thank Sarah Ehlke, MA, Graduate Student, University of Wilmington, North Carolina, Melanie Sosinski, MA, MSW, School of Social Work, University of South Florida, and Cristina M. Gonzalez, BA, CRC, Department of Rehabilitation and Mental Health Counseling, for their assistance with preparing this manuscript for publication.
Disclaimer: The views expressed in this work are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the US Government.
Author Contributions
Conceived and designed the experiments: DF CJDA SLL JAM. Performed the experiments: DF CJDA SLL JAM. Analyzed the data: DF CJDA SLL JAM. Contributed reagents/materials/analysis tools: DF CJDA SLL JAM. Wrote the paper: DF CJDA SLL JM TK JAM.
References
- 1. Lew HL, Cifu DX, Sigford B, Scott S, Sayer N, et al. (2007) Team approach to diagnosis and management of traumatic brain injury and its comorbidities. J Rehabil Res Dev 44:vii–xi.
- 2. Okie S (2006) Reconstructing lives – a tale of two soldiers. N Engl J Med 355:2609–2615.
- 3.
Sayer N (2012) Department of Veterans Affairs Quality Enhancement Research Initiative (QUERI) Polytrauma/Blast-Related Injuries QUERI Center. Strategic plan FY 2013 through FY 2015. Available: http://www.queri.research.va.gov/about/strategic_plans/ptbri.pdf. Accessed 2014 Jan 7.
- 4. Department of Veteran Affairs (2009) VA/DoD clinical practice guideline for management of concussion/mild traumatic brain injury. J Rehabil Res Dev 46:CP1–68.
- 5.
Borg J, Holm L, Peloso PM, Cassidy JD, Carroll LJ, et al.. (2004) Non-surgical intervention and cost for mild traumatic brain injury: results of the WHO collaborating centre task force on mild traumatic brain injury. J Rehabil Med Suppl 43 76–83.
- 6. Hoge CW, Goldberg HM, Castro CA (2009) Care of war Veterans with mild traumatic brain injury-flawed perspectives. N Engl J Med 360:1588–1591.
- 7. American Congress of Rehabilitation Medicine (ACRM) (1993) Definition of mild traumatic head injury. J Head Trauma Rehabil. 8:86–87.
- 8. Cernich A, Chandler L, Scherdell T, Kurtz S (2012) Assessment of co-occurring disorders in veterans diagnosed with traumatic brain injury. J Head Trauma Rehabil 27:253–260.
- 9. Ommaya AK, Dannenberg AL, Salazar AM (1996) Causation, incidence, and costs of traumatic brain injury in the US military medical system. J Trauma 40:211–217.
- 10. Ettenhofer ML, Melrose RJ, Delawalla Z, Castellon SA, Okonek A (2012) Correlates of functional status among OEF/OIF veterans with a history of traumatic brain injury. Military medicine 177:1272–1278.
- 11. Hoge CW, McGurk D, Thomas JL, Cox AL, Engel CC, et al. (2008) Mild traumatic brain injury in US Soldiers returning from Iraq. N Engl J Med 358:453–463.
- 12. Okie S (2005) Traumatic brain injury in the war zone. N Engl J Med 352:2043–2047.
- 13.
Schell TL, Marshall GN (2008) Survey of individuals previously deployed for OEF/OIF. Invisible wounds of war: Psychological and cognitive injuries, their consequences, and services to assist recovery. 87–113. Available: http://www.rand.org/content/dam/rand/pubs/monographs/2008/RAND_MG720.pdf. Accessed 2014 Oct 20.
- 14. Terrio H, Brenner LA, Ivins BJ, Cho JM, Helmick K, et al. (2009) Traumatic brain injury screening: Preliminary findings in a US army brigade combat team. J Head Trauma Rehabil 24:14–23.
- 15.
The Office of Patient Care Services (2005) Polytrauma rehabilitation procedures. In: VHA handbook 1172.1. Washington, DC: Department of Veterans Affairs.
- 16. Sayer NA, Chiros CE, Sigford B, Scott S, Clothier B, et al. (2008) Characteristics and rehabilitation outcomes among patients with blast and other injuries sustained during the Global War on Terror. Arch Phys Med Rehabil 89:163–170.
- 17. Belanger HG, Uomoto JM, Vanderploeg RD (2009) The Veterans health administration system of care for mild traumatic brain injury: Costs, Benefits, and Controversies. J Head Trauma Rehabil 24:4–13.
- 18. Andelic N, Stevens LF, Sigurdardottir S, Arango-Lasprilla JC, Roe C (2012) Associations between disability and employment 1 year after traumatic brain injury in a working age population. Brain Inj 26:261–269.
- 19. Ponsford JL, Downing MG, Olver J, Ponsford M, Acher R, et al. (2014) Longitudinal follow-up of patients with traumatic brain injury: Outcome at two, five, and ten years post-injury. J Neurotrauma 31:64–77.
- 20. Cancelliere C, Kristman VL, Cassidy JD, Hincapie CA, Cote P, et al. (2014) Systematic review of return to work after mild traumatic brain injury: Results of the international collaboration on mild traumatic brain injury prognosis. Arch Phys Med Rehabil 95:201–209.
- 21. Iverson GL (2005) Outcome from mild traumatic brain injury. Curr Opin Psychiatry 18:301–317.
- 22. Daggett VS, Bakas T, Buelow J, Habermann B, Murray LL (2013) Needs and concerns of male combat veterans with mild traumatic brain injury. J Rehabil Res Dev 50:327–340.
- 23. Yasuda S, Wehman P, Targett P, Cifu D, West M (2001) Return to work for persons with traumatic brain injury. Am J Phys Med Rehabil 80:852–864.
- 24.
Kupersmith J, Ommaya AK, Selzer ME, Ruff RL, Lew HL (2009) Guest editorial: Traumatic brain injury research state-of-the-art conference. J Rehabil Res Dev 46:: xvii-xviii.
- 25. Lang KP, Veazey-Morris K, Andrasik F (2014) Exploring the role of insomnia in the relation between PTSD and pain in veterans with polytrauma injuries. J Head Trauma Rehabil 29:44–53.
- 26. O'Neil M, Carlson K, Storzbach D, Brenner LA, Freeman M, et al. (2014) Factors associated with mild traumatic brain injury in veterans and military personnel: A systematic review. J Int Neuropsychol Soc 5:1–13.
- 27. Saltychev M, Eskola M, Tenovuo O, Laimi K (2013) Return to work after traumatic brain injury: Systematic review. Brain Inj 27:1516–1527.
- 28. Sayer NA (2012) Traumatic brain injury and its neuropsychiatric sequelae in war veterans. Annu Rev Med 63:405–419.
- 29.
Waljas M, Iverson G, Lange R, Liimatainen S, Hartikainen KM, et al.. (2013) Return to work following mild traumatic brain injury. J Head Trauma Rehabil (In press).
- 30. Zeitzer JM, Friedman L, O'Hara R (2009) Insomnia in the context of traumatic brain injury. J Rehabil Res Dev 46:827–836.
- 31. Prigerson HG, Maciejewski PK, Rosenheck RA (2002) Population attributable fractions of psychiatric disorders and behavioral outcomes associated with combat exposure among US men. Am J Public Health 92:59–63.
- 32. Cohen SI, Suri P, Amick MM, Yan K (2013) Clinical and demographic factors associated with employment status in US military veterans returning from Iraq and Afghanistan. Work 44:213–219.
- 33. Twamley EW, Baker DG, Norman SB, Pittman JOE, Lohr JB, et al. (2013) Veterans health administration vocational services for Operation Iraqi Freedom/Operation Enduring Freedom veterans with mental health conditions. J Rehabil Res Dev 50:663–670.
- 34. Tyerman A (2012) Vocational rehabilitation after traumatic brain injury: models and services. NeuroRehabilitation 31:51–62.
- 35. Shames J, Treger I, Ring H, Giaquinto S (2007) Return to work following traumatic brain injury: trends and challenges. Disabil Rehabil 29:1387–1395.
- 36. Cicerone K, Azulay J (2007) Perceived self-efficacy and life satisfaction after traumatic brain injury. J Head Trauma Rehabil 22:257–266.
- 37. Tsaousides T, Warshowsky A, Ashman TA, Cantor JB, Spielman L, et al. (2009) The relationship between employment-related self-efficacy and quality of life following traumatic brain injury. Rehabil Psychol 54:299–305.
- 38. Stergiou-Kita M, Dawson DR, Rappolt SG (2011) An integrated review of the processes and factors relevant to vocational evaluation following traumatic brain injury. J Occup Rehabil 21:374–394.
- 39. Gubata ME, Packnett ER, Blandford CD, Piccirillo AL, Niebuhr DW, et al. (2014) Trends in the epidemiology of disability related to traumatic brain injury in the US Army and Marine Corps: 2005 to 2010. J Head Trauma Rehabil 29:65–75.
- 40.
National Research Council (2014) Gulf War and Health, Volume 9: Long-Term Effects of Blast Exposures. Washington, DC: The National Academies Press.
- 41. Vanderploeg R, Belanger HG, Horner RD, Spehar AM, Powell-Cope G, et al. (2012) Health outcomes associated with military deployment: Mild traumatic brain injury, blast, trauma, and combat associations in the Florida National Guard. Arch Phys Med Rehabil 93:1887–1895.
- 42. Ness BM, Vroman K (2014) Preliminary examination of the impact of traumatic brain injury and posttraumatic stress disorder on self-regulated learning and academic achievement among military service members enrolled in postsecondary education. J Head Trauma Rehabil 29:33–43.
- 43. Madaus J, Miller W, Vance M (2009) Veterans with disabilities in postsecondary education. J Postsecondary Educ Disabil 22:10–17.
- 44. Rapport LJ, Bryer RC, Hanks RA (2008) Driving and community integration after traumatic brain injury. Arch Phys Med Rehabil 89:922–930.
- 45. Novack TA, Labbe D, Grote M, Carlson N, Sherer M, et al. (2010) Return to driving within 5 years of moderate–severe traumatic brain injury. Brain Inj 24:464–471.
- 46. Rapport LJ, Hanks RA, Bryer R (2006) Barriers to driving and community integration after traumatic brain injury. J Head Trauma Rehabil 21:34–44.
- 47.
Vocational Rehabilitation Institute (2011) Careerscope v10. Available: http://www.vri.org/. Accessed 2014 Jan 7.
- 48. Frain MP, Bishop M, Bethel M (2010) A roadmap for rehabilitation counseling to serve military veterans with disabilities. J Rehabil 76:13–21.
- 49. Frain MP, Bishop M, Tschopp M, Tansey T, Agonis J (2011) Training needs of rehabilitation counselors concerning veterans with disabilities. Voc Eval and Work Adj Assoc 38:34–44.
- 50. Clemens EV, Milsom AS (2008) Enlisted service members' transition into the civilian world of work: A cognitive information processing approach. Career Dev Q 56:246–256.
- 51.
Department of Veterans Affairs (2010) Physical medicine and rehabilitation individualized rehabilitation and community reintegration care plan. Veterans Health Administration Handbook 1172.04. Washington, DC. Available: http://www1.va.gov/vhapublications/ViewPublication.asp?pub_ID=2229. Accessed 2014 Apr 26.
- 52.
Defense and Veterans Brain Injury Center (2013) DOD worldwide TBI numbers. Department of Defense Worldwide TBI Numbers Available: http://dvbic.dcoe.mil/dod-worldwide-numbers-tbi. Accessed 2014 Jan 27.
- 53. Kretzmer T, Ruiz E (2013) Post-deployment rehabilitation and evaluation program (PREP). Military TBI Case Management Quarterly Newsletter 4 (1):8–9 Available: http://dvbic.dcoe.mil/sites/default/files/uploads/DCoE_QRTLY_TBI-CMNews_Winter_2013-03-19_1215.pdf Accessed 2014 May 13..
- 54. Sigaki CL, Johnstone B, Schopp LH (2009) Financial and vocational outcomes 2 years after traumatic brain injury. Disabil Rehabil 31(6):484–9.
- 55. Benshoff J, Robertson S, Davis SJ, Koch S (2008) Professional identity and the CORE standards. Rehabil Res Poli Educ 22:227–235.
- 56.
Lew HL, Kraft M, Pogoda TK, Amick MM, Woods P, et al.. (2011) Prevalence and characteristics of driving difficulties in operation iraqi freedom/operation enduring freedom combat returnees. J Rehabil Res Dev ((8):): 913.
- 57.
Ponsford J, Spitz G (2014). Stability of employment over the first three years following traumatic brain injury. J Head Trauma Rehabil: e pub ahead of print. Available: http://journals.lww.com/headtraumarehab/Abstract/publishahead/Stability_of_Employment_Over_the_First_3_Years.99771.aspx Accessed 2014 May 13.
- 58.
Mild traumatic brain injury screening and evaluation implemented for OEF/OIF Veterans, but challenges remain [Internet]. Washington (DC): United States Government Accountability Office; 2008 Feb 8 [cited 2008 Apr 7]. Available from: http://www.gao.gov/new.items/d08276.pdf. Accessed 2014 May 13.
- 59. McColl MA, Carlson P, Johnston J, Minnes P, Shue K (1998) The definition of community integration: perspectives of people with brain injuries. Brain Inj 12(1):15–30.
- 60. Salter K, Foley N, Jutai J, Bayley M, Teasell R (2008) Assessment of community integration following traumatic brain injury. Brain Inj 22(11):820–835.
- 61. Dillahunt-Aspillaga C, Agonis-Frain J, Hanson A, Frain M, Sosinski M, et al. (2014) Applying a Resiliency Model to Community Reintegration and Needs in Families with Traumatic Brain Injury: Implications for Rehabilitation Counselors. J of Appl Rehabil Couns 45(1):25–36.
- 62. Dillahunt-Aspillaga C, Jorgensen-Smith T, Ehlke S, Sosinski M, Monroe D, et al. (2013) Traumatic Brain Injury: Unmet Support Needs of Caregivers and Families in Florida. PloS One 8:e82896
- 63. Carlson KF, Shannon KM, Meis LA, Greer N, MacDonald R, et al. (2011) Prevalence, assessment, and treatment of mild traumatic brain injury and posttraumatic stress disorder: A systematic review of the evidence. J Head Trauma Rehabil 26(2):103–115.
- 64.
Institute of Medicine (2008) Gulf War and Health, Volume 7: Long-term Consequences of Traumatic Brain Injury. Washington, DC: The National Academies Press. Available: http://www.iom.edu/Reports/2008/Gulf-War-and-Health-Volume-7-Long-term-Consequences-of-Traumatic-Brain-Injury.aspx. Accessed 2014 Oct 29.
- 65. Ownsworth T, McKenna K (2004) Investigation of factors related to employment outcome following traumatic brain injury: a critical review and conceptual model. Disabil Rehabil 26:765–784.
- 66. Franulic A, Carbonell CG, Pinto P, Sepulveda I (2004) Psychosocial adjustment and employment outcome 2, 5 and 10 years after TBI. Brain Inj 18:119–129.
- 67. Hannold EM, Classen S, Winter S, Lanford DN, Levy CE (2013) Exploratory pilot study of driving perceptions among OIF/OEF veterans with mTBI and PTSD. J Rehabil Res Dev 10:1315–1330.
- 68. Schonberger M, Ponsford J, Olver J, Ponsford M, Wirtz M (2011) Prediction of functional and employment outcome 1 year after traumatic brain injury: a structural equation modelling approach. J Neurol Neurosurg Psychiatry 82:936–941.
- 69. Walker WC, Marwitz JH, Kreutzer JS, Hart T, Novack TA (2006) Occupational categories and return to work after traumatic brain injury: a multicenter study. Arch Phys Med Rehabil 87:1576–1582.
- 70. Ponsford J (2013) Factors contributing to outcome following traumatic brain injury. NeuroReahabilitation 32:803–815.
- 71.
Pogoda T, Stolzmann K, Iverson K, Baker E, Krengel M, et al.. (2014) Associations between traumatic brain injury, suspected psychiatric conditions, and unemployment in Operation Enduring Freedom/Operation Iraqi Freedom veterans. J Head Trauma Rehabil e pub ahead of print. Available: http://journals.lww.com/headtraumarehab/Abstract/publishahead/Associations_Between_Traumatic_Brain_Injury.99738.aspx. Accessed 2014 Oct 21.
- 72.
DeJong G (1982) An overview of the FRUMP system. Strategies for natural language processing. pp. 149–176.
- 73.
Pakhomov S, Buntrock J, Duffy P (2005) High throughput modularized NLP system for clinical text. Association for Computational Linguistics. 25–28 doi:https://doi.org/10.3115/1225753.1225760.
- 74. Thompson K (1968) Programming Techniques: Regular expression search algorithm. Communications of the ACM 11:419–422.
- 75. Lehnert W, Soderland S, Aronow D, Feng F, Shmueli A (1995) Inductive text classification for medical applications. J Exp Theor Artif Intell 7:49–49.
- 76.
Hayes PJ, Carbonell J (1987) Natural Language Understanding. Encyclopedia of Artificial Intelligence. pp. 660–677.
- 77. Spackman KA, Campbell KE, Cote RA (1997) SNOMED RT: a reference terminology for health care. Proc AMIA Annu Fall Symp 1997:640–444.
- 78. Pakhomov S, Hanson P, Bjornsen S, Smith S (2008) Automatic Classification of Foot Examination Findings Using Clinical Notes and Machine Learning. J Am Med Inform Assoc 15:198–202.
- 79. Rao RB, Krishnan S, Niculescu RS (2006) Data mining for improved cardiac care. SIGKDD Explor News 8:3–10 Available: http://www.kdd.org/sites/default/files/issues/8-1-2006-06/1-Rao.pdf Accessed 2014 Oct 29..
- 80. Guillen R (2009) Identifying Obesity and Co-morbidities from Medical Records. Proceedings of the i2b2 Workshop on Challenges in Natural Language Processing for Clinical Data 2008. pp. 868.
- 81. Zeng Q, Goryachev S, Weiss S, Sordo M, Murphy S, et al. (2006) Extracting principal diagnosis, co-morbidity and smoking status for asthma research: evaluation of a natural language processing system. BMC Med Inform Decis Mak 6:30
- 82.
Chiarini-Tremblay M, Berndt DJ, Foulis P, Luther S (2005) Utilizing text mining techniques to identify fall related injuries. AMCIS 2005 Proceedings pp. 109. Available: http://aisel.aisnet.org/amcis2005/109. Accessed 2014 Oct 28.
- 83. Turchin A, Kolatkar NS, Grant RW, Makhni EC, Pendergrass ML, et al. (2006) Using regular expressions to abstract blood pressure and treatment intensification information from the text of physician notes. J Am Med Inform Assoc 13:691–695.
- 84. Xu H, Stenner S, Doan S, Johnson KB, Waitman LR, et al. (2010) MedEx: a medication information extraction system for clinical narratives. J Am Med Inform Assoc 17:19–24
- 85.
Cerrito P, Cerrito J (2006) Data and text mining the electronic medical record to improve care and to lower costs. Available: http://www2.sas.com/proceedings/sugi31/077-31.pdf. Accessed 2014 Oct 28.
- 86.
Ogren PV (2006) Knowtator: a protégé plug-in for annotated corpus construction. Proceedings of the 2006 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology. New York: Association for Computational Linguistics. pp.273–275. Available: http://dx.doi.org/10.3115/1225785.1225791 Accessed 2014 Oct 29.