2.2.1 Business understanding.

This initial phase is focused on understanding the objectives and problem definition in terms of medical curriculum innovation. The goal is to detect outlying and overlapping areas in the General Medicine study program, using data/text mining techniques. The results of the data mining might then be the subject of an evaluation provided by senior curriculum experts. The mentioned goal can be reformulated in terms of data mining issues, such that the task is to investigate the properties of a graph obtained from a dissimilarity matrix over a collection of documents representing individual parts of the curriculum. Our approach is based on Social Network Analysis (SNA), and it was chosen since it can naturally deal with graph-like/network-like data and encompasses centrality notions. First of all, we generate the entire similarity graph based on the similarities of individual medical disciplines as represented by their textual descriptions; specifically, the cosine distance is used. Next, we compute various centrality measures to model the interconnection and remoteness of the nodes (i.e., medical disciplines, in our particular case) in the graph. We are interested in uncovering densely connected subgraphs called communities, as these might represent the most crucial and important parts of a curriculum, in terms of its global harmonization. Based on the community detection, we are able to explore novel and useful information about the structure of the General Medicine study program. We can identify the nodes with extreme (i.e., the highest/the lowest) values of selected centrality measures, which may indicate that these medical disciplines belong to an essential part of the curriculum, or, conversely, it may cause undesirable preference of mentioned disciplines that in fact cannot be identified by manual human inspection. A similar approach has been proposed by Trigo and Brazdil on mining affinity relationships within groups of researchers [31].

2.2.2 Data understanding.

We have already built the database for collecting medical curriculum data in a parametric format. It covers all elements pertaining to global curriculum harmonization, including detailed metadata specification down to the level of LUs, linkages to Los, and standardized biomedical vocabulary. The organization of the metadata and its linkages is provided in the curriculum model, which can be implemented without any restrictions within any relational database technology. In order to effectively process curricular metadata stored in our system, we used aggregation and filtering to get the final dataset from the primary data, followed by application of particular KDD methods for exploring new knowledge (see Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Phases of the CRISP-DM reference model.

https://doi.org/10.1371/journal.pone.0143748.g003

2.2.3 Data preparation.

This step represents an important preprocessing step, which then allows the use of any of the data mining algorithms. All the stated content was created in the Czech and English languages. For the purpose of in-depth analysis of the present pilot, the English version was used, due to easier preprocessing. We identified a set of information-rich attributes, in collaboration with curriculum designers and guarantors, which are suitable for exploring the medical curricula. In our case, for each discipline a single plaintext file was generated, so that it contained merged contents of several attributes (see Table 1) describing LO and its affiliation to curriculum (i.e. link to LU, course, medical discipline, and section). For the purpose of analytical pilots, we processed only the attributes that were related to the particular LOs (i.e., sentences with action verbs, according to Bloom taxonomy) and to all of the descriptive indexes (i.e., grouped outcome, primary index, secondary index). For illustration, the input dataset could have been quantified by approximately 636 normative pages (containing 1,800 characters of text) or with 118,798 words.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Attributes of final dataset describing the medical curriculum.

https://doi.org/10.1371/journal.pone.0143748.t001

2.2.4 Modeling.

This phase uses modeling techniques (i.e., analytic methods, particularly data mining methods) on the dataset prepared in the previous phase. The entire text preprocessing was implemented with the use of the R software and its standard packages tm (a widely used package for text mining that allows us to deal with collections of textual data in a convenient way) and lsa (a standard package for latent semantic analysis). The collection of plaintext files that represent individual disciplines was loaded as a corpus (from tm package) in order to prepare their bag-of-word representation. A usual sequence of pre-processing issues was applied for each document in the corpus after tokenization: transformation to lowercase, stemming (using the Snowball system), punctuation removal, numbers removal, stop words removal, and whitespace stripping. The text preprocessing step was followed by the computation of two matrices. First, a document-term-matrix (DTM) based on tf-idf weighting [32] was generated. Consequently, a dissimilarity matrix was computed on the base of the cosine similarity over the vectors in the DTM. Values were rounded to two decimal digits, and values lower than a certain threshold were replaced by zeros, since extremely low similarities were considered irrelevant. This dissimilarity matrix naturally corresponds to an undirected graph with weighted edges. The dissimilarity matrix can be viewed as an adjacency matrix of this graph (where the nodes are the disciplines, the edges represent similarities between pairs of documents, and the weight of an edge corresponds with the similarity value); thus, we are able to use various SNA techniques to work with graph representation of data.

In our case study, we have concentrated on the problem of community finding and estimating the importance of each node by various centrality measures, which is an easy way to explore the real composition and intersections of individual disciplines in the curriculum. For community finding, the WalkTrap algorithm [33] was chosen, due to its low computing complexity and the fact that it is implemented within the igraph package. The algorithm is based on short random walks in the graph. Its main premise is that short random walks tend to stay in the same community [34]. Here, we experimented with different values of WalkTrap parameters, mainly with the length of the random walk. After uncovering the communities, the following three centrality measures were computed: closeness centrality, betweenness centrality, and Eigenvector centrality. The closeness centrality of a node is defined by the inverse of the average length of the shortest paths to all of the remaining nodes in the given graph. The (vertex/node) betweenness centrality is, in the simplest case, defined as the number of shortest paths that go through a given node. Eigenvector centrality is one way of computing the approximate importance of a given node. The idea behind this measure is that the centrality of each node is the sum of the centrality values of its neighboring nodes. More precisely, the Eigenvector centrality values correspond to the values of the first eigenvector of the adjacency matrix. In all cases, we will use versions of these measures for graphs with weighted edges, and all values are normalized. The short description of these measures is given below, and more details can be found in [35,36].

Closeness centrality: The closeness centrality of a node v in a graph G is defined by the inverse of the sum of the lengths of the shortest paths to/from all the other nodes in the graph G: (1) Where d(i, v) is the length of the shortest path from node i to node v. If there is no path between a pair of nodes, then the total number of nodes in the graph is used instead of the path length. Through the mentioned calculation, we obtain the so-called raw closeness of the node. To obtain the normalized closeness of the node v, we multiply the raw closeness by (n − 1), where |V(G)| = n, and we use this normalized version in subsequent calculations. Nodes (disciplines) with low values of closeness are those disciplines whose content is distant from other ones; thus, roughly said, they are independent of the others.

Betweenness centrality: In the simplest case (i.e., without edge weighting), the raw betweenness centrality of a node v corresponds with the number of shortest paths from all nodes to all others that go through the node in question: (2) Where g_ij is the total number of shortest paths going from node i to j, and g_ivj is the total number of all the shortest paths from node i to node j going through v. To get the normalized betweenness b_n(v) of the node v, we calculate , where . This definition can be extended for weighted networks. The nodes (disciplines) with high betweenness centrality are the ones that are the best for joining the students’ knowledge from different collections of disciplines (i.e., through intersubject transfer of knowledge).

Eigenvector centrality: Eigenvector centrality is one of the methods of computing approximate importance, and is a measure of the influence of a given node. The idea behind this measure is that the centrality of each node is the sum of the centrality values of its neighbor nodes. More preciously, the eigenvector centrality values correspond to the values of the first eigenvector of the adjacency matrix. The eigenvector centrality, in our case, models identification of important disciplines in the curriculum. All three of these measures were implemented within the igraph package.

2.2.5 Evaluation.

In this phase, a checking procedure was performed in order to assess whether the approach we used was appropriate, in terms of its usability in real curriculum management. The obtained results were organized and presented in a clear way for the curriculum designers and the committee experts. From the perspective of medical teachers, who are often authors of LUs and LOs, several clear benefits were identified: (i) transparent overview over the medical education in a particular HEI; (ii) detection of any outlying LUs; and (iii) continuous monitoring of the consistency of a curriculum in the never-ending process of innovation. An example evaluation of the achieved analytical results obtained from senior curriculum experts is given in 3.1.2.

2.2.6 Deployment.

Creation of the models is generally not the end of the project. The knowledge that was gained needs to be organized and presented in a way that the academics can use on a long-term basis. It often involves applying live models within a HEI’s decision making processes. Depending on the requirements, the deployment phase can be as simple as generating an analytical report or as complex as implementing a repeatable data mining process across the HEI. We have systematically tailored our generic process model, in accordance with a pre-defined context of medical curriculum exploration. In general, we aim to analyze and consolidate the experiences of this single pilot project towards a follow-up mapping issues in form advanced analysis of medical curriculum for future usage in comparable contexts. Basically, a global strategy for mapping consists of five steps: (i) analyze a specific field of study; (ii) remove any details not applicable to given field; (iii) add any details specific to given field; (iv) specialize (or instantiate) generic contents according to concrete characteristics of given field; and (v) possibly rename generic content to provide more explicit meaning in a given context, for the sake of clarity [37].

4.1.1 Medical curriculum innovations.

The Bergen ministerial conference of the Bologna Process in May 2005 discussed reforms to degree structures, credit transfer, quality assurance, and curricular development, which are transforming the European Higher Education Area. An outcomes-based approach is arguably best viewed as a fundamental building block of the Bologna education reforms, and brings greater transparency to higher education systems [38]. The process of medical curriculum innovation supported by a robust planning model, which is described here, is based on the well-established concept of a standardized definition of learning outcomes. One of its key features is its dynamism, namely the ability to easily manage any domain closely related to the medical curriculum through upgrades, and to absorb and incorporate all changes into the educational process. For teachers and faculty management, such an approach enables them to effectively administer education processes, to clearly see who teaches what and to what extent, to determine whether the lectures of individual teachers overlap thematically, to assess whether the overall teaching schedule is appropriate, or to decide whether a certain reform of the teaching schedule would be convenient and helpful. And for school managers, the presented tools will provide a practical view of the real content teaching. Further, it will also provide clear and comprehensible data about who teaches what and in what context, as well as information on the deficiencies and overlaps present in the curriculum. It is currently used by hundreds of senior teachers, curriculum designers, and professional guarantors, who use it for content definition and further inspection of the medical curriculum.

4.1.2 Medical curriculum mapping.

With respect to the achieved results, we decided to evaluate the curriculum in co-operation with senior guarantors (senior experts) of medical education at our university. The checking procedure is a mandatory phase of the CRISP-DM methodology, in order to make sure that the given objectives are fulfilled. Expert opinion helps to indicate whether the medical disciplines are balanced or unbalanced, and to identify weak points and shortcomings in terms of inconvenient interdisciplinary relations. In total, three experts were chosen, based on specific criteria: at least 10 years of experience in medical education (educational and scientific background), at least 15 years of professional clinical experience (following the award of the diploma), hold a managerial position in the medical faculty at a scientific level (i.e., at minimum, associate professor level), proven professional experience at a strategic level in medical education, thorough knowledge and proven experience in the principles of developing, applying, and implementing curriculum in medical education, and experience in managing educational projects/program concerning medical pre gradual education. All senior guarantors were familiar with the project activities, but were not deeply involved in the technical evaluation of the curriculum. All of them evaluated the generated graphs (see Figs 6 and 7) of the curriculum, and reviewed the pictures that described possible interconnections among each study subjects independently. The consensus of their review, based on their negotiations and discussion, is presented below. The reviewers were already familiar with the description of the technical evaluation process for evaluating interconnections (in terms of similarities, according the key words mentioned in the curriculum), and were introduced to some technical aspects of the evaluation, but without any guidance, or suggestions, or evocative recommendations, as this could influence the results of their review. As mentioned above, all reviewers held very similar positions, a fact that was due to set criteria mentioned above, and their views did not differ. All reviewers used a simple template to provide feedback, namely a checklist that evaluated the structure of the visualized curriculum. In addition, they described an evaluation in their own words, and discussed it afterwards all three experts together during an unstructured interview. This the method was used to analyze the qualitative feedback (simple semantic analyses and subsequent synthesis of conclusions was done).

Below, the short evaluation of similarity graphs is stated. Below, the short evaluation (a summary of comments) of similarity graphs is stated. The colored groupings of individual communities with related contents across the curriculum makes viewing the particular medical curriculum simpler and easier to understand. The courses that fall into the preclinical disciplines show suitable overlap and are centered together. The interconnection among the surgical disciplines, Anesthesiology, and First aid is excellent, implying good consistency. The distant location of the Ethics course is logical. However, this course is not interconnected with either Communication or Psychology; instead, it stands on the opposite side of the curriculum. It is possible that Ethics, Communication, and Psychology are not sufficiently represented in the preclinical and clinical disciplines, which unfortunately is a prevalent trend in medical education at Czech medical faculties [39]. Another explanation might lie in their different locations within the curriculum, in terms of the year of study (i.e., Ethics is taught in the second year of study). It is very difficult to interconnect specific ethical cases on which the education of ethics should be based, if students have not yet participated in clinical practice or internships. This problem might be addressed by extending the education of ethics to include the internship period.

An interconnection between Microbiology and Infectious Diseases (see Fig 5) can also be rationally explained: this confirms clear interdisciplinary relations. In this context, Dermatology and Venereology are also suitably located, because they can be regarded as specialties focused on the most common infectious diseases. the distant location of the Stomatology course (see Fig 6) is understandable, as Dentistry is a stand-alone program of study in the Czech Republic, and students in General Medicine are merely introduced to this subject. Ophthalmology does not occupy large space within the medical curriculum and that it also has a low perceived importance. In fact, the distant location of preclinical practice in the course of Internal Medicine can be justified with regards to the involvement of different disciplines of internal medicine; otherwise, a high consistency of these disciplines (i.e., Diabetology, Nephrology, and Gastroenterology in the center of the schema [light red color]) is evident. A similar consistency can be seen in the location of courses focusing on community care and public health, including Clinical Practice in Community Medicine (marked in yellow). A high degree of content similarity (according to the location) can also be seen in the preclinical courses (i.e., Basics of Medical Terminology, Anatomy 1, 2, and 3), with an overlap with Histology and relationships with other clinical courses (marked in light red). The distant location of the Imaging Methods course is troubling, in terms of the curriculum consistency. On the one hand, one can argue that imaging techniques in this LU are rather specific, in that it teaches about uncommon examinations, as the common diagnostic methods are shown in individual specialized courses. On the other hand, relative serious shortcoming of this course’s remoteness is that it has no interconnection with other clinical disciplines; each clinical field/course/LU must certainly mention some examination techniques including imaging techniques, as mentioned above, and therefore it is essential to respect both horizontal and vertical interdisciplinary connections and relationships, which are based on a mutual sharing of requirements by individual teachers. The evaluation is also essential for the identification of the extent to which knowledge and skills have been acquired in individual courses. The general view of the curriculum is consistent, involving sporadic distant points that represent courses that focus on soft skills (i.e., Ethics, Communication, and Psychology) on the one hand, and courses that focus on specialized fields of medicine (i.e., Stomatology, Ophthalmology) on the other hand.