Thinking Tools for the Net - Ajattelun välineitä tietoverkkoon



	Front Page RESEARCH TOPICS Automatic Evaluation of Essays Concept Mapping Techniques Visualization of Social Interaction in Virtual Environments WHO & WHAT People & Alumni Publications LINKS Educational Technology @ Department of Computer Science Department of Computer Science in University of Joensuu National Technology Agency of Finland(TEKES) European Commission - Research Directorate Discendum Novo Group Plc Sordino Information Systems Intraweb	Automatic Evaluation of Essays The need for computer-assisted assessment of learning outcomes is connected to two inter-related factors in today's schooling and education markets. First, teachers need to automate the assessment and evaluation process especially in mass courses. Secondly, a student, particularly when following an online course, wants to assess his or her own learning process already before an examination. Furthermore, various evaluation and assessment schemes are widely used also outside traditional education. Our Java-based essay grading system is based on Latent Semantic Analysis (LSA). LSA is a corpus-based statistical method, related to neural network models, which provides a means of comparing the semantic similarity between a source and target text. LSA can also be considered as a computational model of human knowledge representation. Research has showed that the method is able to simulate learning and several other psycholinguistic phenomena. The system (Fig.1) preprocesses the essays by a morphological parser. It allows the user to adjust a set of parameters, for fine-tuning the accuracy. The grade is computed by using both human-graded essays and text materials from a textbook. Figure 1. Essay Grading System We have experimented with system consisted of essays and textbook written in the Finnish language. For the experiments we collected essays from an undergraduate course in education. We typed them out and ran experiments with our Java-based system. The test set consisted of 143 essay answers to a essay prompt. The essays were graded by a professor on a scale from 0 to 6. The grading system was trained by using a relevant chapter from the course textbook. Table 1. The results of the experiments In Table 1 the column "training material" shows the structure of the training material used in the experiment. For example in the case shown in first row of the table, 5 sections from the textbook and 70 essays were used for training the system. The results of the experiment are shown with the proportion of cases where the same score was assigned by the system and human grader (exact) and the proportion of the essays where the grade given by the system was at most one point away (exact or adjacent). The last column of Table 1 shows the Spearman rank correlation between the scores given by human and the system. Tuomo Kakkonen, firstname.lastname@cs.joensuu.fi LINKS: Esseetehtävien tietokoneavusteinen arviointi [PDF] (Tuomo's Master's Thesis, in Finnish) Computer-assisted Essay Evaluation [Powerpoint-presentation] Webmaster