Measuring and Monitoring Student Learning using an Educational Software Engineering Environment

E.A.Dodman and J.F.Coxhead

Software Technology Group
Leeds Metropolitan University
The Grange, Beckett Park , Leeds LS6 3QS, U.K.

Table of Contents

Teaching Philosophy
The Tools
User Interface
Experience of Using the Environment
Current Contact Address


The craft of the software engineer is complex and difficult. The end result of the work of a software engineer is a software product. Much work is being done on measuring both the quality of the final product and the process by which it is achieved. However the quality obviously depends crucially on the ability of the persons developing the product. Education and training is therefore a vital component. Unfortunately quality measures for the product of an educational system ( the graduate student ) and the process of production (the learning/teaching process) are not easily defined or measured. The work described in this paper relates to the education of software engineers and an approach to the measurement of how successful this is in producing a quality software engineer. It is hoped that many aspects of the research can be applied to other areas of education. This paper describes the results obtained from the system and discusses their implications with respect to the provision and measurement of the effectiveness of automated environments for learning. Particular attention is paid to the interface model used and measures of it's ability to satisfy the requirements of ease of learning, ease of use and competence to handle the complete range of student ability and experience.


Commercial software tools are understandably built primarily to meet the needs of the professional practitioner. The combination of the software and the hardware on which to run it therefore reflects certain assumptions about the commercial enterprise in which professionals practice. As most software tool development is market driven the money making value added features result in a wide selection of interfaces which poses a problem for academic teaching and skill transfer. The teaching enterprise, although having to deal with the same technologies, differs greatly in its tool requirements and hardware configurations. The group based requirements of the teaching environment are often at variance with the more individually based commercial working practices. The consequence of this is that the purchase of popular commercial tools for academic use more often than not will not satisfy perceived academic requirements. Overfunctionality, long learning curves for different interfaces and lack of integration may create their own problems. In the worst case courses may have to be re-modelled around making best use of such costly tools as are available rather than the tool being configured to teaching needs. A long-running project at Leeds Metropolitan University has been to devise an Educational Integrated Project Support Environment for use in the training of students of Computing. The philosophy underlying it's development was that of enabling students to be educated in basic design principles while using tools which would enable easy transferability to the commercially available tools commonly found in the software development industry. The environment was to be used throughout the complete course (degree or otherwise) that the student was studying. A consequence of this was that the environment and it's incorporated tools had to present an easily-learned interface to the user that was consistent throughout the entire period of a student's study. The interface had therefore to be straightforward to learn and simple to use by students at the start of their course. It also had to have the capability to handle the sophisticated commercial standard tools used by final year and post-graduate students.


What was clearly needed was a configurable environment in which we could alter tool functionality and interface at will and with minimum effort. Meta-CASE has provided us with such a facility [5]. The product we used to construct the environment, the IPSYS Toolbuilding Kit, is derived from technology which originated in the early ALVEY and ESPRIT projects. A significant breakthrough with Meta-CASE is genericity and re-use at a high level. A good example of this is a graphic design editor which is usually required with most CASE tools. Usually each instance of such an editor is built from a lower level using components of some language library. With Meta-CASE the whole diagram editor can be built to facilitate re-use by parameterising it to support a number of different methods. The power of this type of high level re-use is that generic user interfaces can be constructed across all tools and the functionality of the tool can be quickly altered as large parts of the application interface are also generic. Other features of Meta-CASE include full control over all the database schemas (objects defined in a database to represent a particular method), an inherent messaging system and context related help structures.


A teaching philosophy was defined to incrementally teach students in stages over their course period. The use of meta-CASE technology enables tools to be tailored to the current level of student ability and to be modified to meet tutor requirements whilst still retaining full compatibility and interoperability. The environment provides tools to facilitate the teaching of individual design techniques but also incorporates the use of full industry-standard methods and techniques (e.g. SSADM, IE, HOOD) as required in the later stages of a course. The flexibility of the system enables us to address the problem that academically students need to understand basic principles whereas to get a job they often need facility in currently popular methods. The actual design philosophy and architecture of the system is described in the papers [ 1,2,3,4]. Initially it was thought that three levels of tool would be required starting with basic diagramming tools based on a single technique and ending with the students being subjected to full toolsets based on a particular method. The basic diagramming tools are referred to as tool fragments and the highest level tools are referred to as fully integrated toolsets. An intermediate set of tools known as intermediate tools were developed; these were again based on a single technique but had an underlying data dictionary and supported a number of automated features. However, from trials it has been found that there is no real need for the intermediate tools as the strength of the generic interfaces allows students to transfer their skills directly from the tool fragments to the fully integrated tools.


A major problem with teaching software engineering is that practical experience is more often than not gained from using tools/toolsets developed for the seasoned practitioner and these are overwhelming for inexperienced students. However, the strategy for teaching software engineering is no different from that for most vocational subjects in that the overall view of the subject area is broken down into logical and manageable modules. The modules will usually represent a particular technique which may be taught in isolation (although it may depend on previously covered material). As most techniques are complex in their full extent it is usual practice to start with the smallest possible subset and add to this over time. The tool fragments are designed to mirror such a strategy. What has been developed is a selection of tools supporting different techniques and notations. Each tool interface is identical the only difference being the contents of the menus, the shapes of the graphic objects that can be put on the screen (notations for different techniques) and the rules governing the relationship between the graphic objects (the technique itself). The functionality and syntax of the tool can be reduced to that of a required starting subset as requested by the tutor. For example the tutor may not want to start with a full set of technique icons or to have the rules that govern the syntax fully automated. The missing features can then be replaced over time in step with course development again at the request of the tutor.

There is a second motive to using the tool fragments and that is familiarisation with the underlying operating system. To incorporate fully the learning of the operating system most file management tasks are deliberately left for the student to carry out within the operating system file manager (copy, rename, new version etc).

Figure 1 shows a screen dump of the tool fragment menu demonstrating the range of tool fragments available, all of which may be selected from a master window.


The environment is designed to be used by students at all levels of training. To ensure this a consistent house-style was adopted for the environment interface. The meta-CASE technology used in developing the environment then allowed the construction of a generic graphical user interface which could be tailored to the needs of particular student groups. As the students using the environment were all computing students a control panel metaphor was selected as being the most appropriate for the interface rather than the more common desk top model. This control panel metaphor is the standard model provided by the IPSYS toolset. Although different models of interface can be swiftly generated we decided to base our model on this IPSYS house style believing that the control panel metaphor was more appropriate and easier to use by software engineering students than the more usual desk-top metaphor found in most windowing systems.

Figure 2 shows a screen dump of a tool fragment in use. The bottom part of the screen is a graphical window holding the design diagram currently being worked on. The upper part of the screen represents the control panel with drop down menus, signs and buttons and is is generic to all tool fragments. The blank line between the two parts of the screen is a message window used to report any system messages back to the user. The philosophy behind this approach is that of minimum surprise with standard naming, placing and actions of buttons, switches and menus across all tools. In any particular menu users are always offered only valid selections for that menu. As an example of this we can take a Data Flow Diagram technique where we can add a process to our diagram. The process will have attributes that could be added to it such as name and identification. If these attributes are single instance attributes once they have been added they will disappear from the menu selection.

As students progress through their course they move through the full range of tools ending up using one or more of the fully integrated toolsets (the current selection is SSADM v4, HOOD, Information Engineering and a number of Object Oriented toolsets) but always seeing a common interface with only those differences ( for example the syntax of the method ) necessary to support the particular toolset being used.

Figure 3 is a screen dump representing the use of a full commercial SSADM Version 4 tool ( constructed using IPSYS Tool Builder ). The tool's Structure Editor ( left hand side of the screen ) and Diagram Editor ( right hand side ) are both displayed together.

When students are faced with large complex toolsets one of the common problems is that they may get lost down several levels of menus. Within the IPSYS Meta-CASE environment this is not a problem as everything is hypertext based. As well as having a generic design editor there is also a generic structure editor. The main purpose of a generic structure editor is to be able to represent a structured textual view as opposed to a diagrammatic view of the system. The structured text editor has a control panel to handle such things as printing, saving, history and context. However, navigation and editing are done through the actual textual objects. A menu can be attached to a single letter, a word, a line or a block. Navigation is handled by pointing at one of these areas and selecting the next action from the popped up menu. A path is always visible so users know where they are and any part of the path is selectable to return to any level.


With this type of high level re-use tools supporting similar techniques or methods can be very rapidly prototyped. For example once you have created one data flow diagramming tool then a whole range of similar tools with different notations can be built very quickly. There are several levels of development, the highest being the one just described, the next level down would be to develop a completely new technique or method. This would take a little longer as it would involve the definition of a completely new database schema. However, the generic design editor and structure editor is still used to present and manipulate the technique/method which still makes the development process considerably faster than conventional developments of this type. If you find that there is not a pre-defined built-in function in the existing libraries that you require to support your new method then you have the capability to drop to the language level and create one. The latest innovation with the IPSYS Meta-CASE product is a tool building tool called Toolbuilder. The toolbuilder tool looks and feels just like the tools it builds. With toolbuilder you can define a method using an entity modelling technique; the entity model is automatically converted to its equivalent text structure which is linked to a pre-defined set of templates. Once the templates have been populated the whole model is compiled and the tool is made by running a generic make file.


The system has been installed and operational at the level described for two years and is accepted enthusiastically by students. Students find the interface easy to learn and have no difficulty in transferring between different tools. It is this that has led us to decide that the intermediate tool level we originally envisaged was not necessary. The environment has proved popular in that students not only use it as required in various parts of their courses but drop in and use it on an open access basis for preparing work in other parts of their course. This practical demonstration of popularity is confirmed by student response to follow-up questionnaires. The standard of coursework submitted has improved both in terms of presentation standard and better design of systems. The latter is felt to be due to the fact that the ease of use of the system encourages students to spend more time on coursework. Having been successful in the implementation of the system it was felt to be an appropriate time to move on to the more demanding consideration of measuring how the environment affected, and could be used to monitor, the quality of the work students produce and the quality of their learning experience. This will also incorporate measurement of the tools fitness for purpose. In order to be fully in control of the educational process it is necessary to consider the quality of the student the effectiveness of the teaching approach taken the effectiveness of the environment in the support of course delivery and to provide measurable criteria for each of them. These tasks are in decreasing order of difficulty but all are formidable. In order to obtain this we have started to incorporate measuring and monitoring tools into the environment. This is made readily possible because the IPSYS environment has a built in messaging system incorporated in all its tool instances. All messages generated while in the system are routed to a session log and this log has been exploited by us to monitor system usage. There are 11 types of message covering not only errors but such things as warnings and metrics thus enabling us to obtain a comprehensive view of a student's use of the system.

Figure 4 is a full list of message types produced by the system which may be monitored via a session log.

Measurement tools are being incorporated into the system initially to quantitatively monitor the systems performance and usage. These tools will be used to evaluate the environment's fitness for purpose and enable the building up of profiles of student usage of the system and progress through the course. This will enable detection of aspects of the system which are not satisfying the requirements but, more importantly, will enable the early detection of students who are experiencing difficulties. When profiles of normal student usage are built up it should be possible to provide automatic triggers in the system which will initiate aid to the student. This may be on-line help, supportive tuition from the tutor or, in future, might incorporate on-line corrective training.

The aim of the monitoring is initially to test the effectiveness of the environment and the toolsets in their role as a teaching environment. The next aim is to monitor how well students are assimilating the methods and techniques they are taught. This will initially enable the early detection of students who are experiencing difficulties and allow early remedial help to be given. It will also provide support for tutors in terms of assessment and appraisal. The long term aim is to incorporate interactive teaching and remedial assistance for the students.

Monitoring and measuring can assist in many areas -

1. Monitoring individual tools for fitness of purpose

2. Monitoring students and providing class and individual profiles. These will flag gaps in understanding in both individuals and the group.

3. Assisting tutors in recognising absenteeism, plagiarism and areas of weakness requiring corrective or remedial teaching.

4. Assisting course leaders and course teams in identifying areas of weakness in satisfying the learning objectives of courses

5. Assisting technical staff by providing statistics on system usage, peak loading periods and indicating system weaknesses and shortfalls.

This monitoring phase of the project has reached the stage where the programs for analysing the messages have been incorporated into the system. It is currently being used to analyse data for complete classes of students to enable average student profiles to be built up. This is necessary to enable the detection of students with particular problems. Some of the analysis available to the system is demonstrated in the figures below.

Figure 5 shows a profile of errors made by a single class of students during a laboratory session. The graph highlights one student as having made many more errors than the average and flags the need for further investigation. Such crude analysis does not indicate the student was worse than the others as there is no indication of the type and level of work each student is pursuing but does indicate that the work of this student needs further investigation. Analysis of each error type and frequency over the whole student cohort is required in order to highlight common misunderstanding or teaching omission.

Figure 6 gives a profile of the messages received by a single student during a single session.

Figure 7 shows a class report for a set piece of work. This is of use to tutors as a measure of how much system time students have required to do a particular piece of work and to technical staff as a mesure of system loading.

Figure 8 displays a class profile of all messages generated throughout a two hour laboratory session broken down into 10 minute intervals.


The environment we have described is complex and incorporates state of the art technology and toolsets. However students at all levels find it easy to use largely because of the straightforward and simple user interface level used. Tutors find the environment helpful in that the tools and techniques taught can be tailored to suit pedagogic needs. Monitoring facilities are now incorporated into the environment which will enable measurement of not only the quality of the system but also the quality of the work students are producing. The ultimate goal is to merge this with on-line interactive teaching for the students. The system as described has been designed especially for students of software engineering. It is felt however that the monitoring and appraisal is relevant to students over a wide range of subjects and we hope to utilise the experience obtained here in other areas of teaching. It also has potential application in the commercial sector as a means for facilitating Quality Audit [7].


The authors would like to thank Mark Dixon for his help in preparing the paper and in particular Yvonne Bulmer for all the work and commitment she put in during her period of work placement.


1. Coxhead, Dodman and Harvey
CASE requirements for teaching software engineering.
Proc. 4th UK ISTIP conference: AIT 1991, pps 65-69.

2. Coxhead, Harvey and Dodman
CASE and the education of software engineers.
Proc. SEHE conference: Southampton Institute 1991,pps 174-177.

3. Coxhead, Harvey and Dodman
An integrated environment and teaching philosophy for the education of software engineers.
Proc. 5th UK ISTIP conference: AIT 1992, pps 89-93.

4. Coxhead, Dodman and Harvey
CASE technology: improving quality in the education of software engineers.
Software Quality Management, Eds. Ross, Brebhia, Staples & Stapleton:
Computational Mechanics Publications, Elsevier 1993, pps 317-330.

5. Alderson.A.
Meta_CASE Technology IPSYS Software plc 1992.

6. Alderson.A.
Beyond Today's CASE Technology Toward Meta-CASE
IPSYS Software plc 1993.

7. Coxhead, Dixon and Dodman
Meta-CASE and Audit: Automated Generic Quality Assessment
Proc: 1st International Congress on META-CASE, University of Sunderland, 1995.


Email address