Introduction
Where is Student Modeling Presently Being Used?
Methods of Deriving the Student Model
Summary
This paper is part of a research project whose focus is to explore the feasility of using student modeling in the development of Internet-based courses at Athabasca University. The references for this paper were obtained by doing a search of student modeling references on the Internet. There are a variety of on-line learning systems currently using student modeling techniques. The systems that are described here include OLAE,POLA, ATS, Cascade, Matsadu and Okamoto's student modeling research, and University of Saskatchewan's MicroWeb project.
Each of these systems are defined, and the way the student model is derived is described for each. The suitability for use of these systems in Web-based learning systems is discussed in the summary.
Where is Student Modeling Presently Being Used?
OLAE
OLAE (Online Assessment of Expertise) is a tool to help assessors determine what a student knows, as compared to most student assessments that determine how much a student knows. This tool is being used in introductory college physics. It uses Bayesian nets to observe student behavior and compute the probabilities that the student knows and uses each of the rules in a given knowledge domain.
The student model that is produced consists of a rule-based program that reflects the way the student computes answers to actual problems, both correctly and incorrectly. Bayesian nets are used to address the uncertainty of these rules. This uncertainty is produced by such situations as typos or a student guessing a problem's solution and getting it correct.
The purpose of OLAE is assessment. The analysis is aggragated in various ways to be used as is determined to be appropropriate by the assessor. An example of this would be to improve the distribution of materials for the physics program. Assessment is defined here as the problem of determining what the student knows.
POLA
POLA (Probabilistic Online Assessment) is also used with introductory physics. It is a student modeling framework that does a probabilistic online assessment of student problem solving. OLAE uses knowledge tracing in its student modeling. POLA is able to turn this tracing into a system of probabilistic reasoning, which generates predictions about the solution the student is following. The end result of this is an assessment of the student's mastery of the knowledge involved in the solution.
ATS
The Adaptive Training System (ATS) uses the ML-Modeller as its student modeling component. The purpose of the ATs is to represent a student's knowledge states and transition to expert state, and thereby provide specialized tutoring that has been adapted to the student's learning style. Using machine-learning techniques to emulate the novice to expert transition, this system dynamically models the student's learning progress.
Cascade
Cascade is a model of cognitive skill aquisition. This is generally used to account for the psychological results of the self-explanation effect. Several investigations [7] have shown that in the acquisition of sophisticated skills, such as physics problem solving, or Lisp coding, students who explain examples to themselvesshow improved learning and use analgies more efficiently in their problem solving. Cascade us able to reproduce the self-explanation effect in its learning mechanisns.
Procedural Problem Solving
Matsuda and Okamota [12] have investigated a method of obtaining a student model in the domain of procedural problem solving. In this domain, students tend to learn only the problem solving skills, and the basic knowledge needed to solve the given problem. There is the tendency to not learn as much about why the problem can be solved using this knowledge. Thus even though the student may be able to solve a set of given problems, they may not be able to solve a set of similar problems.
In this case, the student model is used to explain the actions observed during the student's problem solving performance, to be used with an intelligent tutoring system. If the goal of the instructional system is to communicate the problem solving knowledge, then the most important aspect of this system is to identify the knowledge that the student doesn't understand.
Procedural problem solving is defined here as the process of searching for a solution to a problem by applying problem solving knowledge. This knowledge has been divided into three classes;
The researcher feel that students often learn only the Effective Knowledge, and not the Principle Knowledge that is needed to understand the rationale of the problem solving skills. Heuristic Knowledge is necessary for effective problem solving, as well as to be able to understand the Principle Knowledge.
This model is not dependent on any particular domain, and can be utilized using many actual domains as instructional subjects.
MicroWeb
MicroWeb research, currently underway at the University of Saskatchewan, involves the creation of a toolkit that facilitates the creation of cohesive document collections, using World Wide Web documents. The tools include those required to collect, organize, and annonate existing documents. Once the collection is organized, student users will view the colection with a standard WWW browser.
The student model in this instance will be implemented where the teacher chooses to define documents that must be viewed, defines a specific ordering for the documents. As well, the student maybe be limited to viewing documents only in the collection, or be allowed to browse further in the Web.
Methods of Deriving the Student Model
OLAE Researchers of OLAE have defined three obstacles to effective student modeling;
OLAE has been developed with these obstacles in mind. The tasks of this system include collecting and analyzing data, using sound probabilistic methods, and presenting the results in a flexible manner. This is accomplished by creating a Bayesian net that relates to the domain knowledge for each problem. These nets are represented as first-order rules to particular actions, such as written equations. OLAE observes a student's behavior and then computes the probabilites that the student knows and uses each of the rules.
The data is collected on-line using a graphical interface, which allows the student to select an icon and choose to do problems in any order. OLAE records all of the student's actions, parses them, and adds them to the assessment.
The Bayesian net that is created, has four types of nodes. One node represents whether or not a student knows a given rule of elementary physics. The second node represents whether or not the student used actually used the rule, given a specific problem. The third node represents whether or not the student believes a particular fact about the problem, and the fourth node represents whether or not a student has performed a particular action.
The analysis involves a multi-step process, which starts with the domain model and a physics problem. The domain model is based on the Cascade model, is applied to the problem to produce a problem solution graph, which indicates all possible inferences which can be drawn from the problem's solution.
The student model is the Bayesian network generated, which is also based on previous assessments of the student. Once this model has been connected to the problem solution graph, the data is processed from the interface. The resulting probabilistic assessment can be viewed at the level of the rules of the domain model, or at different levels of abstraction.
POLA
POLA generates probabilistic predictions about student performances in the same domain as OLAE - introductory physics - using model tracing, as well as knowledge tracing. This framework is able to infer the predictions about the student's line of reasoning without using heuristics, as OLAE does, even when the problem's solution space is large.
This method involves the use of an AND/OR graph, which provides a compact representation of all the solutions of teh given problem. A Bayesian network is built incrementally from this graph. Predictions are generated from the student's actions about the solution the student is following. The end result is an assessment of the student's mastery of the physics knowledge that is required in the solution of the problem.
ATS
The ML-Modeller, which is the student modeling component of the Adaptive Training Systen (ATS), uses machine learning (ML) techniques to emulate a student's learning state. This system infers which learning methods the student has used in the transition from novice to expert, by comparing the student's solution to an expert's solution. Plausible hypothesis are then generated about which errors and misconceptions the student has made.
This system uses a case-based method, which generates its hypothesis through incorrectly applying analagies, and infers mistakes from overgeneralization and overspecialization. of rules. It also uses fuzzy methods to represent incertainty. The resulting network representation of both student and expert modesl show abstract concepts and relationships, as well as strategies for problem solving.
Cascade
The computer model, Cascade, accounts for the correlation between students who use self-explanation, and the learning improvement seen, along with improved economical use of analagies in problem solving. This is achieved during the exploration of an example, which causes Cascade to acquire new domain knowledge. Through the use of analagy, derivisional knowledge is used to control later problem solving. New domain knowledge is acquired when Cascade's current domain knowledge is complete, causing an impasse to be reached. A new rule is guessed at, and if it works to solve the problem, then it is added to the domain knowledge on a provisional basis. This the new rules become specialized in the process of applying general knowledge to the problem.
Procedural Problem Solving
In Matsuda and Okamoto's student modeling studies, the diagnosis of the student model involves three components. These are the state, the operator, and the search control rule.
The states of a given problem include the initial, intermediate, and the final (or goal states). Every problem has an initial state, but not all problems have a goal state. Searching for a goal state may in itself be the problem. The operators have the conditions of effects caused by applying them to the problem. Because applying operators blindly could exponentially widen the search, search control rules are used to simplify the search.
The operators represent the operations that would decrease the distance from initial to final state, and are categorized as primitive, or as macro (which are a set of primitive operators). The macro operators reduce the effort of searching, and can be categorized into a kind of heuristic knowledge. The search control rules are a set of explicitly represented knowledge that are implemented as the knowledge to resolve conflicts between primitive operators. The sequence of operators that would solve the problem are called the solution set. After a solution is arrived at, macro operators are generated from the primitive operators.
By observing the process of students' problem solving, the system can then identify which operators the student wither isn't using or doesn't know. This infers the principle knowledge according to the student's observable comprehension of the effective knowledge.
MicroWeb
The University of Saskatchewan's MicroWebproject, involves the use of student modeling to show how students should view the documents. This model is created by using logs of student's actions, tracking the time and place of their travel. From this log two things are analyzed;
Based on the student model, a series of hints would be used to either render a forced tour, or to suggest to the student where to proceed. Analysis will also include a time analysis to determine if a student stayed at a particular page a reasonable length of time, or merely clicked on to the next page after a very short time span.
All of the above systems use student modeling in the delivery of online instructions. OLAE and POLA are both specific to introductory university physics, and use the student model to determine where the student is erring in the solution of specific problems. The ATS and Cascade models are used to determine changes in the student state in the acquistion of expert knowledge. Matsuda and Okamatu claim that their model is not limited to any particular knowledge domain. What all of these systems have in common, is that they are used in the delivery of courses that require procedural knowledge domains.
In the background paper to this research, Student Modeling and Web-Based Learning Systems, two approaches to instruction design were compared. The objectivist approach was defined as one that determined a goal, and a method to reach the goal, whereupon there was a measurable outcome. This approach was determined to be the most appropriate in the case of procedural knowledge domains, and is the approach taken in the above systems.
The second approach defined was the constructivist approach. This approach was determined to be suitable for more advanced learning methods, and also was suited for learning on the World Wide Web. The reason for this is that the constructivist approach is learner driven, and the links and nodes of the Web are ideal for this. The one system that I researched that utilizes this approach is the MicroWeb project at the University of Saskatchewan.
This research is currently underway, and it promises to be relevant to the development of Internet course delivery systems. It uses an expert's contruction of the domain to form the basis for the links and nodes in the learning system. Advanced navigational tools will assist the learner through the these links and nodes. The adaptive interface is suited to the needs of the individual learner and adaptive advice will intelligently suggest a preferred path through the knowledge base, which has been based on the student model.
The MicroWeb system would appear to be more suitable than the others, for Internet course delivery at Athabasca University, based on the above reasons. The other types of systems that were examined are using student modeling for specific purposes that would likely not be as appropriate for use at Athabasca University.
Cialdea, Marta . "Meta-Reasoning and Student
Modeling. "
Meta-Reasoning and Student Modeling - Link Unavailable
Gürer,Denise W. , Marie desJardins, and Mark Schlager."Representing a Student's Learning States and Transitions (Abstract)".
Presented at the 1995 AAAI Spring Symposium
on Representing Mental States and Mechanisms, Stanford, CA;
published as a AAAI technical report.
Representing a Student's Learning States
Matsuda, Noboru & Okamoto, Toshio. "Student Modeling for Procedural Problem Solving".IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, January 1994. Tokyo, Japan. Noboru Matsuda
Thomson, Judi & Philip, Tim. MicroWeb - work in progress. University of Saskatchewan, 1996.
Index to Kurt VanLehn's recent publications;
Martin, J. & VanLehn, K. (in press) Student assessment using Bayesian nets. International Journal of Human-Computer Studies, Vol. 42.