CESI Conference January 2002 St.Patrick's College

Abstract
This research examines and contrasts the behaviour of primary school children in schools with, and without, designated disadvantaged status. To see if there is evidence to back up the theory that pupils’ behaviour, as they use educational software, might actually militate against the acquisition of knowledge and concepts.

Is text ignored in favour of more visual stimuli? Is it the case that visual media, because they are less abstract than written text (i.e. things are depicted rather than described), demand of the pupil less mental effort and therefore lend themselves to shallow processing?

In particular this research looks at whether pupils in disadvantaged schools with poor reading ability, have the capacity to use this type of software effectively for project research and other elements of discovery based learning advocated by the New Primary School Curriculum.

To discover how pupils actually behave, a model containing many of the elements of educational software presently being used in schools, was built using Macromedia Authorware and the information on each pupil’s behaviour gathered and processed. As they work their way through the piece, the pupils are required to carry out various tasks and work out from the supplied text and visuals, (their information source) to get the answers to questions. The text supplied, and most of the graphic features are taken directly from software currently in use in primary schools. Every decision and answer the pupil makes is tracked, recorded and stored in an external file (in a floppy disk on the same machine). This allows for a large amount of quantitative information to be gathered. The resulting data can be analysed, and comparisons made between the performances of different groups of children. For example, between 4th class children and 5th class, between children in Disadvantaged and Non-disadvantaged schools.

The data also contains the times in seconds that it takes pupils to make decisions on each interaction. It is possible from this type of data to build up a picture of how pupils behave when they are using software. It may be possible to draw some very interesting conclusions on the suitability or otherwise, of educational software (in its present format), for certain groups of children.

Introduction
Have computers been over-sold to parents and educators? Can “Educational Software” actually improve learning? Does it suit all pupils’ abilities and learning styles equally? Can pupils use the Internet and computer software to carry out research?

No doubt teachers in many schools will have very diverse opinions on the above questions.

In my case I had become aware that there were problems and doubts expressed by my colleagues in my own inner city Primary school with regard to the value of computer, or, to be more specific, Educational software, as a teaching-tool. In particular they expressed concern that the pupils paid very little heed to the instructions on the software and demonstrated a tendency to just click at random without paying much attention to anything other than the changing images.

Teachers attending in-service courses, which I presented over the years, also mentioned variations on this feature or behaviour. Teachers who worked in schools, which had disadvantaged status, in particular, were frequently dismissive, of much of the available educational software saying that their pupils’ poor reading ability militated against the software being of benefit. They would complain that its use as a research tool for discovery-based learning was not feasible because their pupils lacked the comprehension and reading skills necessary to focus on such a task.

Furthermore their ability to carry out research on the Internet was equally if not more prohibitive for similar reasons.

The advocates of technology would contend that one of the main points in favour of computers in primary education is their power to motivate the pupil, particularly the pupil with learning difficulties and those whose learning style leans more toward the visual than the verbal. The visual image presented in much of the educational software is very attractive and there is no disputing its appeal and its power to transfer information in a unique way. Information presented in this graphic form can certainly hold attention and its motivational effect on children are well documented.

The Literature
Now that the computer has become part of everyday life it may be that there are a number of indicators, which have surfaced that are a cause for concern. Jane M Healy PhD, for example makes some observations in her book “Failure To Connect” which have become the main focus of this research.

Healy’s concerns are, briefly, that the educational value of today’s computers may have been “vastly over-sold to parents, educators and the general public, primarily by people who benefit financially from adding computers and software to the traditional educational mix”.

While Healy has respect for computers used intelligently and creatively by educators, she contends that research is needed to determine if, when and how computer applications can actually improve different forms of learning.

Healy pours scorn on much of the “Educational Software” in use in today’s classrooms. She maintains that, quite often, it is unsuited to the learning styles of the pupils. The programmers who create much of the software are not coming from educational backgrounds and there are very little input from educationalists. Moreover teachers are unable to tie the software in with their own curricula. In other words the software is too generic, and there is a problem directing the pupils’ attention on specific areas. Furthermore it is difficult to prevent their being distracted by the bells and whistles, which have been built into the programs.

Those are the general concerns made by Healy. However she makes reference on a number of occasions to the fact that there is a growing reluctance, and in many cases an inability to deal with written text, when it is presented as part of an application. Pupils have become more attached to visual media (which demands less mental effort), and shy away from written text. This may not just be specific to computers; Healy maintains that it appears to be a growing trend across many disciplines. But the genesis of such a trend may be as a result of, or caused by computers. This will surely have major implications for future use of computers in education and it is the primary focus of this research to determine whether this is really how children presently interact with computers and to what extent this “text shy” phenomenon exists.

The other trend referred to by Healy, and again a cause for concern, is pupils’ tendency to “guess and test” rather than take time to think or work out a problems. How shallow is the mental processing of pupils, who are increasingly more reluctant to think or make serious attempts to solve problems when working with “Educational Software”.

Sven Birkert warns in The Gutenberg Elegies: The Fate of Reading in an Electronic age, that increasing multimedia experiences at the expense of written text risks “language erosion”, decline of analytic and logical thought, “ the flattening of historical perspectives” and “the waning of the private self”. [Birkerts, Sven 1994] Surely, if this is the case, there is a need to take a closer look at this medium as it applies to educational software.

Birkert predicts that texts viewed as “difficult” will increasingly be glossed over and Healy maintains that this is, in fact, happening as students are both unwilling and unable to grasp the more subtle meanings or attend long enough to read them.

Salomon [Salomon, 1997] who also had reservations about newer technologies points out that visual media are less abstract than written text, because things are depicted rather than described. They also require less mental effort “elaboration” than does reading text, and therefore lend themselves to shallow processing.

The concern, then, is, that computers if used carelessly in the school environment may re-enforce superficial thinking and give rise to poor habits or behaviour in relation to problem solving.

Healy makes the point that “technology can nudge the brain to expand certain types of mental skills and neglect others, depending on how it is used. It can produce either flexible problem solvers or minds intent on finding one right answer, it can engender either a curious mind set or one dulled by spacing out on inconsequential games” p132.

Design
What I have created to collect the data is an information gathering multimedia program created using Macromedia Authorware. Basically what I have done is re created as faithfully as possible, a section or sample of the original piece of educational software, inside in a new self-contained program, which runs the very same way as the original, except for a number of modifications.

This ensures, to some extent, a controlled environment where all users are presented with exactly the same information and exactly the same problems in the same sequence, and the information gathered will be objective and accurate.

The program presents twenty interactions to the user in a linear sequence. Once the user makes their choice there is no second chance they will be taken straight on to the next interaction. All these features are explained to the pupils before they begin the piece, so there are no unpleasant surprises. The questions are designed to challenge the user to think about the problems presented, and then to choose the option which they think is the correct answer.

Also provided are two special options to which the user may resort, one is an option to Quit and the other is an option titled Can I get some help with this? The rational for providing the latter option, (which provides a strong hint as to the answer without any penalty in terms of marks) is, to determine whether the user will look deeper to explore as many avenues as possible to find the correct answer, before finally committing themselves to making their choice. Should, for example, pupils merely attempt an answer without spending time thinking about the problem, or checking out the available hints, when they encounter a difficulty, then it would be an indication that a “guess and test” strategy was being employed. Similarly, if the Quit option is chosen, or for that matter, chosen frequently, this might indicate a number of factors. It would certainly indicate an unwillingness to engage with the content and some laziness of mind. It might also indicate a lack of confidence in the user. If this were to be a common feature or pattern in the findings, then it may be an indication that pupils were more interested in moving on to the next visual stimulus with scant regard to the problems posed. (Fast response times might indicate something similar).

Once the user chooses answer they automatically move on to the next interaction passing through a preliminary screen which instructs them to read the next question carefully, and every now and then provides a small element of positive feedback such as “Good!” or “Very Good!” (Regardless of whether they had, in fact, scored correctly on the previous interaction). As I was concerned that all users would feel equally confident at each interaction, I felt that any element of negative feedback would introduce an extra variable, which, could only serve to complicate the findings and could ultimately detract unnecessarily from the performance of certain pupils. Admittedly the influence of negative feedback is worthy of study and should have a place in this type of research, however I thought it might be wiser to leave it for inclusion in a later piece, and concentrate for the present on isolating factors such as the users interaction with text, which I have introduced with increasing complexity as the piece moves toward the end.

The last two interactions number 19 and 20, have a different format and were designed to shed light on how pupils might operate a simulation or micro world. In this case the user is required to launch a rocket using one of four specified quantities of fuel. The pupils are warned that before they fire their rocket, they must first test it out using each of the fuel amounts. The idea being that as they test their rocket, they pick up or learn a concept. The concept being…. The amount of fuel must be in proportion to the size of the rocket (too much fuel and the rocket blows up, too little and it can not leave earth’s atmosphere etc.). The test area is quite elaborate and gives adequate feedback to the pupil at every step or choice they make.

I hoped to pick up on a number of aspects of behaviour with this interaction, not least being, a measure or indication of how impulsive pupils are when presented with such interactions. How do they manage simulations, which are designed as a ramp to learning or grasping abstract concepts? For example, would pupils, in this case, ignore their initial instructions and warnings and go straight into launch mode (without testing), select “Full Rocket” and “Fire”. It was also intended to determine whether they read or ignored their initial instructions and indeed if they were capable of grasping the inherent concept.

To determine whether or not they had, in fact grasped the idea, interaction 20 would present a number of options from which the user was required to select, what they thought, was the idea behind the previous interaction.

Implementation
Nine classes from five schools, a total of 220 pupils were tested on this program.

Prior to testing, all pupils were given the same introduction, a set of pre-prepared instructions from a manual. Pupils logged on and were required to enter their name age and class.

All data from their performance on the program is automatically sent to an external file, at the end of the program, located on a floppy disc. The file is given the users name for identification and to avoid proceeding users files overwriting the previous users records.

The advantage of a data collection system like this is that there is minimal disruption to the class, and it’s quick and clean.

Evaluation
· Did pupils attempt answers without spending time thinking about the problem posed? (A “Guess and Test” strategy?).

· How did they interact with text? Did they find answers to questions from the supplied text? Did they read their instructions properly?

· How impulsive were they?

· Did they grasp the concept intended in the “Micro world”? Can they use a simulation to learn a concept?

To discover the answers to the above questions, the number of incorrect interactions scored by pupils were the first data to come under scrutiny.

“Wrong” Interaction Times 4th Classes Disadvantaged

Figure 1a

“Wrong” Interaction Times 4th Classes Non-Disadvantaged.

Figure 1b

“Wrong” Interaction Times 5th Classes Disadvantaged

Figure 2a

“Wrong” Interaction Times 5th Classes Non-Disadvantaged

Figure 2b

How many of the answers at each of the interactions were wrong?

How many of those wrong interactions were attempted in:

· Less than 10 seconds? (Very little thought before attempting an answer!)

· Between 10 an 20 seconds? (Some time given to the problem, but not a lot when it involves searching through text to find the answer.)

· Above 20 seconds? (An honest effort and an attempt made to read and understand the text).

Was there a marked increase in the number of wrong answers on interactions demanding that the pupils read text to find their answers?

Figure 1 above shows that there is a marked increase in the number of Wrong interactions as pupils move through the piece. The interactions demanding some comprehension skills i.e. Interactions 5 and interactions 11 through to 20, in particular would suggest that Disadvantaged pupils indicate a tendency to “guess and test” more than Non Disadvantaged pupils.

If one considers that Interactions 11 to 20 are those in which the users are required to negotiate text, there would appear to be indications to support the hypothesis that pupils may indeed be avoiding or unable to deal effectively with the text. Figure 3 shows the percentage wrong interactions as they applied to 5th classes (Disadvantaged and Non Disadvantaged).

In the “Rocket” interaction, (Interaction 19), How many pupils fired their rocket without testing, indicating impulsive behaviour, and or a failure to read their instructions carefully?

Figure 3 shows that 50% of pupils in the Disadvantaged 5th classes went straight for firing without testing as compared to 25% of Non-Disadvantaged pupils.

What is really striking is the failure of pupils to grasp the concept intended by the simulation, both in the Disadvantaged (65%) and in the non-Disadvantaged sample (45%).

Another interesting finding was that both Disadvantaged pupils and Non Disadvantaged pupils had a similar result on interaction 5 which demanded an element of comprehension but where there was very little text involved. What was required in this case, was that the user had to look carefully at the problem and work out the solution from the picture presented.

It is also interesting to note that Disadvantaged pupils had a greater number of wrong responses on interactions 11 to 20, than Non-Disadvantaged pupils, which would indicate that Pupils’ poor reading ability may be responsible for this.

The next step with this research would be to extract the interactions that do not require reading comprehension (1,2,3,4,6,7,8,9,10) and process those that do (interactions 6, 11,12,13,14,15,16,17,18,19 & 20) to see in greater detail how pupils deal with text on this program.

Figure 3

Discussion
There are many criticisms I am sure that can be made of using a “contrived” program such as this. That, for example, pupils need to be thought how to use simulations before they can be expected to learn from them, in the same way as teachers must train their pupils how to use worksheets.

Figure 4

Another criticism might be, that the text was too difficult in some cases. But keep in mind that it was possible for every pupil to score correctly by resorting to the hint option and yet very few did (10%).

What will be interesting is finding what percent of the “Wrong” interactions, which involved text, compare with “Correct” interactions, which involved text. The breakdown on Figure 4(above) would be quite different if the interactions, which involved text only, were included.

Very few of the interactions were skipped (2%), which is a good sign, and only 3% of pupils, which resorted to a hint subsequently got the answer wrong.

The process of using a program such as this has definite advantages in terms of control and collection of data. It has potential for example to analyse factors such as the influence of negative feedback on pupils, and ways of constructing and evaluating the effectiveness of micro worlds, “such applications can improve on human thinking abilities, and they will open new panoramas of possibility” [Healy p167]. However if we can imagine for a moment that what Figure 4 represents is an accurate reflection of how pupils really cope with text. Where over 25% of pupils are failing to deal effectively with what was required. Should school administrators exercise more caution before they cut funding and space for books, the arts and physical education to make room for new computer hardware?

References

Birkerts, Sven. The Gutenberg Elegies : the Fate of Reading in an Electronic Age. New York: Fawcett Columbine, 1994 p128-130

Healy Jane M. Failure to Connect: How computers affect our children’s minds, for better and worse. New York, Simon & Schuster 1998. ISBN 0-684-83136-8

Salomon, Gavriel. Of mind and media Phi Delta Kappan January 1997, pp375-380

Salomon, Gavriel. Interaction of media, Cognition, and Learning. San Francisco: Jossey-Bass, 1979, p.237.