Key points:
- No evidence of effectiveness of SCOs, i.e, that they improve students’ capabilities. What’s the point having SCOs if we don’t know they’re effective? Has there been a gain from pre- to posttest scores?
- Some simple metrics that would help: How often has the SCO been used? Presumably the more heavily used SCOs were thought to be effective at least by the instructional designers.
- What prior knowledge is assumed by a SCO?
- Are there individual differences in the effectiveness of SCOs? Meaning, do people with certain characteristics learn more from SCOs than others? Can SCOs be described so that we can study whether some types of SCOs, say those demanding induction, are better for some students than others depending on deduction?
These are difficult questions, and answering them may take a lot of time, but if we don’t start asking them now, when?
My career has been devoted to studying learning from instruction, in a variety of ways with a strong emphasis on educational technology and presently Dexter Fletcher and I have just finished a book on computer games and instruction which has been sent off to the publisher. ADL is Dexter’s brain child. Dexter and I have worked very closely for over a dozen years on lots of publications in a variety of areas.
I think that what I have to say is pretty well summarized in the white paper I wrote. I very much appreciate the technical accomplishments of SCOs, getting them to run on any system, etc. As a research concerned with learning and instruction, I continue to regret the fact that issues dealing with learning from instruction always receive second, third, or fourth consideration. Technical issues always first. And my aim is to bring some of the technical issues dealing with learning from instruction to the fore, because I think if they’re not addressed early the chances are high that they will never be addressed. And that’s my concern.
I am most concerned, for example, there isn’t a shred of evidence presented anywhere about the effectiveness of any of the SCOs. As someone concerned with instruction, what’s the point of having six million SCOs if nobody knows whether they’re effective? This is not an issue which can be solved quickly; I’m aware of the difficulties, but I would like to see you begin to address these enduring questions. Because they’re not easy, but for example: Why don’t we have a piece of information saying how frequently the particular SCO has been used? That’s awfully crude and obviously doesn’t address the issue of effectiveness very well, but it’s better than nothing. If somebody gets a SCO which nobody else is using, well be careful. But if you get a SCO that’s been used four-thousand times, well, you know at least other people have confidence in it.
The second piece of information I’d love to see in a SCO is what is the relationship between pre- and post-test score. In other words, right after the SCO was used is there any information that students using this SCO gain some capability, whether that capability be affective, motor, or cognitive; doesn’t matter. There ought to be some piece of data kicking around somewhere. If there isn’t, create the pressure for producing such data. As I understand it, when the SCO is called, space has been left to record those data.
SCOs are a marvelous tool for developing instructional materials, and there are technical marvels in having come so far. But you get to a point where you say to yourself, you know, technical marvels and all that, do people learn from this stuff? And by and large SCORM is silent on that issue. Which is a shame.
I have nothing but praise for the technical accomplishments of SCORM. But let’s address issues dealing with learning and instruction at this stage. We can’t solve these questions, but let’s pose questions so that innovative people will begin to think about them and collect data to address them.
There are more important data: What is the prior knowledge assumed by a SCO? When an SCO is used — some prior knowledge is generally assumed; what is it? If we don’t have that, it is very difficult for an instructional designer who knows anything about instructional design to determine how to use it. Because what is usually wanted from a SCO is to take the student somewhat beyond the competency they came in with. So you have to know what they came in with! We don’t know that. SCOs are completely silent on that. Again, that’s not an easy question; I understand that. But let’s begin to think about it. Because otherwise, you will have this technical marvel of things that are wonderful, but we don’t know whether they work, and what we need in order to make them work.
The third item on my list, which I’ve devoted a good part of my career studying, is are there individual differences in the effectiveness of instructional methods, or objects in this case? In other words, do some instructional objects – are they more ideal for one type of learner or another? For example, do some learners fare better with SCOs demanding induction, while others do better with those requiring deduction. Again that is a hairy topic, and again that’s not a question you can solve. But it is a question we have to begin to address.
Now I understand there are spaces left in the SCO for including data dealing with student characteristics, not only prior knowledge, but for example, what is the student’s meta-cognition like? Meta-cognition deals with the student’s ability to manage their own cognitive resources. I have done twenty-six studies dealing with the student’s ability to assess accurately their prior knowledge. Why? Because good students obviously are very good at this, and if they know something, they either skip it or skim it. Poor students who don’t know that typically try to study everything, and you and I both know nobody can study everything. This is one of the reasons they remain poor students.
I’m belaboring you with that because by and large, prior knowledge as a variable has been extensively studied. I’ve been studying and writing about adapting to students’ prior knowledge for more than 30 years. Research suggests that students who are reasonably knowledgeable about a topic can do with a very lean version; maybe even an outline or a summary. Students with very low prior knowledge, really need detailed instruction. This is a relatively simple way of beginning with one of the major goals of ADL, which is individualizing instruction. In order to individualize instruction, you’ve got to know something about the person; you can’t individualize it without having that knowledge. So that’s another thing I am suggesting that needs to be addressed at some stage. Obviously, I have an ax to grind, because I have written, for some time that it is useful to adapt instruction to individuals’ prior knowledge. Where and how do we include this in SCOs?
None of these things that I’ve raised with you are easy, none of them can be solved immediately. It is true that other types of instruction haven’t dealt satisfactorily with these issues either! But, these issues must be addressed at some stage if ADL and SCOs are to become major resources for designing instruction. And there’s an old expression which says, “If not now, when?”
We presumably need some sort of central repository to feed this back to the person developing the content or others.
Right, and Dexter has led me to believe, and I’ve read stuff that Dexter and Phil Dodd have written that there is allocated space in SCOs for some of this information, for student’s prior knowledge and information about the students’ abilities. Whether it be meta- cognition, whether it be intelligence, whatever. I don’t know to what degree any SCO has used that, and I’m suggesting that it’s important to begin to address some of this saying, Hey, why don’t you include data about students’ prior knowledge, their metacognition, other student characteristics in SCOs? That and data about the effectiveness of the SCO can eventually be used to adapt it to student characteristics. Individualized instruction almost assumes that different people are optimally instructed in different ways.
These are things that we need to begin to address if SCOs are going to be instructionally useful. At the moment they are useful for the creation of instructional materials, very useful. Obviously, that’s why people want to jump on board, because they’re going to save money. That’s fine, I have no problem with that; now, let’s see if the increased ease of creating instructional materials translates itself to improved learning by students. Even if the learning is not better than it is in the classroom, that’s also good. Because we can then study cost/benefit data to determine if ADL is going to be cheaper than having a person standing in front of a classroom lecturing the 30 people. So if you can demonstrate that the SCOs are as good as in- class learning — I think that’s an important finding to have.
So if they’re cheaper, even if they’re as good, then in a sense, they’re better.
Ease of access etc., more people, it’s obviously the cost savings related to having more people to use it. And also, one of the things that Dexter has been keen on over the years is learning anytime anywhere, which has been built into ADL. Dexter can’t speak too forcefully about these issues, since ADL is his brain child and Dexter is much more aware of the technical complexity of what has been achieved, which I am perfectly happy to compliment people on. I’m saying let’s take the next step, let’s relate these technological marvels to people’s learning, because ultimately that’s what SCOs are really all about. That we hope people will learn at least as well, and obviously many people feel, hopefully more effectively than we can in the classroom, with one teacher and thirty-five students. I’m from Missouri — show me! That’s what I wanted to preach to you! Now I am happy to listen.
Something people have done with SCORM, is to have pre-tests and post-tests, so within that, you can assess where a learner is and you could present them with different SCOs at that point or, you could have –
Where are those data located? Do they come as part of the SCO packaging?
Well, no. SCORM is silent on that. People have to go and add, they could add a SCO which is a pre-test, but there really isn’t a SCORM concept of a pre-test.
Wouldn’t that be useful to consider having that information, and finding a standard place where it’s located. Which would mean if it isn’t there, that means we don’t have the data. But if there are data, they ought to be summarized somewhere within the SCO. Gradually, what I’m hoping for, is that as users begin to do that, gradually it will become matter of fact, a regular practice. The thing that’s upsetting and disturbing is that no one seems to have talked about that. I haven’t read every white paper, though I’ve read a sample of them, and they’re all very interesting and deal with very important and issues. But none of these issues deal with human learning and instruction.
I’m trying to think of what other people have suggested that could relate to this, — we need to address competencies somehow in how we map them; and I think that goes at what you’re saying, how in order to identify, if we expect the SCO to adapt to what it presents, or if a different SCO is going to be selected based on a learner’s knowledge, then we have to have a standard way of defining what the learner’s knowledge is, and that would be some sort of agreed upon competency model.
I’m not sure we’re ready for a standard way, because I think there may be different points of view which make that complicated. The major issue in the field right now dealing with instruction. Constructivist folks by and large deeply believe that a learner constructs his own knowledge, and we have to stand outside and help him do that. Whereas those emphasizing explicit instruction really believe we have to do more for the learner than that. So, you’re not going to get a lot of unanimity in the field, largely because many of these issues are still up in the air. But I am saying, it’s important to begin to address these and say, “Hey, what do you know about how effective this SCO is? And what do we know about the kind of knowledge and skills and attitudes we are assuming the learner comes in with. And finally, the question of can the SCO contribute to individualized instruction, meaning teaching some individuals one way and some individuals another.” That pretty well summarizes the concerns I have.
If we can’t really standardize a way to define what the knowledge is that the SCO is teaching, what sort of knowledge it expects, what sorts of competencies it expects, then would you envision that someone, an instructor, an LMS administrator, has to get involved and kind of look and say, this is the other instruction that these people are going through; I know that if done well, this instruction over here, that they should be coming in at a higher level on the SCO and somehow then put that information into the system?
What you’re saying makes perfect sense. The problem is that SCOs will be used in many different ways, and there is very unlikely to be one standard prerequisite, or two, or three, or four. Because the chances are people are going to use SCOs to the degree that it makes their life in creating instruction easier. If it doesn’t make it easier, they’re not going to use it. Now what we’re saying is, in addition to making it easier, let’s begin to think about whether it’ll be more effective than anything else out there. And, where are we going to store these data, and in what form? Like for example, data may be one thing, and include a pointer to a place, to a repository where these data are might be found. Because the data can take up huge chunks of space, probably more space- consuming than the SCO itself. So maybe you want to create a pointer and have a depository somewhere where people can post data of this kind. My guess is that initially, the data will be relatively loose, whereas gradually, as people begin to think about this, the standards for the data will probably toughen over time. But I’m willing to scale back, mainly because when you have no data, you don’t want to immediately mandate what it should consist of. And people will say, “Oh this is too complex, I’m not going to do it.” So let’s say, let’s make it easy, and let’s be relatively free with regards to the shape of the data. And then as people begin to include such data about the SCOs, at that point we can start asking — hey, we would like the data at least to have elements A, B, C, D, E, whatever. But I think it’s important to begin in a modest way so you don’t scare off users.
So that’s the data that’s tracked by the SCO, but similarly, it sounds like you envision the same thing for the data that’s passed to the SCO about what this learner already knows.
I think calling other people’s attention is a way of beginning this discussion. We obviously aren’t going to finish this discussion, but I hope in this version that these issues will be addressed. Because in the past, it was the enormous technical difficulties in implementing SCOs and having them run on any system, these issues were pretty much relegated to “We’ll get to those issues dealing with student learning later.” As I said before, later is now.
