Key Points:
- SCORM working really well as-is
- JavaScript API is beneficial in that it allows accessing the code and seeing what is communicated
- Moving towards more assessing simple tasks within simulations
- Would like an API to communicate between Flash and the LMS
- Disconnected content support would be useful, need for mobile is inevitable
- Should be able to easily report on any tracked data in LMS
- Sometimes have content report via email sent to a person responsible for collecting scores
- Reason is to be able to have the content portable, outside the LMS
- User-defined variables would be useful, for storage, and reporting
- Important to measure on-the-job performance changes after training
- Interoperability, ensuring vendors adhere to the standard is important
- We’re going to see (in courses) less being said and more being done
- Learners should be shown the “risk” (why they should care).
- “I am hoping in the industry there will be a greater effort around making people feel what they’re doing, making them feel invested, like something is at risk and on the line.”
- There will be heavier use of simulations that are more detailed and accurate
Would you mind briefly describing your role within your organization and your use of SCORM and the standards, or related activities?
My function has been, for the better part of ten years, designer, developer, programmer. I’ve been really everything from a page of notes being handed to me and to a PowerPoint being handed to me, everything from the speediest, to be put in front of a PowerPoint, being in front of an Articulate presenter, to the LMS, to building things from scratch, to closely following in Articulate, I started in heavy involvement with Captivate, and trying to move towards Captivate and Flash for more flexibility. There’s really not anything I don’t do on team, including supporting other members of the team and other developers and designers that may be scattered all over our environment. It really is a kind of jack of all trades job at this point.
Early on, we were in an environment that was strictly AICC, because the LMS version we were running at the time, they couldn’t seem to tune it to do what it was supposed to with SCORM 1.2, so I fought it and fought it and finally, we got into a new environment.
We wound up with more flexibility and a lot of those restrictions came off and I’ve become the big SCORM advocate on the team and among my community, because a lot of them fought it because AICC seemed to them to be really simple and easy to implement and SCORM required more understanding of the fundamentals of what was being communicated between the content and LMS. I personally pursued it because I found it was enormously flexible and with the JavaScript in between, I could actually access the code and see what was being rather than trusting what was hard-coded into my course with the AICC. It was more visible and out in front of me, and it offered a lot more granularity as far as what I could capture and report and so. A lot of people continue to fight it because like I said, it’s been their experience with this particular LMS structure that AICC is whiz- bang, and because of the need to know and pursue and understand more technical aspects of what’s happening, people have avoided SCORM, I think, out of intimidation, and I’ve been eating it up because I gained an understanding of it pretty early. So we’ve had an LMS experience over the past 9 or 10 years that scared a lot of people off. But I’m slowly steering them over to what SCORM can do and the power in that platform.
You mentioned the JavaScript API as an advantage in the sense that it allows you to have a place to hook in and see what the course is actually doing in terms of communication; but it sounds like with AICC the course tends to directly post the results back and there’s no way for you to get at it.
No, the major differences between the two standards, one of the main points I have made early on to my team is showing them the guts of AICC and the guts of SCORM and basically that the knowledge and tweaks that each of those four configurations files require and the understanding they require, and basically tell them that this standard hasn’t been touched in years, and I’ve kind of watched it progress, and it had kind of been left in the dust while this JavaScript middle-man approach evolved and when people get into the guts of AICC, I think they’re more intimidated by – ‘s nothing intuitive to be found in those files. When you look at them, you really need to, if you don’t know AICC speak, you’re in big trouble. If something is missing in those files that someone wants, and so on, then people are generally at a loss for understanding what to do with those. And having four of them required to get it done, versus an IMS manifest and a couple of JavaScripts, I personally — if I don’t have to, I’m never going back. It’s too bottled up, closed off and constricted. It’s not letting anyone know what it’s actually doing, and I’ve had a lot more success with SCORM than with the Test Bench tools that are out there and available and I really like being able to look into what’s being passed before I subject them to an LMS.
How do you go about doing that? If it’s JavaScript based, do you go in and kind of present your own API to the course and have that pass on to the LMS, or you mentioned Test Bench tools?
I haven’t had to do much in the way of custom APIs, I mean what we’re doing — it was interesting [reading the forum, people’s requests] because we are moving, as a design discipline, we are moving towards simplicity. I don’t need SCORM to do the dishes for me and that kind of stuff. Our designs are more intuitive and moving more away from multiple choice, true/false kinds of quizzes, at the end of a big PowerPoint splatter and we’re getting more into having people perform tasks and actually grab scores and values on interactions they perform within a simulation environment and making it more relevant to the job than the traditional quiz with a finish button. We’re trying to, as much as possible, get the user’s mind disconnected from the traditional click click and if you don’t get 80% you’re not complete, etc. We’ve been addicted to that for years, and people are starting to see there’s really not the value that they once saw in that. And so what I do a lot of times, one of my favorite things, and it’s a request I would make of any new iteration of SCORM, if it could be accompanied by a really nicely designed test suite, test application.
One of my favorites has been the Reload player, and I’ve had great success with that, I have not only as pre-LMS testing and understanding what my variables and contents look like, but also as a teaching tool: I’ve taken it out more than once to instruct the team on how SCORM works and the types of naming conventions are being used and what data is being contained in those variables and how they’re used before hitting the LMS, because once they’re in the LMS, nothing is really visible to them anymore. But I’ve been able to not only test, but show them the importance of various settings, because as I get new people into more complex tools like Captivate where it will basically allow you to make a series of settings to adapt it to pretty much any system, they have to know more, of course, on a technical side and they have to know the consequences of that radio button, and where to put it. And so testing a simple, a testing suite that looks and feels like Reload because Reload has been tremendously intuitive and it was easy for me to get started in it. And the way they lay out the activity tree and so on is brilliant and is easy to see, especially when something is going wrong.
So keep in mind, again, take this with a grain of salt because our environment, we exist sort of one step back from everybody. And where are now at SCORM 1.2, we have been told an upgrade to 2004 will take place sometime in the next nine or ten months, and of course that will take us to all the advantages of that one. So anything on 2004 I’ll be ecstatic about. I am looking forward to 2004 and a lot of people have had it for some time.
I haven’t found situations where I really was fighting the limits of what SCORM can do to the tune of a lot of comments in the forum. Where they are saying it’s not reporting enough detailed data, and I thought — I’m not really giving it that much. Because my design is probably over-complex if I am. ANd so there’s a balance you have to strike. But I haven’t had to customize APIs — I think something I’d like to see and move toward, and am moving towards personally this year, is more working in Flash and understanding how to get Flash to pass information to SCORM and to an LMS and sort of get into an unlimited environment that way. And Captivate is pretty close to unlimited for what we do here. But Flash would be wide-open doors, and the trick is building the API to get that out of Flash and into an LMS, effectively, and reportable. In the version, if you could simplify that or what’s around Flash, that would be outstanding.
The other items in the forum that I thought would be great ideas, things like disconnected content, and cacheing results and then passing those during a later session, would be useful to us. I think we’ll inevitably find ourselves in the realm of mobile devices. I would like to be there and that will land somewhere between Captivate and Flash, more than likely, but our demands as far — I can’t think of a day when I would go out there and build a piece of content that would absolutely hit the limit of what SCORM can do right now. I just don’t have that level of complexity that’s needed to get the knowledge into their heads. The reporting, if anything, we’re getting away from scores and instead tracking: how long did they spend on slide 3, and how long did it take them to arrive at a successful response. That is something already trackable in 1.2 and they do a very good job with measuring latency and seconds and so on — if our design in the way we are trying, and we try to adhere heavily to Michael Allen’s approaches to all this as far as context and relevance, etc., but I don’t think I’m going to blow anyone’s mind at ADL.
I guess in that case, the other thing we want to get are what sorts of things are working well, that you would want to make sure not to lose in a future version, because there’s a big push for simplicity that a lot of people are asking for. And you mentioned that a lot of people are asking for SCORM to do the dishes and I’m pretty confident that we’ll be looking at something that is more simple What are you really looking forward to using in SCORM 2004?
The one thing I’ve keyed in on again, because we’ve tuned our approach and our complexity around 1.2 for so long, one of the first thing to catch my eye was that 2004 can store a longer string for location data. The more detail is able to be inserted in that and of course, the system would interpret a longer string and be able to handle a longer string, because — with location, it has to do with how it’s written and how it’s passed, because I try not to load a tremendous amount of pages into an activity, I keep it brief for retention, and flexibility around that is great. But other than that, I have — what we’re looking forward to is being up to date and having more stability. Being more standard-adherent, because when they do the upgrade, they’re going to jump something like six versions.
What edition of 2004 will be supported?
I’m not sure if I can get that back to you because everyone has been pretty tight about handing information from that end. But I think, like I said, it’s going to be about a six-version jump in Docent, so what I’m anticipating is a lot more stability, a lot more ease with publishing this stuff and getting it into the system without it having problems interpreting my values and passing them through like it’s supposed to.
The items I would really like to stick around are something similar to the way it handles now — interactions, and the time differential for each of those interactions has been useful, because when we get into more behavioral measurements, we’re dealing a lot with detailed reports. And we only did one or two, we didn’t want to overdo it, but the reports how long someone spends on the interaction, how long, again the standard, and how long it takes to arrive at a correct response. And that detail recording as it is has been terrific. Storing is fine the way it’s captured now because, design-wise, scores are going to wind up having different meanings then they used to. We’re going to have to assign more scores to activities, rather than build traditional questions. The total time, the lesson mode, the session time, everything I have had to date has been fine, and I would love to see it stay.
What we tend to report, when they aren’t reporting completion, or complete/ incomplete, pass/fail, in a single score, we are trying to get more into detailed interaction scoring and the way it stands in 1.2. The data format has been fine for what we’re doing, weighing those numbers against each other, because we’ll go ahead and do that in Excel, etc. I think if there was to be an improvement in the way the LMS publishers implement in the new version of SCORM or iteration of it, would be to make reporting customization as flexible as possible so we don’t go through this stressful process every time they want to add data to the LMS every time they want to report a score. My attitude about it is it would be great if they came up with something on the admin side because very few of them are actually programmers or seriously technical. They just know how to run the consoles. It could be a drag and drop reporting interface, a drag and drop build of reports. Here’s every available field, not only with the technical variable name within the SCORM standard, but a plain English tag that tells the non- technical user what the field is capturing.
As simple as they can keep getting the data out. And having a serious, stable, non-LMS method of getting the data out, that’s easy to configure — email reporting, etc.
It would still be the LMS sending you the data, but in an email?
Yes, and basically if we had a content piece we were going to put instead on a web server that we use extensively for situations where either scoring or reporting isn’t required or we’ve got content within the course that has to be externally linked, we point it out to this web server. And ability to easily, have a learning activity compile the results and basically send them to a pre-determined person via email, even if it were completely disconnected from an LMS. Having an email service that would basically connect with your client and have that data and send it off to somebody. I’ve had that request in the past, where they don’t want to deal with an LMS, but they want one guy to sit there and sort of collect scores. And that’s not something I encourage, but I think it has to do with them wanting more flexibility location-wise, they want people to be able to be disconnected, to roam around and still have that data land somewhere. And if that data were relayed over email or the LMS, whichever was best, that would be great for portability’s sake. We have a lot of remote people who aren’t always on fast connections with our software that everyone uses and they might appreciate the ability to download everything locally, and work locally, and still retain the data.
As far as exotic things, one of the more exotic things I’m working on right now that might give you an idea of needs for types of data in the future is I’ve got a situation now where I’m going to be testing and building a 10-key simulator. We have situations at the bank where they are doing document processing, going through and reconciling checks and deposits and so on. So what I have to do is And basically put a key code, an account number in front of a student and having them key that number on their keyboard as fast as possible, measuring their accuracy, time, and number of key strokes, etc. And having the ability to translate that into a variable or type of data that can be passed through SCORM to have a meaningful outcome on a report. So in the future, being able to take conten calculations, and pass them through as something the standard understands and can do something with, and that translation will more than likely have to happen on the Flash side of course, through action scripts, etc.
That kind of flexibility will be really useful to us in the future, because like I said, the scoring will be very non-traditional as we move along, it won’t be the numbers everyone has come to expect, because those numbers most of the time don’t have a lot of meaning as far as can they do the job. So if the new system were to have flexibility about that, and I think someone put in there, which I really locked on, was user-defined variables. That would be tremendous. I could name that variable whatever I want and put whatever I want in a container, and could have it picked up and at least, if nothing else, retained by the system and put somewhere in the table so I can go get it later, and assign it some meaning and report it. That’s kind of the — like I said, where I see this team evolving to, because we still have people kicking and screaming for 80% or better, but it doesn’t carry the weight it used to and they’re starting to understand 80% or better has little meaning it terms of job performance. It might get you certified, it might get you signed off on compliance, but it doesn’t do much to tell me how you do your job. And we’re getting heavily performance-support and performance-based about our design. They’re going to be very, compared to the rest of the bank, the majority of developers, they’ll be nontraditional.
Our flexibility in the future will be more around the capability to capture numbers that relate to performance and away from traditional mastery scores and those types of standards that are currently there.
Are you looking to do that directly with your content, your instruction, and assessment? One thing that some people have mentioned is tying in on-the-job performance, so do you have any thoughts in that regard?
That would be a tough nut to crack, but I am in complete agreement with that. As far as meta-data and capturing a change in behavior, I think we’ve talked about that within our group, as far as what’s an assessment level? Because we’re still hung up on the 1, 2, 5, and all that. But the one has happened, and from the impact on the business standpoint, it’s relatively meaningless, but what we’re getting into now is we are having more and more conversations about what the lines of business who actually own these and these systems who say, How do I know that it happened? In other words, we have have help ticket system, so if I’m teaching technology connection, which is our help desk, if I’m teaching Tier One analysts a particular skill in troubleshooting, and my goal is to reduce the number of escalations to Tier Two, I should be able to go into the ticket system and say, How many were escalated to Tier Two in the last six months and how many in the six months following the course were escalated to Tier Two and see some impact.
That will be considered the only valid end to what we’re doing, is that something changed. You would be surprised how few of these conversations, even recently, have taken place. Somebody should have been sitting down with someone in a line of business and saying, What is happening that shouldn’t be, or what isn’t that you’d like to happen, and how do we know that it happened? We’re realizing that — I had a conversation with my team and told them frankly that these are the things that justify me being an employee here. If I’m not making it happen and I can’t know that it happened, what am I here for? But again, this goes back to Allen and several others; if there isn’t a problem or a risk of something occurring that shouldn’t, why am I being brought into this project? So the measurements you’re describing will be essential, those two: during the activity, and then being able to go back and capture metrics to say that in the real world, it is currently taking place and here’s how well it’s being handled. And if there’s some way to land that — it would be tough if someone wanted to push that to an LMS — I wouldn’t see that need but I think maybe a structure or set of guides, or best practices, around a new SCORM standard, that would contribute to that standard being well-rounded. To say, we’re giving you all this LMS stuff you want, but have you considered the total approach being changed, that your 80% is a placebo, not really doing anything for you. I think this is important, and that in e-learning, if something isn’t occurring that should be or vice versa, then why in the world am I getting the call. And it’s brutal truth, but I think it’s the foundation of the whole thing.
If I can build 20 courses I may only feel good about two of them. Because two of them were more meaningful and the rest were “e-sedatives” and if I’m in the business of building learning deterrents then I’ve got a serious problem. With my clients, we’ve talked about different approaches and I preface it with, “This may be painful and not something you are accustomed to, but here’s what we’re going to try to do with this,” and practicality, and less is more — and am I overtraining? And when we get into SCORM and reporting standards, and data being handled, etc., am I over-measuring? Or not measuring enough? Am I building end-data overkill? The forums are fascinating because there are things in there that never would have occurred to me because of this more simple path I’m going down. I think it needs to have more flexibility built in, the new standard, and if it’s stable, that will address 70% of my wish list. If there’s flexibility around variables and is stable, and the part you can’t control are the publishers adhering as closely to it as they should.
What do you think will change in the e-learning field in the next 5 – 10 years and what should change?
I think you’ll see things get a lot more visual, I think, and again from our experience, less talk — and the initial move was no text on the page, and narrated, and make it available in this company — less being said, more being done. They’re going to ask us to do more, I think — One of the lessons I learned from Allen Interactions, which was really important, was the value of introducing risk. I think even now, as advanced as people are in the industry, precious little risk is introduced in content. Joe Student, here’s why you should care about what I’m presenting, here’s what you have to lose, here’s what you have to lose if this discipline we’re teaching you doesn’t sink in.
The risk used to be that you were required to complete content by an arbitrary date. That’s not – -I went back with several people and said, That’s not good enough. That makes me click next and yawn and quit as quickly as possible, and it doesn’t tell you about how well I do anything.
So if the risk is going to be risk for the business, because the risk, or lack of, really gets down to should I even be building this for you? If you can’t articulate some sort of risk or damage potential if I don’t build it, maybe I shouldn’t be. It’s hardcore practical, it’s a tough question sometimes, but I think you’ll see a lot more creativity around interactive; scored objects rather than traditional question and answer. I’m really losing my taste for multiple choice and true/false. Those are great, but when they sit at the end of the content, and I test them on content they left two minutes ago, I am not measuring anything as far as did I design anything decent that can be retained. My 80% score has little meaning that way; but if that score was arrived at by having them go through the process in a simulation of configuring a web service, they’ve seen the environment, they’ve performed interactions, and not only that, I’ve designed in opportunities for them to screw things up along the way. My 80% now has meaning, and I think that maybe I did a good job of teaching them because as far configuring mime types, he was in there a total of ten seconds, which is good, because he’s never seen it before. He had an understanding of what he was doing, and my metrics are getting closer to whether transfer had actually happened.
This is discussion that is ten years old, I just don’t know if a lot of people are actually doing it. In the future, I think the reporting will take on a different meaning because it’s no longer just simply to satisfy a requirement in the project, but numbers will be looked at repeatedly because they reflect behavior and have a meaning as far as what people know how to do and what they’re expected to do, rather than, did I simply cram these 300 people through by a certain date. So I think you’ll see a lot more graphical, interactive content; I think you need to. I think you’ll see heavier use of simulations that are more detailed and accurate. And I think that the more we design content to put stress on the student when necessary, it has a huge value in retention.
One thing we went through with Allen was simulations that had to do with correct or incorrect conversations with high risk employees. Another had to do with literally a Flash-based activity where I was cooking a meal for a customer at Denny’s, and if I didn’t cook the right things in the right order in the right amount of time, the standards were missed, the customer was dissatisfied and went somewhere else. Even as a lay person going in, never having cooked at Denny’s, the clock ticking and my customer on the line, the best way I could put it to my management was that they made me feel it. I am hoping in the industry there will be a greater effort around making people feel what they’re doing, making them feel invested, like something is at risk and on the line, if that means what I’m working on now, that has to do with checking over a check and determining whether or not it’s negotiable, having a meter up in the corner that’s telling them with each mistake, how many dollars are being lost. Making the loss or gain personal. A lot of it has to do with design, because without good design, you won’t get meaningful numbers in SCORM or anything else. I think there will be a greater burden on designers to put themselves in a seat and one of the first questions I have now when I sit down and get on the phone with a client is to say “Why do they need to understand the lesson? Is there job hands-on with the material?” I continue to be surprised that these questions aren’t being asked, even though way back at Learning 2002, or whatever it was in Orlando, everyone was harping on it. The traditional approach is an addiction, and I hope in the next few years, that addiction will start to break and become instead, How can I ratchet up what I’m considering building here so I can make them feel it and own it? Your scoring coming out of that can be very simple, as far as the format and what it means, but I can get all those meaningful numbers if my design is good.
