Key Points:
- We currently track simulations, games, virtual worlds using a web page as a SCORM bridge
- Simulations contain random events, this complicates assessments, objectives
- Sequencing is too complex, should be able to have state of other SCOs available and manually sequence
- There is a need to move away from the JavaScript API
- Teachers or administrators should be able get information about students, then assign grades
- There should be a way for course to report on data from the LMS
- Packages get too large, work around is to create smaller packages, but then it should be possible to sequence across them, and treat several of them as one unit.
- Global objectives shouldn’t need to be mapped, it would be easier just to access them directly as if they were local.
What new or innovative training or learning are you working on?
Well, at ECS we’re an engineering and simulation company, so our main focus is creating simulations, games,virtual worlds and so on. But they all involve training, of course, so my main role here is to kind of be the LMS guru, so I’m constantly integrating, or trying to find ways to integrate games or dynamic training environments in general with a web based LMS or with a server, so what I’ve done is actually use web pages to be the front end, to launch simulations, games, or virtual worlds. And then to have the game communicate back to the web page, objectives and so on, and then use that web page to communicate back to the LMS using SCORM for things like remediation and tracking the user and so on.
For the simulations and training you are creating, what is it that you want to track about a learner that goes through the experience?
We have, what I’ve built, is called an assessment engine. It’s used to assess simulations and dynamic engines. Instead of just saying, if you got six out of ten right, you would have a like a tree, what we call an assessment model, and then you’d go through and at any given point in time, you can check off those tasks, those objectives, that’s kind of a like a rule engine. So those are the type of things that I try to translate into SCORM. And one of the issues I have is that of course SCORM will not have the hierarchical objectives system. So basically, right now, just taking the top level, did you pass or fail? And maybe a couple other objectives.
So the objectives that you logically create are hierarchical, but you can’t represent that in SCORM?
Right, not usually, I mean there are workarounds and so on. But that’s one of my wish list items I would guess, either to get rid of or have something separate from the objectives system. I think I heard something or read it just by going on the website that you showed us, that there’s a whole data model now that you can store saved data in. I haven’t explored that yet.
Our assessment models are quite complex, because you’ve got a dynamic environment; I don’t know if you worked with simulations before, but if its raining out, it’s a Tuesday, and the guy’s wearing the blue shirt, and he did it under five seconds, then he met the objective. But otherwise, you know what I mean? It’s got all these different branchings and so on that can happen in a simulation, that random events can be happening within it. It makes the assessment difficult, more complicated then straight — did you meet your objective or not?
For example, a combat medic simulation. So let’s say you’re walking down the street in a simulation, and an IED goes off and there’s four casualties around you. And those are random; you don’t know if they’re gonna be, their responses are actual models of the bodily functions in there — blood pressure and heart rate and so on. So you gotta go and treat each one of these four casualties. Let’s say you do badly on putting a tourniquet on, for example, on one of the casualties. But you pass the other three objectives. Well, I’d like to be able to sequence that user back to that lesson on applying tourniquets.
So right now, you don’t have hierarchical objectives, but you could have each of those as objectives, but then you have to kind of set up rules on each one of those different objectives to sequence one back. Is that the kind of challenge you are facing there?
Yeah, I’ve worked around, done a lot of JavaScript and stuff, but essentially I end up with a hack I just created for objectives, and I use them like variables. Local objectives, like variables, I can call them from wherever I am, and then use that to do whatever I need.
I found the sequencing hard to get it do what I want it to do. Not even with simulations, but just with some of the general sequencing schemes I wanted to apply with standard courseware. Maybe it’s a simplicity issue, or it’s a sample example issue or something of that nature. But to be able to do that type of remediation, to set those rules, you know, if you pass, if you fail the simulation, then you have to do one of three things: either you have to go back and take remedial courseware, or you have to retake the simulation again with this new data, or you get to advance to a different section. I just found it hard to do that.
That’s been a fairly common complaint about sequencing, that it’s been hard to figure out. One thing that people have asked for, besides making sequencing simpler is just to have enough information and control to be able to bypass it and do it themselves, when they want.
I think it almost needs an overhaul of the system. If it would be to make it very easy, or just have some type of rules, that you can just turn things on and off, from your own script or your own web services, which we’ll probably get into a little bit. The rules are very limiting. Like in a standard sequence, I want to take the pre-test. If you pass the pre-test, then you can skip all the objectives. I’ve seen examples on how that works, but it never — in front of me, I can’t think of the reason, but it would never work the way I wanted it to. So if maybe the answer would be to say, Alright, we’ll just give you the tools, we’ll just give you the access to the sequencing and then you can kind of build it yourself. I’ve seen on that website, someone mentioned that you could provide your own table of contents.
Yeah, that’s one of the ways of bypassing sequencing. I think another up there is that you should be able to have enough information for more than a SCO that, all that information that sequencing would have. So you can look at the state of other SCOs and make a decision about where to jump to.
I’m on board with all that. I think I voted three points for that one on the website.
Anything else that’s been particularly difficult to track or to do or possibly even that you’ve wanted to do but just haven’t even been able to? Or even that you think you might want to do in the future, but haven’t really needed to do yet, but could foresee a problem?
We have to move away from the JavaScript API. You probably know all about that one. But let me just second or third that notion that all the modern web applications that I work with have this method of either HTTP requests, or other types of web services interface, and that would help while on the simulation side as well. Obviously, they don’t say that we’re initiating, when to send data back and forth that way. Having a teacher/ admin functionality on that, where you would have — so the concept of an administrator role that could get access to information on the students and then assign them grades or approve them, I know that wasn’t the original intent of SCORM. We work in virtual worlds, so there’s kind of like a team aspect to what we do a lot of times. The ability to access information on other SCOs, let’s say, or from other users, obviously the teacher, or whatever title would apply some elevated level of permission that would allow someone to get information on — just for reports for one, we want to make a report, right? I’m the teacher or an administrator of this class, I want to see the scores that everybody has, that are assigned to me. I know that’s kind of an LMS functionality, but you can’t get that information on a web page, you can’t really make your own reports. You’ve got to rely on the LMS there.
So the report is within the SCO and someone with enhanced privileges launches it and can get that report?
I would say, “Get objective for student x,” or “Get all objectives for students for this lesson,” and I could iterate through them and create a report or something with a web page. There’s a lot of people — this comes from my clients, there’s an email there — a lot people assume I can do that. But really, that’s not a part of SCORM. If I want a report, that’s up to the LMS that you choose. And they don’t understand that — well, can’t you build a webpage? Well, I can’t really because I don’t have access to that information.
The biggest problem I’m having now doesn’t have to do with simulations; it’s straight training data. It’s the size of both the manifest itself and how large it gets, because everything gets included in one? And then, the size of the content package it entails, and that everything has to be one package – the latest one I had built was a gigabyte. Then you can’t upload any of those things into an LMS, that you don’t have control over the server, because usually, you can’t upload files that size. So you break them up into different content packages, but as soon as you do that, you no longer have control over the sequencing. So I guess the concept of maybe where they’re going to originally, a sub-manifest or being able to break out things into different pieces, just the manifest itself in this course had seven modules and each module had three to seven lessons in it, the whole manifest was just huge and I had a hard time traversing it. And I was doing it manually, with a text editor. There’s a tool I tried to use, but the tools didn’t seem to work out for me, either. But anyway, to break those things up somehow. To modularize it.
In terms of content, did this tie in, in your mind, with the need to use distributed content when the content may not be running on the LMS, or does the content still need to be on the LMS, it just needs to be updatable and in pieces, and possibly some of those pieces are shared between courses?
I would say both. The ability to have external resources is always valuable, I do think most people though would want to store everything within the LMS so the LMS can load it up. I was thinking about this this morning, how I would do it, and it’s only been an hour, so — like you would have a resource, you would have a directory, right, and you would have an XML snippet, like an IMS resource, actually, all it describes is just that resource, that directory. And you can zip that up in a different file and you could zip up each resource and you could zip up each organization, if you had to, but it would be so small at that point; but you could zip up each resource as a separate module, if possible, and you could have organization manifests and have them all related by ID somehow. And then have one main manifest. You see what I’m getting at? So you could then upload, things separately to an LMS.
So each manifest would be the root of a sub-package you would upload and then you would have a a manifest that would just refer to all those.
Yes, you would essentially be able update the main manifest in pieces, you would be able to add to it without having to repackage the whole thing up. You could upload each piece and say I’m going to create a SCORM package and call it P1, and I’m going to upload some resources to it and they all have the same ID. So the LMS would track it — it’s kind of a major overhaul — but would track it piece by piece then you could upload it. You could upload an organization and that organization will reference some resources. If a resource hasn’t been uploaded yet, there’s just a standard error message that gets displayed. If a particular activity tree you represent, references a resources that hasn’t been loaded yet. I think that may be an interesting way to go. But with all this content that’s coming in, Flash content we’re working with — simulations, video, these packages are just getting huge.
Maybe I can step you through this latest problem I have this one standard coursework package, it’s flash- based, seven modules, and maybe all told, twenty lessons.
So the first thing we want to do is the sequencing I mentioned, so you want to do the pre-test. The pre-test I mentioned, if you passed it, or get a certain grade on the pre- test, you can skip over the preliminary content, otherwise you have to go on to the next lesson: lesson one, lesson two, lesson three. And I had all sorts of problems just sequencing. So we touched on that. But then they want it all on one content-package, and of course, it just got too huge. There were three, four gigabytes, so I broke it out into different modules, seven modules. Well at that point, you can no longer sequence those seven modules because they’re separate content packages. So I got that all right, and I got it down, and shipped it off to them. And they said, well, this last module is too big, it’s 900 megabytes and we can’t change our server settings, so you have to break it up. We have to break that up into individual lessons. Now once I did that, that means in order to keep in the system, I have to go back through all six modules and basically turn them into individual, basically one SCO per package. And just about everything I’ve done with SCORM, I’ve almost ended up with that as the solution and basically break it up, one SCO per package, and then let the LMS decide what happens with it.
At that point, you can’t sequence it at all. You can’t specify you’re preferred sequencing.
Right, so I guess I’m really leaning towards — you would upload the activity tree on its own, in that one use case. And now each node on that activity tree now defines something, whether it be a separate package that you upload, a content or external resource we were talking, like distributed content, or points to, from a simulation launch mechanism, whatever that may be. Whatever that may be — whether you’re launching a game from a webpage, or whether it’s a physical simulator, that instructor then has to set up, and then some way to sequence all that.
Or even, would you say one of the nodes could be a pointing out to the student that they have to go attend some classroom training at this point in the sequencing and then the instructor has to put in their grade, and then they move on to the next point.
Exactly. Definitely one of them. And that’s one the things that I do when I work with my clients in homeland security. We have SCORM packages and they have live classroom. It’s a big hack trying to fit that in, not just for SCORM but because the LMSs we’re using don’t have a concept of integrating live training, classroom training, with web-based content. And now they’re supposed to log in with the LMS before they go the class, take some introductory training; and when they pass that, they get approved to sign up for live classroom — like three days at the Holiday Inn, type of thing. Then they go to the live classroom, and their instructor checks them off that you passed or , then that opens up in the LMS the refresher training or some other course. So yes, I would definitely say that classroom is part of it. I would sell all these things you would be able to set up, have the SCORM API be a web services API, so that somebody through their mobile phone or smart phone or web page or automatically with software embedded into some simulator, send “Student xyz passed this node,” and then whatever LMS is implementing the API will just, mark that down and then the student could log-on to the LMS later and see their status.
What do you want to know about your learners?
I want to see what courses they’ve taken, what scores or what data we’ve stored about them, whether that’s scores or progress or whatever their measures are. The concept of a role may be interesting as well; what roles does that user hold. I haven’t seen that anywhere, really, except for where we talk about a teacher versus a student. But there’s a concept of a type of student. If the student is of type “data entry,” I’m using my DHS client as an example, do they get sequenced through this tree? If the student has this role, an analyst role, do they get sequenced through this branch? I know that’s contrary to my previous request of making everything more simple – To make it really simple, maybe extract sequencing out of the manifest, just have the ability to turn on and off any given node.
I’m just saying pull the sequencing out and define the sequencing and make it generic enough to fit all potential use-cases in the manifest in XML, is very difficult to do. And the more you try, the more complicated those sequencing rules and stuff become. So at that point, well maybe it’s should be something that should be done in XML, maybe it should be something done in code. At which point, you’re saying maybe we should pull it into JavaSCript. And keep basic sequencing, maybe the basic sequencing strategies just turn on, just call it the sequential strategy, each one has to be done, and that’ll the 80% rule, and the other 20%, no matter how many rules you try to define rules in XML, to my opinion, just gets too complicated. If you have enough grasp of logic anyway to do those complicated rules, you probably already know how to do script. So just to be able to turn, disable — two things, make it real simple: just be able to turn on disable/enable or hide/show any given SCO for any given learner, and the ability to jump to any, to force that user to jump to any particular node. Just having those two things would then allow the content provider to have total control over what they need. It’d be adding, well here’s my HTML for my table of contents, I want to control a table of contents, then you have everything you need. That’s how I’m coming at it. Now I understand the LMS vendors will be saying they want control, but as a content-provider, I’m thinking the other way.
Flash-content prevalent is a note I have. Maybe that ties in with the API?
Flash content shouldn’t have to talk to JavaScript, which then talks to the LMS?
Yeah, that’s always the solution. And content developers and Flash, never know anything about SCORM, so you got to explain to them. You know, you have to tell me when you reach the last page, you have to tell me when they pass, so it’s easy to do you — but that was just a note. There was a web services API that Flash content could call it so you wouldn’t have to build the JavaScript.
I had written the whole global objects concept — where you have your standard objectives, your primary objective, you need to map it to a global objective, if you want to be able to read that objective from other SCOs. If I were to do it, I would just say, All objectives, you have a flag, is it global or not, you could say just say true.
And if it’s global, as opposed to a mapping, you can just read it from anywhere is what you’re thinking here, right?
Two SCOs, one reference to the same objective, you just use the same ID. that may tie back to where you have this concept of a distributed content package. Maybe I can coin a term there! But a distributed content package is where everything is separate so you could maybe still achieve that effect. The resource is really the SCO, but you define that separately from the SCO which defines objectives, so when content vendors want to package, they don’t package a SCO, they package the resource. And then somebody can take that resources and that resource can say, Okay, these are the things I expose, these are the data points I expose. And then you would map those data points within the SCO that you create your mapping, can choose to listen to that data point or not.
So, I’m telling you this because I’ve done it before in simulations, I’m trying to define an assessment model for simulations that’s generic. I have this concept, where I’m training you and you have to walk into a room and flip the light switch three items. Well, there may be six different simulations that may implement that concept, well I want to create an abstract concept of that light switch, or that event happening. So maybe you cold have a resource and you’d say, This is my training resources, I’m going to expose pass or fail. Or I’m going to expose a score. Or I’m going to expose a number of things. And then the SCO decides what data points within the resource to use and take advantage of. And your separation would happen that way.
So you could define in the LMS, via SCORM, a lesson, a leading control lesson, not knowing exactly what kind of content you’re going to use to fulfill that SCO, so that’s kind of a node on a tree that doesn’t have any content. And now, I’m going to go shopping and find pieces of content that do leading control, and then upload them. And I’m going to map whatever my objectives are in my SCO to my data point to my resource.
That’s kind of like your sub-manifest approach. Except now it’s not a sub-manifest, it’s just a resource.
If it sounds like I’m making this up on the fly — well, I kind of am. I imagine you’re in the LMS, , you upload the manifest, but its a shell, so you are basically uploading the activity tree. Then you click, the LMS has to provide the interface so you can click every node, and you upload a piece of content to facilitate that node. And then that content will expose data values that you then map to your activity tree. So instead of having one big content package, you have a distributed — you upload the manifest separately and then build each piece of it.
Maybe this is a way to look at it: right now, you upload everything at once, a whole manifest — but almost every LMS I have worked with has a way to create, outside of SCORM, it’s own mechanism to create courses. Whatever mechanism you have, I’m going to define a top level module; now I’m going to define a lesson. Now I’m going to build content for that lesson. Why won’t SCORM move into that area? Where we kind of standardize the tree, you build the tree first, so there’s an API to create an activity tree. And upload an XML to manifest. And then after that activity tree, the separate process is an API to upload a piece of content for a node on that tree. And then, different LMSs could implement that different ways.
What about a roll-up mechanism or a way for the different nodes of the tree to communicate, and is there logic within those nodes themselves to essentially — do we actually have roll-up rules, or does a top level node query the other nodes, and set it’s own score based on some internal logic?
You’d still have the roll-up rules — I haven’t really used them so I don’t know too much about them. But you could say the same — just a simple change that I kind of am getting excited about right now, is you would be able to upload that manifest separately, from the resources. So upload on organizations just an XML file, and the LMS will just create this shell of the table of contents, if you will, and then you would right click on any node and say, okay, I’m going to upload the resource for this node, or delete, or update the resource for this node. And that would solve all those size problems, I would think. All the problems with that huge manifest you can’t really .
Anything else you haven’t had a chance to say yet?
I just want to add a vote in for one thing that’s already up there and that’s UI differences. LMS UI differences? Whole frames and tables of contents, and navigation controls. Let me just second everyone’s heartache with that one.
