- People want to break out of the LMS, incorporate social media.
- SCORM is focused on a single person, but there is demand for multi-user experiences that can be quantified.
- Knowing the learner role would be useful, eg: to only present relevant portions of a large course based on job function.
- LMS lack of reporting is a severe problem, there should be a way to report on everything. For example, suspend data can be used for troubleshooting or to store an unanticipated value.
- A SCORM reporting API separate from an LMS’s reporting tool is not enough, managers want to see SCORM data in context with other LMS data via the LMS’s reporting tool.
- Everyone has their own way of handling learning experiences and tracking.
- Framework / wrapper support is important for a web service API, particularly SOAP; otherwise, people will have to write their own code and run into trouble.
- SCORM can already handle many tasks that people are requesting, they just don’t realize it. Authoring tool support may have a lot to do with this.
- Rapid development tools will continue to be used.
- It would be very helpful if authoring tools exposed more SCORM features, because many users are limited to what authoring tools provide.
What new or innovative things do you think people are trying to do, and particularly, where it’s kind of pushing the envelope of what can really be supported with current standards?
I guess there are two things. One is breaking out of the LMS, which is pretty well documented. People want to bring social media and other technologies into the picture which don’t always work in an LMS setting. The other is to multi-user experiences. SCORM is currently focused on a single person doing one thing. I see a lot of people wanting to push SCORM so they can have social media interactions with other people and somehow quantify that.
Is it necessary to track the social media? Because I’ve been hearing different facets of that. Some people say it’s only necessary to be able to find it, and I guess I’ve heard this more about informal learning, which social media sometimes, frequently, falls into that category.
I don’t think anybody really has an answer. If I ask ten different people, I get ten different answers. I think some people want to have social media and be able to track it. I think some people don’t care about tracking and they just want to be able to access things, for instance, bring a Twitter stream into a course, or something like that. And then I think there are people who don’t care about detailed tracking, but they do want basic tracking, such as “was this completed?” or “did this person participate?” But that doesn’t get into the more granular data that you could handle with an interaction model.
What is it you mainly hear about people wanting to track? Aside from social media, and particularly, are there areas where they feel like tracking is inadequate?
The vast majority of the people I hear from — who tend to be newbies and not die-hard developers like John Campbell — tend to only want to track the bare necessities. They want completions, they probably want a date associated with it, though I imagine that would be automatically done by the timestamp in the LMS. Some people, from time to time, bring up quiz results. But the thing I hear most is “completion, completion, completion,” and also “suspend data.” People like having that one field they can use for whatever they want.
I think people find the SCORM 2004 model of completion — success status and completion status — more confusing than SCORM 1.2’s lesson status. Because in 2004, you have two different data elements you need to use. And in 1.2 you only have the one. Newcomers find 1.2 easier, although the more advanced people find 2004 to be more appropriate.
Have you run into people want to know things about the learners that they aren’t getting from the LMS? Like, at the moment, you would get the learner’s name and ID; are there other bits of information that they want?
I’ve definitely heard people over the years talking about being able to bring in other bits of information. I know it was mentioned on the Tin Can User Voice site, where people, for instance, are wanting to access the results of other training. I can see that being useful in a lot of cases, especially for corporate training. If you didn’t do particularly well on something, but you did complete it, then maybe when you take the next course, you can have a little extra remedial work at the beginning. At our medical center, we had a situation where not everyone had to take certain parts of the training. For instance, let’s say we had an annual safety training course. For regulatory reasons, everyone had to take certain parts of it, but beyond that core, there were modules within the course that some people had to take and some people didn’t. And it was based on your job. We had no way of knowing what their job was through the LMS, so we were never able to segregate the modules and only give each person what they individually were required to take.
You could split it up into different courses that —
Well, we could, but when you have hundreds of jobs — it’s very difficult. We thought about building in a little logic at the beginning of the course where you could just say what your job is and it would give you the modules related to what you just entered, but that would be really easy to manipulate; someone could select an easier or shorter path and take that even though it’s not really related to their job.
Another way this could have been solved is if there was a good way to say what job they selected — that that be reported on — then that would be caught.
There would have been ways to do that if the SCORM data were reportable in the first place. But our LMS didn’t really report on that data.
If we could report on something as simple as suspend data, just that one field, it would become extensible to us, and we would be able to use it in ways that suit our needs that might not have been thought of by the people who developed SCORM or the LMS.
Suspend data wasn’t really intended to be meaningful.
Here’s a good example of where that would be useful: even if suspend data contained gibberish that the average person wouldn’t understand, we frequently had people contacting us saying, “I completed the course, but it doesn’t show a completion.” I was storing an array in suspend data that tracked every page that had been seen and interaction that had been completed. I, as a developer, could look at that array and instantly know to tell you where they left off on a course, and whether or not they even came close to finishing it. With our previous LMS, I had direct access to the database, so I could look up my array in suspend data and say, “Look, this person only went halfway. There’s no way that they even came close to finishing it. Tell them to finish it.” In other cases, I would look at it and say, “Well, this person did do everything, but for some reason, one page wasn’t marked as being completed, so just go ahead and give them the completion and don’t worry about it.” So there’s a real use-case for reporting on suspend data.
Of course, you could do a lot of that with the interaction model, but it’s not reportable, either. That was a big pain point with my management in my last job, where they wanted to use these interactions and know which questions people are getting wrong, because if you have 100 people taking the test, and 80 of them get the same question wrong, maybe it’s not them, maybe it’s the way the question is worded. So we can evaluate the effectiveness of our questioning. But if you can’t report on that, then it’s useless. And we never could, so we didn’t use the interaction model. There was no way to report on it, so why bother using it?
The “I want to get at my SCORM run time data” item, I don’t know if you’ve looked at it recently, but there’s a whole discussion about ways to do it. It’s one of the ones we said we were going to do something about, but there’s various options. One of which, going more down a UI mandate road, which seems dicey, is to just mandate what fields need to be reported on in order to get conformance. But what about a tracking system that doesn’t even have any reports, but it’s polled by other systems and so on. That could be a problem there, I think. Another option that is very controversial is having a reporting aspect to the API, similar to what you said about a course being able to read completion of another course, being able to pull all the relevant details of other courses and attempts. Tif you were doing one record at a time, it would be slow. Another thought is a data dump capability.
Those all make sense in some fashion, but to me, they all kind of dance around the real problem, in my opinion, which is that the LMSs simply haven’t reported on SCORM. They never really cared, and it is something they should have been doing. There’s no reason not to do it. But it’s the chicken and egg thing — if they’re not making it available as a report item, then people aren’t going to use it. And if people aren’t using it, they’ll say, “well, why do we need to make it available to report?” So it’s a circular argument.
So to me, I think that if you had it available in the SCORM API, I don’t know where that would be helpful. Because the API, in my view, is usually only accessed by an individual in a course. If you’re a manager, you check your whole department’s completion rates, or like I said about the question, if you had 100 people taking it, you want to see how many people got it right, because you’re trying to evaluate the effectiveness of your question, or course. You don’t want to launch an individual course; you want to go to the LMS, and run a report on a group of people. So I would hope that there is some way to tell LMS vendors “look, if you’re going to have SCORM, you need to have a certain level of reporting support.” Otherwise it’s kind of pointless.
Where I was going with the API or data dump, the way those would try to address it, is the idea would be that if the LMS supported that, and the reporting wasn’t good, and that continued to be the case, then the third party tools would spring up to parse the data dumps appropriately.
I see that as being a viable hack. I guess I’m thinking of it in terms of managers — they want to be able to go to one place. And if they want to merge that data with other reports from the LMS, then that means that not only do they have to export it, with that API tool, but they would somehow have to find a way to merge it with all the other data. And that’s really the role that the LMS reporting tool is supposed to fill. So to me, those are all sort of ways of getting around the fact that vendors really just don’t want to do it.
That’s interesting. I’ve been resistant to push for straight out reporting requirements, because one — I’m still hearing people complaining about basic interoperability, and one thing that’s leading me to think is that any requirements should be very easily and ideally very automatically testable. This would be one that isn’t. But you make some good points.
The irony there, I think, is that telling them that there’s a baseline level of reporting they have to support is much easier than implementing sequencing and navigation. If sequencing and navigation are going to be required — which I don’t know if they will in the future — that implementation is so much harder to do than just putting in a report that lets you access your interaction data, you know? Maybe it can be a best practice, and not a mandate. But I think that there should be at least some kind of mention of it in the documentation. Vendors should do this. Although it’s not required, the best practice is to make this data available as part of your base reports, or something.
That could maybe work hand-in-hand with a data dump, so that if LMSs were bad enough, they could be hacked around, then maybe that could embarrass them into working better, and if enough people starting doing that, it would maybe solve the chicken and egg problem to an extent.
You could also try the carrot and the stick. You could say, for us to certify this product, it needs this level — say, the gold level, which means that all of the API is supported — but if you want to get the platinum level, you have to include reporting. There’s a certain base level they have to meet. So maybe that’s one way to do it, because everyone loves to say that they’re certified.
I’m going through questions and trying to parse them to be more appropriate for you because you’re kind of a special case…
One thing that just came to mind is that you said I’m a special case — you know, I don’t know anyone who isn’t a special case when it comes to this stuff. It seems like everyone has their own special requirements. I think that’s one of the reasons you get such a wide array of comments when you ask the question: what should SCORM look like? You get so many different answers, because everyone has a different way of doing things. I think one of the early assumptions for SCORM, the early versions, for 1.2 and I guess 2004, they kind of assume that people are going to work a certain way. They try to build some flexibility in there, but there are a lot of assumptions being made as to how it’s going to work, what people are going to do with it. I think we’re starting to see, with the way social media is kind of changing the landscape, that people aren’t always going to want SCORM to work that way. And they need more flexibility now, because so many people are special cases.
I guess more focus on flexibility is the upshot of that, what we should be doing.
Again, thats kind of a really broad statement for me to make, but you can’t make it too extensible, or then, what’s the point? You lose all your standardization, and your interoperability, but at the same time, you have to give people flexibility to do the things they are trying to do. And you can’t always know ahead of time what it is they’re trying to do. So that’s why I think suspend data is the queen of the prom. That’s the one everyone wants to hang out with, because they can use it however they want.
Though, probably one of the ideas that was up there would be used — being able to define any variable you want and throw a value in it –
I’m not sure about that. I’ve been trying to keep up with the CMI 5, and I have only heard bits and pieces. From what I’ve heard, they’re trying to make it so there’s a very small base set of elements, and from there, it’s going to be extensible, you can add your own. But, my understanding is that each element maps to a field in the database. How can you do that without changing the structure of your database?
It doesn’t have to; that’s kind of up to the LMS to figure out. You tell them it needs to be persisted, somehow. And if the element is marked printable, then it needs to be included in reports, somehow. According to CMI 5, so far. So you might wind up with having a bunch of columns called custom field 1, 2, and 3, and at some point, if they run off the end of that, I you just wind up having to throw things in a big text field. That becomes a hassle to parse for reporting though.
Yeah, that’s what I was going to get to. If you can’t report on it, it’s not useful. Depending on what it is.
The interesting thing is that it would then be mandated that it be reported on. But it does lead to some hoops for the LMS to jump through.
If you’re letting people define fields, then how are you supposed to be letting them add that to a report? That makes it much more complicated. I’m not saying it’s not possible, but it definitely gets more complicated. And the whole point, I thought, was to try and make things simpler.
I think when we say that things should be simple — definitely as simple as possible for the content developer. If there has to be some complexity on the LMS as a result of that, ideally there wouldn’t be, but if there has to be then that’s a more appropriate place to do that.
That has be balanced with the fact that the more complexity has to be put on the LMS, the more resistance to adoption there’s going to be and that’s bad. So, printables, being able to define any variable there and being able to say it’s printable that’s a concern there, true. As far as more user-defined variables for storing data, not necessarily printing them, is an easier problem. Because then, there’s more room to put them in a text field and parse them somehow. At which point it becomes more like suspend data.
But then you could also run into performance issues, too. Like scaling issues.
Depending on how much data it is, yes. Depending on the total amount of data, these should be solvable problems for the LMS but it is giving them a little bit of a headache there, to implement that properly.
So you think maybe a smaller number of fields that are expected to just show on reports — probably more than just the suspend data now, at least, right? Because there could be a real suspend data, as it were, one that’s really just meant for restoring your state, and here’s another one for just any random thing you want to report back as a text bucket, to kind of differentiate between the two.
I think that would be fine. If you’re getting into awareness of previous attempts, that is kind of related. I just keep thinking, after reading a lot of the comments on the User Voice site, there’s so many people who think they want wholesale changes to SCORM, but I think some of them don’t really understand that what they’re asking for is already there. If you find a way to use it, if you find a way to adapt it to your needs. For instance, the interaction model is actually pretty good. It’s clunky to use, but the fields themselves make sense, and they seem to be suitable for all kinds of purposes. There’s even a generic one that isn’t specific to a type of interaction; you can do whatever you want with it. But people don’t really seem to be using those. I think part of it is because it’s hard to use the numbering. But also, most LMSs I’ve seen don’t report on it, so people wind up skipping it and creating their own course logic to handle that interactions. I think SCORM’s interaction model would work for the vast majority of people, they just need an easier way to get to it, to be able to use it.
So that’s goes to why we have “make it simple” really far up the list. Also, ADL is currently working on a documentation effort to clear some of these things up for people.
That’s something I’m working on with my SCORM wrapper update that I haven’t finished in the last two years. I have a whole little system for dealing with interactions that kind of treats the interaction model almost like a browser DOM where you can get elements by number. GetElementByID, so it can look up an interaction by number or ID. It’s not the fastest, because it has to work with the way SCORM is currently constructed, but I think it would make it easier from an end-user’s point of view if they were okay with a little bit of a performance hit.
We’re looking to try to solve a lot of these things with a new API, and making that a web service API of some sort, so either if it’s SOAP-based, presumably it would be passing XML and we could potentially be passing JSON for a REST-based API. Either way, we get structures that are hopefully easier to work with for things like interactions.
By frameworks, you definitely don’t just mean XML handling frameworks, you mean something to present the same set value, get value, syntax?
This is where maybe the folks pulling for REST are coming from, maybe you don’t need much of a layer for REST, especially if you are just saying “this is complete.” It should be a very simple call.
REST is much easier for content developers, and I’m sure you saw that huge long thread that Avron started [referring to a LETSI email thread]. There are a lot of really good points in there for both sides of the coin. I think REST would be easier for a lot of people, but, there are security concerns. And there are certain benefits to both sides that it’s not an easy choice.
Until that thread really, I was kind of thinking that perhaps LMSs should be required to provide both, because as long as the APIs are pretty similar, it shouldn’t be too hard from an LMS level of what’s hard or not to translate back and forth. But, given that a big part of the concern is with security, presumably LMSs that, where the admin was very concerned about security, would wind up turning off the REST side, which defeats the purpose of requiring them to have both, and then you run into the interoperability problem.
But it’s all kind of a ridiculous argument anyway, because as much as anyone can argue about how secure it needs to be, you can’t do anything high stakes online. It’s just that simple. If someone is in a room, taking a high-stakes exam, they can have a book right in front of them that gives them all the answers. They could have a friend sitting next to them, feeding them all the answers. If it’s not proctored, you have no way of knowing if that person is using any kind of reference material. And so it’s kind of, it’s just kind of — it’s not a solid argument, in my opinion. But laws are passed, anyway.
Especially with a web service API, there’s more problems opened up there where if you have something that doesn’t have appropriate security, and you wind up exposing credentials, then it doesn’t matter if some courses are proctored. I guess if it’s proctored and you keep a record of who is there, etc… but if you get the appropriate credentials, if you can connect to the system from outside the proctored environment, then you go in and maybe change the score of the course. There are some legitimate concerns with security.
Some people brought up the points of the legalities of what’s required. I wasn’t aware of some of that stuff. And then you get into European law versus American law. It’s hard enough to keep track of who is the mayor of a given city, let alone all the laws of that city or state or country.
So that’s particularly tricky.
What do you think will and should change about e- learning in the next 5 to 10 years?
That’s an interesting question — that’s really two questions. The proliferation of rapid development tools: It’s really changed the landscape and has become more and more about non-technical people making courses. And of course that has pros and cons. But it makes it much harder to have a system like SCORM, because the end-user doesn’t know anything about it. They don’t care about it, they just need to know whether or not they can turn it on. And so that means it’s really up to the tool to be the implementers, to be the people who are forward thinking and trying to support these standards.
For instance, the interaction model, to get back to that again: tool vendors don’t use it. So, it’s become a single SCO world, and within that SCO, you’re not even using the full data model that’s available. You’re using the bare necessities. A lot of the talk about simplifying SCORM — people are kind of focusing on that. “Let’s just stick to the basics, because that’s what people care about.” Again, maybe that’s all they care about, because that’s all they ever know about, because they don’t see anything else. I think it would be really great if the tool vendors start building in support for things that already exist, like interaction data. Even comments — from the LMS, from learners — I’ve never seen them used anywhere, but I don’t know why they wouldn’t be used. I mean, I can think of all kinds of use-cases for them. That’s why I keep saying, I think that what exists isn’t bad, it’s just that if people aren’t using it, either because it’s not intuitive enough, or because the vendors who create the tools don’t build in support for it. So I would like to see the vendors building in support for it.
What I think I’m going to see is the vendors not building in support for it. I think in five years, there’s a really good chance that the SCORM support is going to be at about the level it is right now. SCORM 1.2 is not going away, because it’s so much easier to use and implement. Well, SCORM 2004 does all the same things, and the base API is just about the same. It’s not hard, if all you’re doing is suspend data. These tools don’t even let you fetch a learner’s name. I mean, how hard is that? Why not give the ability to say, okay, we just launched the course, give me the option to display this person’s name on a welcome screen. They aren’t even doing that. I just see a lack of interest on the vendor’s part.
Now, if a web service comes out, I think a lot of people would be interested in using it. But even if a web service is developed, I don’t know that it would be that widespread within five years. Maybe, but I wouldn’t bet on it.
The tool vendors waiting on the LMSs and vice versa.
And I think, in the case of the LMS vendors and the tool vendors, I think it’s a case of doing as little as they have to. It costs money to do things. That’s a fair notion. I can’t blame them for not wanting to waste resources on something, but I think that they’re being kind of blind to what’s available to them. I think a lot of the frustration that the community is venting — they’re saying, SCORM sucks, and SCORM this and SCORM that — what they don’t realize is that the frustration isn’t actually with SCORM, they’re working with what the tool vendors provide them and what the LMS vendors can afford. And most of the time, they aren’t giving them the full suite, or they’re giving them lack-luster implementations.
“Satisfied by measure”, is useless, unless you’re taking the situation where the SCO itself is not written by the same person or organization who wrote the manifest, and you actually want to come in and reuse the SCO and change the behavior. Otherwise, you might as well set the values you want in the SCO and be done with it. That sort of reuse hasn’t been common.
I think packaging is going to continue to be used, sometimes, but then you run into situations where people want to have content servers, or you have something like a hardware, a stand-alone hardware simulator that’s not running a package, it just is what it is. And it wants to report back values, or people want to report progress that someone has read a.
For those cases, where the content never touches the LMS, packaging doesn’t make much sense.
If you’re introducing web services, you’re going to have all kinds of situations like that. But as it currently stands, without the web services, packaging could be made simpler. Requiring the XSD files to be there is kind of annoying.
Is there anything else you feel like you haven’t had a chance to say yet?
Nothing that really comes to mind. I think for me, the big thing is reporting. It really — in all the years I have been working with this — it’s usually in a corporate environment where we need to be able to report on the results and we never can. We have been able to report whether it’s completed, and what score they got, and that’s it. It has been incredibly limiting. So it’s kind of hard to enforce that on an LMS, but I really would like to see reporting addressed in some ways, because no matter what you do to the API or run-time or the data model, is it really gonna matter if you can’t report that data?
Do you think, moving forward, it’s going to be like 2004, in terms of everything’s required and there’s nothing optional? Because it seems like there’s quite a bit of — I don’t know if “scope-creep” is the right word, because this is technically a new thing. But it’s quite a wide range. And if all these different proposals are being taken in and addressed, that’s a really big task.
I think I’d rather everything be required, but if it comes out that we’re looking at a monumental task for an LMS to implement, that might have to be rethought. There’s an interoperability hit when it comes to optional things.
So again, from a support point of view, I can just imagine all the different calls coming in to you guys, to your forum, and to my Google group. “I’m trying to do a new SCORM course using a Java wrapper, and it’s not working; what’s going on?” There could be a bug in that wrapper that’s not actually a bug in the new API or web service. So I think this project is really good, it’s exciting, but I can also see a lot of heads hitting desks.
I wonder if there will need to be wrappers for every language, really, or —
If there aren’t, then people have to write their own code to parse what gets returned. And they’ll have to write their own code to send it. And SOAP — I can tell you, as someone who was new to it, who was just trying to use it, it’s not that intuitive. The XML is not so bad, but the rest of it is kind of complicated, and it’s actually much harder to use than I thought it would be.
It depends on the language. Like if you look at Java, in particular, or .Net, then there are great frameworks out there to just completely abstract the SOAP part away, and you’re just dealt with XML, and even the XML is abstracted away, and you’re left with native objects that are populated off that XML.
Yes and no. Yes, that’s generally true; one of the issues I was coming up against was the particular authentication that was being used on LETSI RTWS was not one that was widely supported in all the different web services’ frameworks that I was using. They all used a standard, like a user-name/password kind of deal. That was actually the hardest part, getting the authentication.
So that — so for RTWS that’s a problem. And that’s maybe something to revisit.
So there are frameworks out there, but the ones I was trying to use in that particular case, at least, were not really plug and play because of the authentication differences. Maybe that won’t be an issue if this new web service is structured a little differently or if new framework comes out in the meantime. If you have a good framework, it does make working with that stuff pretty easy. But again, you also get into the whole issue of how are you going to handle your interactions, and if there are rules about numbering them, or ensuring that are created in an incremental order. Then there’s certain validations that need to be done, and a simple web framework is not going to validate against what the new API is requiring in terms of what data is entered.
I really hope the validation can come down to being valid against a schema, pretty much. That’ll, in a way, actually make things much simpler. And then you don’t have to do a set value call —
And you could even use a validation tool that reads the schema and tells you what you’re doing wrong.
Right, so hopefully, to the extent that wrappers, they can hopefully be generic wrappers for the sort of API we’re doing, as opposed to necessarily being SCORM wrappers.
Even then — John Campbell and I recently had a conversation about abstraction. Which led to me writing a blog post recently talking about how you generally want to abstract your course tracking code; unless you’re absolutely certain you’re only ever going to use one form of tracking, like SCORM 1.2, then you want to abstract the functions that are being called from within your course, so that you can use a layer that translates it to whichever tracking mechanism you’re using. SCORM driver has that right? Because it has support for AICC and SCORM 2004 and 1.2, so it means it has some form of abstraction going on. So again, that means within the course — maybe a “wrapper” is not the right term — but there’s still going to have to be some kind of abstraction done, and people might be interested in some kind of framework that takes care of that for them, which is kind of what I was trying to do. That way, in their course, all they ever have to do is call the function that says “set to complete.” They don’t have to know anything else, they don’t have to know about the data model; the framework will take care of it.
It seems like that wouldn’t be something Project TinCan would necessarily provide or ADL would provide, but would be looking to the community to provide, or not, depending on the popularity of the platform, I guess.