Key Points:
- should be able to use SCORM for surveys
- next version should add more value, to increase adoption
- sequencing is too complicated, results can be unexpected
- publishing the same course to different LMSs can yield different results, time tracking varies, for example.
- need ability to host content on internet, allow paying customers to access it
- there should be a standard for LMSs to exchange tracking data
TinCan: You’re not the only one to find cross-domain issues aggravating, it’s a must-solve.
Bernd: Another kind of frustration that we’ve had is just the lack of data — like you can create an exam and a survey in SCORM, but the reporting on that data is so impractical it’s almost not worth using. If you want to — especially surveys, if you want to record, there’s no way in SCORM, as far as we know, to say, “This is the question text, and these are all the options the user was given, and this is what they picked.”
TinCan: Yeah, all the options makes it — really the best thing, the closest you can get with objectives, you’ve got an identifier and, you’ve got a description, so you can kind of get the question text in that way. But it’s a little less than ideal, for a lot of reasons. One being that you wouldn’t be able to really get, you don’t have enough to fit in there and these are all the possible answers with interactions you can say this is right answer and this is the answer they picked, but you don’t really have a good way to say these are the available answers. And as far as the question text itself, the description isn’t always long enough to get that across and also it would probably be something that would be better off for reporting purposes in the manifest, so — because it doesn’t change every time you run the course, but if you put in the objective description then you have to, at run time, set it at every time you do the quiz or survey.
Bernd: We’d love to be able to use SCORM to do our exams and surveys, kind of get ourselves out of that business, that we have our own custom tools for exams and , but then we deliver content obviously our custom tools don’t work, so it depends on the deal. Some customers, they just take it and run it through SCORM. Others, we just give it to them in a text file, the questions and the answers, then they have their own exam and survey tools they import those into, so. That would sort of be nice.
I guess another thing we would like to see from this is easier — we’d like to see a bigger jump in what this can do; between SCORM 1.2 and SCORM 2004. SCORM 2004, from our experience, it hasn’t really hit it off yet, because there are definitely some differences and advantages to 2004, but it’s not enough that the industry has just totally embraced it and forgotten about 1.2.
TinCan: There are certainly some things that could be big advances but perhaps the problem has been that they just aren’t the changes the industry wanted. Sequencing, it’s actually really big, it does a lot, not everyone got enthusiastic about it.
Bernd: In our courses we don’t really deal with that. We generate our courses in SCORM 2004, but we found that if you get too complicated with the logic, and how that course can be completed, learners tend to get confused, so they have problems.
TinCan: So the course executes exactly as you want, you don’t have trouble with the LMS interpreting the sequencing the wrong way, but you find it confuses the learners?
Bernd: I think I’ve had situations where learners get to points and then maybe it’s been because the courses were developed in a way that doesn’t work with people not going through it exactly the way that they thought they would. So the flow can be so complex they can’t complete the course, because they don’t know the way they’re supposed to go. It’s not an issue necessarily with SCORM itself, but just the fact that it’s easy to — it’s not easy to do that.
Bernd: That’s kind of where, and from my perspective, that’s kind of where the — where I’ve heard some issues. The sequencing gets complex, and it’s either not done correctly from the course development end or the learners don’t go through the processes exactly the way they are supposed to and then they can’t complete, because they —
TinCan: I hadn’t heard of learners directly being confused by sequencing before. It sounds like a different way of putting that the sequencing needs to be fixed.
Bernd: And maybe that’s more just the understanding of it from the developers’ end, is problem more than to just do the sequencing. Yeah, I think that’s one of those that’s SCORM 2004 has that 1.2 doesn’t, but from our experience, it hasn’t been one that’s been taken advantage of much. Like you said, it may not be one — it’s a big thing, but not necessarily what the industry wanted to jump on.
Ben: I don’t know if there’s a lot I can add other than to say, the main pain point that we have with SCORM is that our content developers can publish a course that is ostensibly SCORM-compliant, and we get a different result, in terms of how the content behaves, depending on which LMS we post to. And that is absolutely maddening. To me it speaks to a flaw in standards, if as an industry we can’t agree to certain principles we can publish to and just have faith that they will work. And I realize there’s a lot of complexity there, a lot of LMSs out there, people implement APIs in different ways and so on. But it seems from my perspective that it is far too complex to, for instance, continue to run into these issues and to build whole departments around the fact that we can’t count on predictable behavior for a course after it’s published. That’s really been a pain point for us.
Bernd: Yeah, and that can be things as simple as the big “C,” little “c” when you’re doing completed one LMS and they want it to be a capital Completed and then wants a lower- case complete.
TinCan: That’s interesting, because that means one LMS is right and the other’s wrong. It’s not, I guess that might be a test-suite issue, and some of those items, I think as you move up the editions of SCORM 2004 in particular, the LMS test suite has just continued to grow. But then there’s an issue of LMS vendors saying they are SCORM 2004 compliant, when they don’t specify what edition they are compliant to and may not have passed the test suite.
Bernd: Yeah, that’s kind of where, at least for me, I don’t necessarily know what the standard defines and what it doesn’t, but we’ve seen that in — and all the tools we use too, allow to customize how we send completion. You can actually go in there and type whatever you want as the completed word that goes across during the state, that goes across, and to me it seems like, and I think Ben agrees too, that they shouldn’t have that flexibility, it should just be, it’s completed and that’s that. If it’s not completed, and this is the state.
TinCan: Yeah, that’s just not right. So in a way, for that particular issue — what that speaks to is maybe more clarity somehow in the testing regime and part of it I guess is the community just needs to actually investigate which LMSs have been certified. It’s on the ADL’s website what people have actually passed conformance tests.
Bernd: In our case we deliver content to people and they have their own LMS and we can’t necessarily control what LMS they have. They, it becomes a matter of trial and error. Like here’s a course, does it work?
TinCan: Maybe from that point of view, a mini test-suite, some test courses designed to send to LMS vendors to evaluate, to let them see for themselves whether they are conformant or not. Or prove the point, might help there.
Ben: From my perspective, it seems as though if the standard supported or made it easier to build a basic course and have assurance that because you built to the standard, it would work across the board in LMS; that to me would be the ideal. The problem feels to me — I don’t know if you have experience in the health care arena – a like HL7, in the ʻ90s, and even in the 2000’s it’s been used, but HL7 as a way to send EDI files.
Anyway, HL7 basically, it was infinitely customizable and because of that it introduced a tremendous amount of complexity that required business partners to all communicate and synchronize and be in tune with one another, to the point of defining delimiters and looping sequences, etc., for these kinds of health care files, claims processing, things like that. And almost overnight, XML came into play and everybody I know of got off of HL7 as fast as they could because it introduced so much expense and so much complexity that really nobody wanted to use, but you had to. Because the standard was so flexible, it was really no standard at all. And with XML you had something that was self-documenting, it was easy to read, developers understand it and it made communication with insurance companies and state boards of health, etc., so much easier.
I’m not from the e-learning world, but the problem that I see with SCORM is it feels a lot like that HL7 problem, that if you are a professional SCORM developer, you have no hope of actually understanding this and knowing that if you build a course, that it will really behave the way you want it to. As a result — and we’re fortunate you folks at Rustici are out there and we partner with you — but, I feel like as a developer, and Bernd as our lead developer at the company, he should just be able to crack open the standard and figure it out and we just can’t do that. We have to outsource this. And our needs are pretty simple: we just want to put a course out there and let it bookmark, and have it marked completed at the end and that’s about it. We don’t do anything with sequencing, we don’t do much with interactions, we don’t do time-tracking. And yet, still have this tremendous pain point, very very expensive for us, and from my perspective, that’s again where I’m seeing the pain and what feels, from an outsider perspective, like a tremendous flaw in SCORM.
….
TinCan: Regarding your cross-domain problems, have you looked at LETSI RTWS?
Bernd: That’s one thing that would be nice too, and maybe the web services that are there can already do a lot of what we want, that there’s no way, no standard way for two LMSs to talk to each other if we want them to do that. Where if we had a customer that had their own LMS but wanted to record some information back to our LMS, there’s no standard way to do that.
TinCan: So with RTWS, you would, the trick will be you have to get them a session ID to work with, so you’ll have to work out some non-standard way of doing that, because the browser launch, which kind of defeats the purpose, RTWS doesn’t address that. That’s the sort of thing that will have to be addressed for Tin Can, among other things.
Other things people have brought up include talk about wanting, in addition to an API for storing data, another for pulling that data back. So people could do reporting right within that course, and depending on the permissions that user running the course has, that they could potentially go and see how other people did in that course. If that’s appropriate, they can get a report out of it. Then if there’s a standard reporting API, be able to tie in some sort of third party reporting tool to an LMS. People have talked about better support for mobile learning, better support or any support for team-based learning, where you have to track that several people in a team may be taking the same content or have to interact with each other. Do any of those spur more thoughts?
Ben: I think an API that would let us pull back reporting data would be fantastic.
Bernd: Yeah, I think they all sound like good ideas. Not things that I really thought of either, like I can’t say they necessarily spawn others either, but I think there’s some potential that this could go and I’m just trying to think of how we would make all these highly interactive courses, where users are taking the courses at the same time and interacting with each other; in our situation that’s probably not too useful, but. It could be interesting.
