I don’t know how many times I’ve said to someone on the phone, “SCORM is difficult, especially for the LMS provider.” There are many moving parts, countless interpretations, and vagaries in the specification itself. For the most part, we handle these things exceptionally well. Sometimes we make mistakes, and sometimes those mistakes can compound themselves.
The Source of Today’s Problem
In SCORM 1.2, mastery_score and lesson_status can interact with each other strangely. Frankly, the specification can be interpreted in two ways.
From Section 3.4.4, “The SCORM Run-Time Environment Data Model”, in the cmi.core.lesson_status section (henceforth called “The Narrow View”):
After setting the cmi.core.lesson_status to “completed”, the LMS should now check to see if a Master Score has been specified in the cmi.student_data_mastery_score, if supported, or the manifest that the SCO is a member of. If a Mastery Score is provided and the SCO did set the cmi.core.score.raw, the LMS shall compare the cmi.core.score.raw to the Mastery Score and set the cmi.core.lesson_status to either “passed” or “failed”. If no Mastery Score is provided, the LMS will leave the cmi.core.lesson_status as “completed”.
From Section 3.4.4, “The SCORM Run-Time Environment Data Model”, in the cmi.core.lesson_status section, incorporating text before and after “The Narrow View” (henceforth called “The Holistic View”):
Additional Behavior Requirements: If a SCO sets the cmi.core.lesson_status then there is no problem. However, the SCORM does not force the SCO to set the cmi.core.lesson_status. There is some additional requirements that must be adhered to successfully handle these cases:
- Upon initial launch the LMS should set the cmi.core.lesson_status to “not attempted”.
- Upon receiving the LMSFinish() call or the user navigates away, the LMS should set the cmi.core.lesson_status for the SCO to “completed”.
- From above After setting the cmi.core.lesson_status to “completed”, the LMS should now check to see if a Master Score has been specified in the cmi.student_data_mastery_score, if supported, or the manifest that the SCO is a member of. If a Mastery Score is provided and the SCO did set the cmi.core.score.raw, the LMS shall compare the cmi.core.score.raw to the Mastery Score and set the cmi.core.lesson_status to either “passed” or “failed”. If no Mastery Score is provided, the LMS will leave the cmi.core.lesson_status as “completed”.
Herein lies the big difference. The bullets are intended only for the cases in which the LMS has been forced to manage the status on its own. In a piece of content that sets its status (as we’ll discuss below), we believe the LMS is not supposed to intervene with regard to the Mastery Score.
What we did wrong, a while ago
In SCORM Engine 2007.1, we went with this logic, which maps to the “Narrow View”:
If cmi.core.lesson_status has been set and cmi.core.score.raw has been set, compare the Mastery Score to the cmi.core.score.raw and set the status to “passed” or “failed”.
Ultimately, as this logic rolls up through the course, this tolerates content we believe is wrong and reads as “completion_status=complete and success_status=passed” or “completion_status=complete and success_status=failed” to the client LMS. Put another way, it cleans up the mistaken interpretations made by the content author. (It’s an understandable mistake.)
This seems OK at first blush, but then you start running into content that expects the other behavior. If you’re a content author, one that reads the spec holistically, and you’ve intentionally set a value for lesson_status, and the LMS overrides it, that’s pretty confusing. If the spec were totally clear on the subject, we would stand behind it. Given that the spec is ambiguous here, we can appreciate the author’s point of view.
So, we did what we do. We made accommodations.
How we accommodate different interpretations of the specification
We have long believed that the best way to have a highly compatible SCORM player is to accommodate different interpretations from content. This is a perfect example of why we do this, and it allows us to properly support content in a way that other LMSs and players just don’t.
From our release notes for 2008.1:
Mastery Score Overrides Lesson Status – In SCORM 1.2, there is a debate about when and if the LMS should override the lesson status reported by the SCO with a status determined by the reported score’s relation to the mastery score (i.e. if the reported score is 60 and the mastery score is 80, then should the LMS set the status to failed even though the SCO said the status should be passed?). This setting allows you to choose whether or not the LMS should override the status based on the score for this course.
Alright, this is great, right? Now we can have our cake and eat it too. (The fact that cake is gross will have to be another post.)
Every time we add a new package property like this one, we have to make a decision on the part of our clients. We have to decide what the default is. In some cases, this is easy stuff. When we’re tolerating departures from the standard, we simply go with the standard as the default. This is a tough one, though, because the spec is a bit ambiguous. In this situation, we go with what we believe is the correct interpretation of the standard.
In this case, we decided to opt for “false”, or, mastery score does not override status. We think that a content developer who’s smart enough to set his or her own status is also smart enough to retrieve the mastery score and compare against if they want to. We’re erring on the holistic side of things here, and I still feel good about this decision.
I do not, however, feel good about our mistake.
The Mistake
We chose the default. We deployed the new version of the SCORM Engine. And we added the necessary columns as part of the upgrade script. In doing so, we used the default value.
Big Mistake. Big. Huge.
–Vivian, Pretty Woman
(Note, this is not a wide spread problem. It’s isolated to content with an atypical interpretation, but it is very problematic for those courses. I just like to quote movies.)
Some of our clients have content that expected the LMS to make the comparison against the Mastery Score even though they’d already set the status themselves. This content had functioned without issue for some time. And in upgrading to 2008.1, they introduced a problem with older content.
With the new default, though, this is what happens. A course could set cmi.core.lesson_status to “completed” and then report a cmi.core.score.raw that exceeds the Mastery Score they’ve provided. The content could assume that the LMS logic defined in Section 3.4.4 (the narrow view) would then change the lesson_status to “passed”. Because we’ve opted to go with the holistic approach by default, the status would in fact not be changed. This scenario, though, isn’t a big deal. The client LMS would still interpret this course as sufficiently completed and all would be well.
The mistake manifests itself, though, when the cmi.core.score.raw is less than the Mastery Score. In this situation, the status values would remain “completion_status=complete and success_status=unknown”. To the client LMS, this appears to be a course that is probably complete and has no testing, when in fact, it’s really a failed test.
The conclusion? We have picked the right defaults for people going forward, but we probably should have set the defaults in the upgrade script to stick to the old behavior. (We have, in fact, gone back to the 2008.1 upgrade script and made this change for those of you who have yet to upgrade.)
Now What?
Well, we just, last night, discovered this side effect behavior, and it obviously merits immediate action for some clients.
For those of you who ran against 2007.1 and have some concern that you may have courses that function like this, you can opt to revert to the old logic. If you’d like help doing just that, you can simply ask us for the queries to revert to that default. We’ll help you through that and we’ll help you examine any potential “false completions” that have happened since you deployed 2008.1.
If you’re building a new SCORM Engine integration, you can opt to go with our defaults. It is our experience that more content (including some from a big authoring tool vendor) benefits from our new default behavior. But that doesn’t mean it catches every scenario. This is something that you and we will continue to be on the lookout for. In fact, we’re going to see if there’s any sort of a heuristic that we could deploy successfully to handle this ourselves. (We’re not optimistic, but we’d like to catch this one without human intervention.)
