Key points:
- potentially, every user action should be trackable
- tracking all actions for a game would result in a massive amount of data
- need to avoid adding too much overhead when tracking interactions
- to be determined: can a useful standard set of interactions for games be found, so that there is meaning across different games?
- little practical difference between games and simulators
- backwards compatability
Can you tell me a bit about what metrics you think are relevant that you’d want to be capturing?
That’s the big question, right? What do we capture. Consider in Starcraft, you can measure every click. Playing Starcraft, to me, almost every click, probably about 85 to 90% of them are decisions. You’re doing something, you’re invoking something, you’re moving something, something is occurring. And if you were to amass all that data, if you could figure out how to filter some of those pieces, that alone gives you a pretty interesting picture of how people are doing things. The program that I’ve worked in prior to this company, it was an FAA program, where they built a task analysis for flying an airplane. The task analysis can get really granular so you can go down to everything, go down to every switch that had to be touched, or all sorts of things like that. But the flight instructor would then sit behind the pilots and grade them against the task analysis and get a really interesting picture of how they’re performing versus what they’re being taught. So that is one possible idea. But the question is, what does any of that data mean? Going back to the Starcraft idea, we could totally capture all of that data, but what does that mean?
Is it enough to enable capturing all that data – does it have to be up to each individual platform to somehow give it meaning?
That’s kind of — that’s where I think I start to get, there’s a bit on an abstract piece to it that I’m not, I haven’t been able to wrap my brain around yet. I mean, I keep thinking about it, when we start talking about games, it’s not like there’s one type of game play. The first person shooter style of game play — so you think about America’s Army. Now America’s Army does track some metrics, but that’s a lot different than being able to say tracking clicks. Just by virtue of the interaction.
Now, clicks is one level, but might it even be relevant to say “where did they hover their mouse and pause before they eventually clicked somewhere else?” Might that tell you something? But now you’re just getting into an even more ridiculous level of data capture.
Yeah, which also leads to another part of the problem, right? How do you capture that amount of data? That’s massive amounts of content, you start thinking about the volume of data and a classroom of like 10 people, things just start blowing up. So that’s certainly a big question, what, how and how to set up. My big obstruction is how is this going to apply to all the different platforms, be it strategy, first person shooter, etc. So those are things, the challenges that I see, for sure. I’ve seen on the Project Tin Can site, the same thing happens with simulators.
I think I even put them together in the same post to say this is for games or simulators who have this issue –
You almost have to. In fact, I was having a discussion the other day, someone asked me what my top ten games list was. Just me personally, just my take on it, and my favorite game all my life has always been Microsoft flight simulator. The question is, is that really even a game? It’s funny because when you think about the gaming model and the simulator model, the two have metrics that work so well together, or are so similar, what’s the difference? Because my response to that, is well, let’s think about this: There’s Operation Flashpoint, is that a game or a simulator? The immediate answer is it’s a game, you can buy it at Best Buy, etc., it’s not a simulator. But when DARPA takes that very same game, and they modify it, just using the tools that come with the game, and make it a convoy simulator, which they call it, is it now no longer a game? It gets very confusing.
And at some level, does it matter?
I guess that’s a good question as well. Does it matter? I don’t know the answer. There are lots of people who will say it absolutely matters. So, I don’t know. But, in the AICC we face that same question when it comes to simulators. We’ve always had the question, well, how do we track simulation data? When we get into a 737 simulator, how do we track all that data so we can do something with it? There are some answers, but right now, that data doesn’t track. For example, switch clicks — I’m sure simulators probably do capture a fair amount of data — but when someone starts typing something into the flight computer system, does it track that? Do we need to track that? If we need to track that, what are we looking for? So there are some big questions.
How much data do we push through a standard API, or to what extent or at what point do you say this is an ID you can use to look up this data elsewhere?
I see. So you basically record something — at that point , would you be collecting data to a database, or would you collect it to a flat file somewhere? And then somehow work on allowing the flat file to fit a standard? It wouldn’t kill your servers.
And would we attempt to make that a standard format? Or would that be based on the game or simulator that was writing it?
I guess the way out there approach would be to try to identify the different interactions in simulators or games? The challenge is that becomes such a massive standard, right? Such a massive piece to record. My concern about some of this is that the volume of data will be so high, and the requirements will be so high, my fear is that it won’t be used. So there’s got to be a way to simplify the content that it captures, to make it so that developers will make it a useful tool.
I also wonder – from the perspective of someone who has games that people would, in theory, use for training — what would the requirements be, or what can we do to make this backwards-compatible? What would I have to do, what would I have to invest to make our game SCORM compatible? Certainly from a business point, that’s something that’s also important. While obviously going forward in some things, it would be easier to implement or accomplish, what about folks who have already gone forward?
B: So they implemented some other proprietary tracking mechanism, or they just don’t track —
They don’t track, and there’s gotta be a way to implement that with their existing games, you know, now their customers are coming to them and saying, “We’re being asked to do this, how do we make it happen?”
It may be though, if you want to only implement part — say on the simplest level, I want to send the pass/fail, that ought to be really easy, and you shouldn’t have to do anything else if you don’t want to.
For the pass/fail stuff, when I think about it, for all the flexibility SCORM currently has, and most of the people we talk to really just want to do pass/fail anyway — so that may be, it all may be a good point and is something maybe worth thinking about, in terms of what it would be to implement this. It’s just another piece to the puzzle.
I think the other question, and this is almost something that has to obviously be sent to a much larger user group, is okay: you have a game, how is that game functioning that you want to track data — what I’m thinking of right now is more of an instructional design question. I’ve always strayed away from free form simulators, and games are at some level free-form simulators. But I’ve stayed away from them simply because, going back to the idea of pilots: if you give a pilot something and let them do whatever they want, they will do whatever they want. Only becomes problematic when you’re trying to train them how to do something. So if you don’t have a guided methodology, you start to get negative learning. They don’t learn the procedures, or they learn new procedure, but it’s not the proper procedure. That doesn’t mean it’s not right, it’s just not the way it’s supposed to be done.
So how do you envision using games; what do you want to track with them? Those are two big questions. I’m not sure what the answer is.
What would be the answers to that for yourself? Or I guess it’s all for your customers.
Well, for our customers, for example, our use of Peacemaker, is a very educational conflict- resolution tool that gives people multiple sides of the same story. So in that case, in our case, using Peacemaker, we’re not really structuring a procedure, there’s no real right way or wrong way to do it. You just kind of go through it. In thinking about my own question — it becomes an instructional design question, perhaps rather than an actual technical question. Because from an instructional design point you may want to guide students down a path, whereas from a gameplay perspective that’s not really relevant. Some games do guide you down a path, you know, rails and whatnot.
