Anyone up for instituting a virtual research group?

The Pirolli & Fu paper is interesting and overlaps with quite
a few of the issues we talked about in San Antonio. As I
understand the model, the declarative memory component is
based on a latent semantic analysis network that draws both on
prior work (the Tipster corpus) and a web-specific augmentation.

Broadly, processing seems to operate in three stages:
1) Identify the current display state and the user action that
followed that state,
2) Update the display state (if the user action changes it),
generate a predicted action based on the new state (drawing on
the hybrid spreading activation-production system model), and
identify the user action that follows this state,
3) If the predicted action matches the user action, fire the
predicted action, If the predicted action does not match the
user action, fire the user action.

Although I think this is the kind of work we need to be doing,
I think that the paper overstates the power of the model. One
reason is that, while stage 3 records “misses” (and uses this
data in the subsequent chi square analysis), it treats
decisions on different pages as if they were independent of
one another, something that is clearly not the case. The
decision a reader makes on the first page viewed is almost
certainly related to the decision she makes on the second.
Moreover, setting up the model to always choose the correct
decision keeps the model's behavior on track by reseting
“overall” error in each cycle. I suspect this reduces overall
error quite a bit and inflates the model's goodnbess of fit.

A variation on this same problem seems to arise in the data
analysis. The chi-square presumes that observations are
independent, but the decision making that is explored in this
study is one that occurs across an extended period of time
involving multiple related decisions. It also isn’t clear to
me why the chi square was computed to assess whether the
predicted actions differed from random choice rather than the
choices actually made by readers in the study. I’m not at all
surprised the model is not random but that doesn’t seem to me
to be the point.

Two ideas particularly intrigued me in that they suggested
specific steps that could be taken without requiring the
complexities involved with integrating a latent semantic
analysis module. In the analyses we presented at NRC, we
informally categorized pages according to content and noted
that navigation varied quite a bit even when the content that
seemed to be driving the process was the same across subjects.
We didn’t really know what to make of it, but Pirolli and Fu’s
use of color suggests another way to qualitatively capture
what is going on. With some adjustment to the routines we use
to generate graphics we can color-code the nodes that appear
in path diagrams according to the content they present. This
won’t “explain” what is going on but I think it could give us
insights that we can follow up on. I suspect that color may
help us see content-related patterns and possibly even reveal
interactions between navigation patterns and content. Best of
all, we have all the data we need to generate new graphics and
see if they are useful.

Secondly, I’m intrigued with the notion that task difficulty
might be reflected in the “branchiness of the WBGs (first
paragraph in section 4)”. My work on navigation has
incorporated specific measures of “branchiness” (path stratum
and path compactness) and in some studies these measures seem
to be related to performance. I’d also like to follow up on
an issue that I don’t think is really acknowledged in this
study: how the nature of navigation and decision making
changes within a single extended reading episode. Pirolli & Fu
seem interested in contrasting within-site and across-site
(e.g., Figure 4) but I don’t think this comparison can be
based on the few pages before and after the transition from
one site to another. I think there is a need for a broader
view across the episode of reading within a site if we want to
understand the logic of the decision to leave that site. Data
we presented at another NRC session supports the hypothesis
that as readers navigate within a site, they develop expertise
that changes the time they spend on pages and the type of
navigational patterns they employ.

I would very much like to follow up on Julie’s general
invitation to explore these issues and suggest that we develop
a proposal for a study group to meet at NRC 2005. Perhaps if
we keep up the conversation online we can define a
collaborative approach to studying these issues that draws on
the varied interests and types of expertise we all bring to
the table. If we make progress before the conference, we can
focus on practical issues when we meet in Miami. If we don’t
make progress electronically, we can use our study group
sessions to resolve the issues that remain.

Anyone up for instituting a virtual research group?
Any thoughts on the idea of submitting a study group proposal?

John