We introduce revisit, looking under the hood of interactive visualization studies.
It is no surprise that well designed interactions can improve the way that people use data visualizations.
But, how do we assess the impact that interactions have on performing visual analysis?
For visualizations with few interactions, data analysts can design the study to control
for the effect of each one.
This can be useful in understanding how each interaction affects a user's accuracy or speed
when completing a task.
However, for more complex interactive visualizations, it becomes harder to control for all possible
forms of interactions.
Additionally, participants can use different analysis strategies, or sequences of interactions,
to solve a given task.
This leads to the question, how do analysis strategies impact participant performance
when using interactive visualizations?
To address this, we propose a new study workflow that allows researchers to better understand
the effect interaction strategies have on user performance.
We implement the data analysis methods in this workflow
in an open sourced web-based tool, reVISit
reVISit provides support for analysis of both qualitative and quantitative results, as well
as provenance visualization through event sequence mining
and playback of participants sessions.
Let's take a closer look at reVISit in action.
Heather is a researcher who recently ran a user study
analyzing the performance of network visualizations.
Her study compares node link diagrams to adjacency matrices for finding paths in a network.
During her study design, Heather had to balance the interactions available between both
node link and matrices to not unduly favor one condition over the other.
An important part of finding a path in a network is being able to identify neighboring nodes.
In the node link, this can be easily be done by clicking on a node to highlight neighbors.
In the adjacency matrix, Heather added an interaction that highlighted neighbors in
green when a user clicks the column label.
Now, she wants to be able to assess whether or not this added interaction actually helped
improve performance for the adjacency matrix.
Heather moves over to the revisit system, in the home page.
Here she can see a high level overview of how her participants performed on each of
her tasks, as well as the mean interaction patterns used in those tasks.
Hovering over the interaction patterns
she notices that users who use the column label
interaction performed at around 70% accuracy
compared to the 60% accuracy without
She wants to drill down into this a little bit more.
so she goes over to the task analysis pane, and then first groups on condition.
Then she wants to group upon whether or not a user used the column label itself.
Grouping on that and expanding the columns,
she can see that users of the column label interaction
have a 67% accuracy
compared to a 45% accuracy for the participants
how did not use the column label technique.
Having validated that there exists a pretty large difference between the accuracies of
individuals who used the column label and don't, she wanted to drill down into exactly
how those individuals who don't use the column label
go about solving this task for the adjacency matrix.
As she is interested in the answers that they gave as well
she goes unhides that column
and then scrolls down to look at how the individuals who didn't use the column label
approach this task.
as she scrolls down, she sees a fair amount of these answers, of MViews.
As this is an incorrect answer to the problem, she delves into the playback interaction to
see how users are using this to solve the problem.
Playing through each of the strategies she sees that the users search and then select
for MViews, which is located right in between Lane and Rob.
Interested by this, she sees that likely what is occurring is users are looking directly
in between Lane and Rob in selecting the node that exists between them that is an institution,
which corresponds to MViews.
This represents a conceptual misunderstanding
of how edges are visualized inside of an adjacency matrix
This methodology was employed to analyze data from two published crowdsourced user studies,
which we present as case studies in the paper.
The combined analysis of study results with the interaction patterns helped our analysts
uncover novel interaction strategies, identify gaps in participant training
and measure the impact of design decisions used in the study.
You can check out reVISit at this URL.
Thanks for watching our video.