1 00:00:01,020 --> 00:00:06,450 We introduce revisit, looking under the hood of interactive visualization studies. 2 00:00:07,680 --> 00:00:13,100 It is no surprise that well designed interactions can improve the way that people use data visualizations. 3 00:00:13,100 --> 00:00:19,820 But, how do we assess the impact that interactions have on performing visual analysis? 4 00:00:19,820 --> 00:00:24,530 For visualizations with few interactions,  data analysts can design the study to control 5 00:00:24,530 --> 00:00:26,400 for the effect of each one. 6 00:00:26,400 --> 00:00:31,570 This can be useful in understanding how each interaction affects a user's accuracy or speed 7 00:00:31,570 --> 00:00:33,300 when completing a task. 8 00:00:33,900 --> 00:00:39,190 However, for more complex interactive visualizations, it becomes harder to control for all possible 9 00:00:39,190 --> 00:00:41,180 forms of interactions. 10 00:00:41,640 --> 00:00:46,340 Additionally, participants can use different analysis strategies, or sequences of interactions, 11 00:00:46,340 --> 00:00:48,760 to solve a given task. 12 00:00:48,760 --> 00:00:53,649 This leads to the question,  how do analysis strategies impact participant performance 13 00:00:53,649 --> 00:00:57,300 when using interactive visualizations? 14 00:00:57,300 --> 00:01:02,410 To address this, we propose a new study workflow that allows researchers to better understand 15 00:01:02,410 --> 00:01:06,750 the effect interaction strategies have on user performance. 16 00:01:06,750 --> 00:01:09,640 We implement the data analysis methods in this workflow 17 00:01:09,680 --> 00:01:12,320 in an open sourced web-based tool, reVISit 18 00:01:12,880 --> 00:01:17,740 reVISit provides support for analysis of both qualitative and quantitative results, as well 19 00:01:17,740 --> 00:01:20,760 as provenance visualization through event sequence mining 20 00:01:20,760 --> 00:01:23,430 and playback of participants sessions. 21 00:01:24,440 --> 00:01:28,100 Let's take a closer look at reVISit in action. 22 00:01:28,100 --> 00:01:30,860 Heather is a researcher who recently ran a user study 23 00:01:30,860 --> 00:01:33,240 analyzing the performance of network visualizations. 24 00:01:33,840 --> 00:01:39,840 Her study compares node link diagrams to adjacency matrices  for finding paths in a network. 25 00:01:39,849 --> 00:01:44,159 During her study design, Heather had to balance the interactions available between both 26 00:01:44,159 --> 00:01:48,109 node link and matrices to not unduly favor one condition over the other. 27 00:01:48,109 --> 00:01:53,130 An important part of finding a path in a network is being able to identify neighboring nodes. 28 00:01:53,130 --> 00:01:57,710 In the node link,  this can be easily be done by clicking on a node to highlight neighbors. 29 00:01:57,710 --> 00:02:02,549 In the adjacency matrix, Heather added an interaction that highlighted neighbors in 30 00:02:02,549 --> 00:02:06,789 green when a user clicks the column label. 31 00:02:06,789 --> 00:02:12,250 Now, she wants to be able to assess whether or not this added interaction actually helped 32 00:02:12,250 --> 00:02:14,660 improve performance for the adjacency matrix. 33 00:02:14,660 --> 00:02:18,530 Heather moves over to the revisit system, in the home page. 34 00:02:18,530 --> 00:02:23,420 Here she can see a high level overview of how her participants performed on each of 35 00:02:23,420 --> 00:02:27,780 her tasks, as well as the mean interaction patterns used in those tasks. 36 00:02:27,780 --> 00:02:29,580 Hovering over the interaction patterns 37 00:02:29,580 --> 00:02:33,450 she notices that users who use the column label 38 00:02:33,450 --> 00:02:38,000 interaction performed at around 70% accuracy 39 00:02:38,000 --> 00:02:41,420 compared to the 60% accuracy without 40 00:02:42,500 --> 00:02:45,400 She wants to drill down into this a little bit more. 41 00:02:45,420 --> 00:02:51,800 so she goes over to the task analysis pane,  and then first groups on condition. 42 00:02:51,810 --> 00:03:00,070 Then she wants to group upon whether or not a user used the column label itself. 43 00:03:00,070 --> 00:03:01,780 Grouping on that and expanding the columns, 44 00:03:01,780 --> 00:03:08,380 she can see that users of the column label interaction 45 00:03:08,480 --> 00:03:11,820 have a 67% accuracy 46 00:03:11,860 --> 00:03:15,120 compared to a 45% accuracy for the participants 47 00:03:15,120 --> 00:03:18,490 how did not use the column label technique. 48 00:03:18,490 --> 00:03:24,110 Having validated that there exists a pretty large difference between the accuracies of 49 00:03:24,110 --> 00:03:28,790 individuals who used the column label and don't, she wanted to drill down into exactly 50 00:03:28,790 --> 00:03:32,040 how those individuals who don't use the column label 51 00:03:32,040 --> 00:03:34,580 go about solving this task for the adjacency matrix. 52 00:03:35,480 --> 00:03:38,460 As she is interested in the answers that they gave as well 53 00:03:38,460 --> 00:03:40,900 she goes unhides that column 54 00:03:40,900 --> 00:03:45,900 and then scrolls down to look at how the individuals who didn't use the column label 55 00:03:45,900 --> 00:03:48,460 approach this task. 56 00:03:48,470 --> 00:03:54,760 as she scrolls down, she sees  a fair amount of these answers, of MViews. 57 00:03:54,760 --> 00:04:00,900 As this is an incorrect answer  to the problem, she delves into the playback interaction to 58 00:04:00,900 --> 00:04:04,380 see how users are using this to solve the problem. 59 00:04:05,700 --> 00:04:12,480 Playing through each of the strategies she sees that the users search and then select 60 00:04:12,490 --> 00:04:19,370 for MViews, which is located right in between Lane and Rob. 61 00:04:19,370 --> 00:04:27,020 Interested by this, she sees that likely what is occurring is users are looking directly 62 00:04:27,020 --> 00:04:33,259 in between Lane and Rob in selecting the node that exists between them that is an institution, 63 00:04:33,259 --> 00:04:35,490 which corresponds to MViews. 64 00:04:35,490 --> 00:04:38,280 This represents a conceptual misunderstanding 65 00:04:38,280 --> 00:04:43,540 of how edges are visualized inside of an adjacency matrix 66 00:04:44,940 --> 00:04:49,580 This methodology was employed to analyze data from two published crowdsourced user studies, 67 00:04:49,590 --> 00:04:52,780 which we present as case studies in the paper. 68 00:04:52,780 --> 00:04:57,800 The combined analysis of study results with the interaction patterns helped our analysts 69 00:04:57,800 --> 00:05:03,080 uncover novel interaction strategies, identify gaps in participant training 70 00:05:03,080 --> 00:05:07,280 and measure the impact of design decisions used in the study. 71 00:05:07,280 --> 00:05:09,559 You can check out reVISit at this URL. 72 00:05:09,559 --> 00:05:10,850 Thanks for watching our video.