1
00:00:01,600 --> 00:00:05,840
Think aloud protocol has long been one 
of the most commonly used evaluation

2
00:00:05,840 --> 00:00:11,120
techniques for usability testers, and is 
also used by visualization researchers,

3
00:00:11,120 --> 00:00:18,320
both to evaluate tools and methods as well as 
to investigate the formulation of insights.

4
00:00:18,320 --> 00:00:23,720
But, think aloud user studies are a costly, 
time intensive process, which up until

5
00:00:23,720 --> 00:00:29,960
now are almost always run synchronously, 
either in-lab or via video conferencing.

6
00:00:30,720 --> 00:00:36,520
In fact, in a survey of visualization papers 
we conducted, only 1 of 67 think-aloud

7
00:00:36,520 --> 00:00:43,800
studies were conducted asynchronously, 
meaning without a researcher present.

8
00:00:43,800 --> 00:00:49,720
In this paper, we make two primary contributions. 
The first is a technical contribution called

9
00:00:49,720 --> 00:00:57,720
CrowdAloud, a tool for conducting and analyzing 
asynchronous think aloud studies. The second is an

10
00:00:57,720 --> 00:01:04,320
investigation of the feasibility of crowdsourced 
think aloud studies. We conduct two separate user

11
00:01:04,320 --> 00:01:10,920
studies, one which compares crowdsourced think 
aloud to in-lab thinkaloud, and another comparing

12
00:01:10,920 --> 00:01:17,000
think aloud responses to more traditional 
text responses. In those studies, we found

13
00:01:17,000 --> 00:01:22,760
that think-aloud is comparable to in-lab studies, 
and produces more insights than text responses,

14
00:01:22,760 --> 00:01:28,400
with a similar quality. This video, however, 
will focus on the first contribution, Crowdaloud.

15
00:01:28,400 --> 00:01:36,760
CrowdAloud is built on top of reVISit, an 
existing tool for conducting user studies,

16
00:01:36,760 --> 00:01:43,960
but one without any audio capabilities. 
reVISit utilizes a domain specific language,

17
00:01:43,960 --> 00:01:49,840
or DSL, for defining tasks and study structure.

18
00:01:49,840 --> 00:01:54,480
CrowdAloud can be enabled in 
that DSL with one line of code,

19
00:01:54,480 --> 00:01:59,840
and which tasks should be recorded in 
the study can also be set in the DSL.

20
00:02:01,320 --> 00:02:04,120
Optionally, you can add a mic-check task,

21
00:02:04,120 --> 00:02:10,840
which will stop participants until they have 
demonstrated they have a functioning microphone.

22
00:02:10,840 --> 00:02:12,920
After a study is complete, CrowdAloud can

23
00:02:12,920 --> 00:02:18,080
be used to conduct a qualitative 
analysis on the think aloud data.

24
00:02:18,080 --> 00:02:24,000
CrowdAloud has 2 primary views, the replay 
view, and the tagging & transcription view,

25
00:02:24,000 --> 00:02:29,840
which are intended to be used together. First, 
let's talk about the tagging & transcription view.

26
00:02:31,240 --> 00:02:34,280
Recorded audio from the study 
is automatically transcribed,

27
00:02:34,280 --> 00:02:40,240
and researchers can edit 
the transcription at will.

28
00:02:40,240 --> 00:02:46,920
Researchers can also create codes, 
each with their own name and color.

29
00:02:46,920 --> 00:02:52,400
After creating a codebook, each audio 
snippet can be assigned codes. Codes are

30
00:02:52,400 --> 00:02:59,840
specific to the logged in user, allowing 
multiple users to independently code.

31
00:03:01,240 --> 00:03:05,240
The true strength of CrowdAloud comes when 
researchers utilize provenance data in their

32
00:03:05,240 --> 00:03:10,840
study. Using provenance, CrowdAloud can 
recreate the state of the original study,

33
00:03:10,840 --> 00:03:16,480
replaying the study similar to a video. The 
provenance data is synced with the audio data,

34
00:03:16,480 --> 00:03:20,920
so that researchers can always see the same 
stimulus that participants saw while listening

35
00:03:20,920 --> 00:03:29,760
to their comments and insights. Here is a real 
example from one of our crowdsourced participants.

36
00:04:02,120 --> 00:04:06,400
The replay view is also synced with 
the tagging view, even if they are

37
00:04:06,400 --> 00:04:12,120
in separate tabs, allowing researchers 
to use multiple monitors for analysis.

38
00:04:12,120 --> 00:04:17,400
To aid with analysis, we also have an overview 
table view. The table view includes basic

39
00:04:17,400 --> 00:04:24,120
information for each participant in a task, like 
time spent, interaction count and word count. You

40
00:04:24,120 --> 00:04:32,560
can also view tags which were assigned for this 
specific task. Additionally, researchers can

41
00:04:32,560 --> 00:04:38,040
search for specific tags, in this case to find all 
participants who made a UI/UX comment, question,

42
00:04:38,040 --> 00:04:48,760
or suggestion. From here, we can jump directly 
to their task analysis, to see what they said.

43
00:04:48,760 --> 00:04:52,640
We hope that our findings, as well 
as our tool CrowdAloud, encourages

44
00:04:52,640 --> 00:04:58,960
the visualization community to conduct more 
crowdsourced think-aloud studies in the future