Hi! Today I'm going to talk to you about our paper "Misleading Beyond Visual Tricks:
How People Actually Lie With Charts." If you search the term "deceptive
visualizations," typically the charts that will show up will contain inverted y-axis,
truncated y-axis, and other visual tricks that violate common visualization design guidelines.
These techniques may interfere with our ability to quickly make accurate readings from
the chart so we often call such charts "lying." But why do we always see these exact same charts
when we talk about deceptive visualizations? Well, this can probably be traced back
to Edward Tufte and his notions of "graphical integrity" and "lie factor."
Hopefully everybody has seen such prototypical examples where if a chart, for instance,
shows this line to denote 18 miles per gallon but this line to denote 27 1/2 we call it lying.
Much research has followed up on these ideas and to this day we're studying the effects of the
lie factor on human perception. But, taking a step back: is this how people actually lie with charts?
To answer this question, we collected and analyzed
10,000 COVID-19 data visualizations shared on Twitter and used them to create a typology of
common attributes of misleading visualizations. And today we'll share with you our findings.
This talk is structured as follows: first, we'll discuss the notion of
reasoning errors in visualization posts and talk about how people lie with charts. Secondly, we'll
discuss what role visualization guidelines still play in this framework. And lastly, we'll talk
about how even well-designed charts can support misinformation and what can be done about it.
Let's start with reasoning errors. So, we define reasoning errors in visualization
tweets as unsupported assertions or logical fallacies as basis of an argument, and we find
that 84% of visualization posts that express an opinion about or offer an interpretation of
COVID-19 data contain such reasoning errors. We identified seven common types of errors:
cherry-picking, setting an arbitrary threshold, incorrect causal inference,
issues with data validity ,failure to account for statistical nuance, misrepresentation of
scientific results, and incorrect reading of chart. Let's go through a couple of examples.
So, in this post the author shares a screenshot of Mexico's government dashboard of excess mortality.
There's a very visually salient feature of the chart: a sharp drop in mortality.
The author adds an annotation that assigns a cause-and-effect relationship between the
introduction of the alternative drug Ivermectin and the drop in mortality,
and the tweet text further explains this argument. So,
in this example the proposition that Ivermectin helps with COVID---which the scientific consensus
has disproven---is supported through incorrect causal inference with the help of cherry-picking.
Let's take a look at another example.
So, this is a chart from an actual CDC report showing COVID cases in
a single county over the course of a month.
Here, light blue represents fully vaccinated people and dark blue represents all others.
There's again a very visually salient feature which is the difference in category composition.
And again the tweet text assigns a cause-and-effect
relationship implying that vaccines caused COVID.
But the elephant in the room is: what is the vaccination rate in the general population?
Without knowing, it's not possible to make any conclusions about the efficacy of vaccines.
So even though there are many more cases among the vaccinated, there are likely also many more
vaccinated people at the time. So in this example, the proposition that vaccines further spread COVID
is supported through several attributes, most notably failure to account for statistical nuance.
Now let's move on to the role of visual tricks. We find that 89% of charts with
reasoning errors do not violate any common visualization design guidelines. Importantly,
this percentage is very similar across every cut of data: so, with or without reasoning errors,
posts in support of COVID restrictions or anti-mask posts are all in the range of 87--89%.
Let's discuss a couple of examples that do have visual tricks. So, this chart shows COVID
hospitalizations drop sharply in the UK, with the author concluding that "vaccines work." The
truncated y-axis potentially exaggerates the sharp drop and one might assume that cases went to zero.
So let's look at another example: the chart on the right plots the rate of people vaccinated
versus people died. The careful selection of scales on this dual axis chart exaggerates the
spurious correlation leading the author to state, quote: "deaths rising in line with vaccinations."
So in conclusion, visual tricks may exaggerate the
effects of reasoning errors that are already present in the chart.
So what could be done about it? You might have observed from examples
that the majority of misleading charts are screenshots from reputable sources,
such as government reports, data exploration websites, or news media.
Previously shown examples all contain screenshots of charts that were not
intended to support any of these conclusions. Their vulnerability
to misinterpretation primarily comes from having very visually salient but unexplained features.
But additionally from including warnings against misinterpretation in the limitation
section of the report where they would not "survive the screenshot of a chart,"
in the case of data exploration websites---from offering an unrestricted set of interactions that
could be interpreted by the user as the set of valid comparisons,
or from annotations being added directly onto the chart with new information.
Let's take a look at the structure of these arguments. So, typically we consider a chart that
shows a sharp increase in cases. This can form the base premise of an argument. One can, for example,
use annotations to add another premise---around the time of increase there was an important event,
say, the start of a vaccination campaign. Taking all these premises together,
one might make a general conclusion that the event caused cases. This is an example of an inductive
argument. But inductive reasoning is inherently uncertain and only deals with the extent to which
the conclusion is credible given the premises are logically sound. And in most examples we've seen
they are logically sound, they're simply not plausible enough for the generalization made.
So given this, protecting vulnerable visualizations could be operationalized by
making misinformation arguments less credible and promoting skepticism. Importantly this should be
done through salient features of the chart design, such that it cannot be missed or cropped out.
For example, let's take a look at a COVID chart where an Event X might have caused cases
in Country A. We can make this proposition less credible by, for example, providing case charts
of other regions to not allow for cherry-picked examples, or visualizing important events that are
likely to truly explain a rise in cases---such as appearance of a new variant, or visualizing
the uncertainty and death estimates that may stem from low testing rates or methodological issues.
In conclusion, we find that visual tricks are not the main driver of visual misinformation online,
reasoning errors are. The majority of misleading visualizations in our
data set are screenshots of charts from reputable sources. And if a chart is not
designed with a biased reading in mind it is vulnerable to misinformation arguments.
Thank you!