Hello. I'm Devin Lange.
I'm a
PhD student at the University of Utah and a member of the Visualization Design lab.
I will be presenting work done by myself Shaurya
Sahai, Jeff Phillips and Alexander Lex.
In this presentation, I will discuss academic misconduct,
specifically the fabrication and falsification of tabular datasets.
In addition, I will discuss our ideas and contributions
towards battling this problem.
We believe that most scientists work
hard to expand knowledge and improve society.
Unfortunately, some cheat which can lead to devastating consequences.
One notable example of this is key Alzheimer's research
that is now in question.
This research led to 15 years of misguided focus in the scientific community
as well as the development of pharmaceutical drugs.
All of this work now has a massive question mark above it
because the foundational research contains data that appears to be manipulated.
And this is just one example of many troubling cases in
just the last few years.
So what can we do?
First, we can look at what is already being done for similar problems.
Plagiarism in writing is something that editors of journals check for with purpose
built software such as iThenticate tools also exists for detecting
duplicated regions in image data, often a sign that the data has been manipulated.
But what about tabular datasets?
Let's examine another story.
In this particular story,
we look at a junior research faculty member, Dr.
Kate Laskowski.
Early in her career, she started a new collaboration
with a prominent member of her field who provided the data
for a paper they published together about the behavior of spiders.
Later, after a closer investigation, she determined the tabular datasets
he provided could not be trusted
in this incredibly difficult situation.
She's decided to retract the paper and publish a blog chronicling the story
about how she came to her conclusions
for this data,
I would like to propose
three high level steps we can take in the scientific community
to help prevent the manipulation of tabular datasets.
First, it's important to understand
properties of manipulated data.
Next, I think there's room to create more tools
to help identify manipulated data.
And finally, we we must consider how best to implement best practices,
both individually and as a community, to reduce data manipulation.
To begin
understanding fraud can be challenging due to the adversarial nature of this topic.
The way that we approach this for our work was by examining
tabular datasets associated with retracted papers or papers
that had an expression of concern issued.
We also reviewed the public arguments
for why these papers should be retracted.
Ultimately, we found ten datasets
from the field of biology, medicine, psychology and marketing.
We have labeled each with a unique key.
After investigating all these datasets, we identified patterns
that exist across them.
One pattern that we identified is unexpected formatting,
which I will discuss in detail later.
We also recorded the four datasets
that exhibit this pattern.
We did this for seven other patterns and organize them into four higher level
themes format in numerical, structural and domain.
We refer to these patterns as artifacts of manipulation.
You may
notice that we have included unexpected leading digits as an artifact,
even though we did not identify it in any of our datasets.
This is because Benford's law, which describes
the expected frequency of leading digits, is a well-established technique
for identifying fraud, especially in financial data.
However, we found that the criteria required to run
this test were frequently not met.
Our goal was to organize these artifacts so that they are general enough
to cover a wide range of scenarios while still being useful.
And we believe we have successfully struck that balance.
However, we acknowledge that there are artifacts that we may not have captured.
So we have created a living document online that lists what we have found
and invites others to suggest changes or additions to this document.
Moving on to the second point, right now, there's a gap in tools,
aid and data, forensics for tabular data for plagiarism.
There are purpose built tools such as authenticate, which are widely used.
But to our knowledge, no such tool exists for tabular data.
There are various statistical tests that can check
for the plausibility of data, such as Bedford's law, or determining
the probability that a dataset contains a duplicate number.
However, these require careful setup and cannot always be used.
Furthermore,
we worry that applying these tests when it isn't appropriate would result in
false positives viewed with high confidence by the user
because they come with the the authority of rigorous math
as a result, our tool highlights these artifacts of manipulation,
but also embeds advice on what can cause the artifact,
including benign causes
Introducing variant.
On the left hand side, a list of analysis is available.
Each analysis is designed to highlight a different artifact.
The analysis explanation is the embedded advice.
It introduces the pattern, explains what to look for
and gives warnings on how it could be misinterpreted.
The summary
charts provide aggregate information about the column of data below,
even though the visual and coatings are simple, the data transformations
performed are carefully crafted to surface artifacts of manipulation.
Finally, the tabular visualization gives access
to the raw data.
So we've talked about these two
ideas of understanding fraud and creating tools clearly
bear linked understanding fraud, directly guided the design choices affair. In
other words, once you know the patterns, you can create tools to highlight them.
Now, I will spend some time going
into the details for some of the artifacts of manipulation.
For details on the others.
The user referred to the paper or the living document of artifacts.
To begin, I will discuss unexpected
formatting with an example from the driving dataset.
Formatting here does not refer to the values of the data,
but rather things that change the appearance of the data.
In formatting tools like Microsoft Excel.
This includes things like the font, font size and methods
of text emphasis such as bold, italic or underlined.
Variation in formatting is is not automatically suspicious.
For instance, it is common to emphasize column headers.
However, there can be more subtle variations.
Unless you have an incredibly keen eye, you probably didn't notice
that I've formatted these numbers in two different fonts.
While not immediately
incriminating, I would consider this to be
suspicious or at the very least strange and worth investigating.
Further, this exact pattern exists in one of the retracted datasets
this column represents the odometer reading for 13,488 cars.
This formatting difference can be visually subtle.
We do not recreate them in the tool.
Instead, we choose to highlight when there are differences.
In this case blue for Cambria font and white for Calibri
switching to overview mode lets more rows fit on the screen
and sorting the data reveals some interesting patterns.
All of the values of zero in this data
are in the white font.
Most of the values
between 300 and 1,000 are blue.
Scrolling through the data, we can see
regions of white at round numbers,
but we don't see the same effect
for the blue values.
Lastly,
skipping to the largest values in the dataset.
If we select
all of these values and expand them, we can see that the fonts
all alternate perfectly between
blue and white.
Moving on,
I will discuss one type of numerical artifact.
Miracle artifacts relate to an actual values, recorded
duplicate numbers and digits referred to.
One whole numbers
or sequences of digits are repeated more frequently than expected.
This can suggest that data may have been copy pasted
or manually entered evaluated.
If the number of duplicates is more than expected requires understanding
the position of the data and the number of samples.
Finding a few duplicates in a large dataset is not suspicious,
but finding many duplicates and small data sets may warrant a closer look.
This dataset
represents the amount of time in seconds it takes for a spider
to reemerge from its enclosure, a proxy for that spider’s boldness.
600 or 10 minutes is duplicated many times in this dataset.
However, the period was capped at 10 minutes,
so it's reasonable to ignore that in our analysis
in this stage is that there are many values that are duplicated
two or three times watching values
will highlight them and bring them to the top of our list.
Here the number 104 shows up three times, which we may consider
moderately suspicious given the precision and the size of this dataset,
structural artifacts relate to the values of spreadsheets,
as well as their position within the spreadsheet.
A region refers to multiple cells with a special relationship.
This could mean cells that are adjacent in Collins rows
or even nearby cells with gaps in between them.
In some scenarios, a few duplicate numbers may be a weak
signal of manipulation or not suspicious at all.
But repeated regions
are a much stronger signal that something suspicious
is happening.
Our, last example that we've looked at actually includes
an example of this here, a complete row
with the numbers 85, 180, 104.
228.1 and 151.34
seems to have been copied.
Finally, domain artifacts
apply knowledge beyond the single dataset.
In a given domain, there may be prior knowledge about what is an expected
distribution for single dimensional data or expected relationships.
Between multidimensional data.
We consider deviations from these expectations to be artifacts.
Let's consider a hypothetical dataset
that measures the height and weight of people.
It is possible that more sophisticated
techniques for generating fabricated data, such as scripts
may not leave behind the early artifacts we've discussed.
Furthermore, the distribution of each column
may also match to meeting expectations.
However, if cons are generated independently,
they may exhibit strange relationships such as no correlation
when there should be one or even one that's opposite from expectations
as seen here, where taller people tend to wave less.
In this case, it
isn't difficult to think of ways to generate 2 columns
with appropriate relationships,
but this gets more difficult as you introduce more columns.
Every additional column introduces a new set of relationships
that can expose a potential flaw in the data fabrication process.
Looking again at the driving dataset,
we can look at two columns interactions the odometer readings of cars
at one point in time, and the odometer reading
of the same car later in totted.
Each point corresponds to a single car.
A point here would be a car that started with zero miles
on their odometer and ended with approximately 25,000 miles
and this would be a car that started at roughly 100,000
miles and ended at 125,000 miles.
The points on this line correspond to cars
that drove zero miles in this time period.
Similarly, the cars on
this line drove exactly 50,000 miles.
We can notice in this plot here that there are zero cars
that drove more than 50,000 miles, despite many driving near the boundary.
Given that there are many cars between 45,000 and 50,000 thousand miles driven,
we would also expect there to be at least a few with more than 50,000 miles.
That brings us to our last main point.
Preventing manipulation of data will require more than just tools,
and knowledge will require changes to the policies and best practices
of the scientific community.
One idea we think is worth exploring is for journal editors to review data
sets for manipulation similarly to how they currently check for plagiarism.
Individuals can also perform similar checks when they receive data
from their collaborators.
This is a
difficult problem and we don't claim to have all the answers.
There are potential unintended consequences with this work.
For instance, false positives could still lead to misguided accusations.
But because of this, we believe tools like ours should be used in
the review process where authors can respond to concerns with their data.
And the result is a rejected paper, not an accusation
that can threaten an author's career.
There's also a possibility that the bad actors
could use tools like ours to improve their falsification data.
And while some may do this well, we have seen with existing plagiarism
tools as they still continue to catch many instances of cheating.
Still, it is important to consider these types of broader impacts.
We discussed these two concerns as well as others in more detail in our paper.
At the end of the day, even after spending so much time looking at these cases
of data manipulation, we still believe in the scientific community,
but we can and should
do more to prevent the manipulation of tabular datasets.
We have taken a first step and we urge you to consider how you can help
the community understand fraud, build tools and implement
best practices.