Data Visualization and Exploration

Ozan Kahramanoğulları

Yes, I see.

In English we say “I see” to mean “I understand”. So the point is that we understand things better when we see them, because we evolved to find visual patterns quickly.

state	energy		state	energy		state	energy		state	energy
AT	9.374		EL	7.087		IT	10.182		PT	7.254
BA	2.240		ES	8.191		LT	4.680		RO	4.430
BE	6.388		2020	7.672		LU	10.506		RS	2.420
BG	2.226		EU28	8.012		LV	4.379		SE	7.821
CY	6.968		FI	5.453		ME	3.396		SI	5.494
CZ	3.879		FR	8.059		MK	2.853		SK	4.686
DE	8.616		HR	5.355		MT	3.826		TR	6.291
DK	14.007		HU	4.461		NL	7.400		UK	10.620
EA19	8.230		IE	13.620		NO	11.534		XK	2.140
EE	2.806		IS	1.866		PL	4.232			NA

Energy productivity (gross domestic product (GDP) by the gross inland consumption of energy) in 2014 http://data.europa.eu/euodp/en/data/dataset/xWiT1fbpF5q1ZCvLQc2upg

• Choose a story you want to tell, raw data is not very meaningful
• Display just the information you need to support your story

Trump tweeted this during his impeachment campaign. The problem is that the amount of colors do not correlate with the amount of votes.

Visualization is better at summarizing data

Consider the following summaries of a 2 dimensional dataset

• Mean 9, 7.5009091
• Standard deviation 3.3166248, 2.0315681
• Correlation coefficient 0.8164205

This is called Anscombe’s quartet

https://en.wikipedia.org/wiki/Anscombe%27s_quartet

Anscombe’s quartet comprises four data sets that have nearly identical simple descriptive statistics, yet have very different distributions and appear very different when graphed. Each dataset consists of eleven (x, y) points. They were constructed in 1973 by the statistician Francis Anscombe to demonstrate both the importance of graphing data when analyzing it, and the effect of outliers and other influential observations on statistical properties. He described the article as being intended to counter the impression among statisticians that “numerical calculations are exact, but graphs are rough”.[1]

Visualization is better at summarizing data

datasaurus_dozen

https://www.research.autodesk.com/publications/same-stats-different-graphs/

https://cran.r-project.org/web/packages/datasauRus/vignettes/Datasaurus.html

X Mean 54.26 Y Mean 47.83 X SD 16.76 Y SD 26.93 Corr: -0.06

The National Hurricane Center publishes graphics like this one so the public can plan ahead for tropical storms and hurricanes. However, many people who see these maps read them wrong.

Studies show that some people misinterpret the map as indicating the hurricane getting bigger over time. Others think it shows areas under threat. Research by Hurakan, a University of Miami team I’m a part of, suggests 40 percent of people wouldn’t feel threatened if they lived just outside of the cone.

People who are inside the cone, but far from the center, tend to prepare less than those closer to the central line. All of these interpretations of the map are incorrect.

These misunderstandings have real-world consequences. As Hurricane Irma approached Florida’s east coast in 2017, people near the cone’s center began preparing and even evacuating.

When Irma was a few days away, the central path turned gradually to the west coast of Florida, and some people in that region hadn’t done enough preparation for the storm.

How to interpret the National Hurricane Center’s map, then? The cone actually represents a range of possible positions and paths for the storm’s center.

The dots in the middle of the cone correspond to the N.H.C.’s forecast of where the hurricane’s center could be in the following five days. But there’s a good chance that the actual center of the storm won’t end up being at those positions.

To create the cone, the N.H.C. surrounds each estimated position of the storm center with circles of increasing size. These circles represent uncertainty.

This means that the storm center may end up being anywhere inside those circles — and even outside of them. The uncertainty circles grow over time because it’s easier to predict what will happen one day from now than five days from now.

Finally, a curve connects the circles. The result is what is popularly known as the “cone of uncertainty.”

The National Hurricane Center says cones will contain the path of the storm center only 60 to 70 percent of the time. In other words, one out of three times we experience a storm like this, its center will be outside the boundaries of the cone.

The National Hurricane Center says cones will contain the path of the storm center only 60 to 70 percent of the time. In other words, one out of three times we experience a storm like this, its center will be outside the boundaries of the cone.

The cone graphic is deceptively simple. That becomes a liability if people believe they’re out of harm’s way when they aren’t. As with many charts, it’s risky to assume we can interpret a hurricane map correctly with just a glance. Graphics like these need to be read closely and carefully. Only then can we grasp what they’re really saying.

https://www.nytimes.com/interactive/2019/08/29/opinion/hurricane-dorian-forecast-map.html

In this course, we will study techniques
to explore and visualize data
in order to communicate with people.

{.larger .absolute top=200 left=0 width=“350” height=“300”}

The goals of this course

• Give you the tools to carry out your own data analyses

• Learn how to explore data

• How to preprocess and clean data

• How to visualize data in appropriate ways

  • Study human perception

• How to program to develop reproducible analyses

A data exploration workflow

Source: https://r4ds.had.co.nz/explore-intro.html

Second edition: https://r4ds.hadley.nz/intro

First you must import your data into R. This typically means that you take data stored in a file, database, or web application programming interface (API), and load it into a data frame in R. If you can’t get your data into R, you can’t do data science on it!

Once you’ve imported your data, it is a good idea to tidy it. Tidying your data means storing it in a consistent form that matches the semantics of the dataset with the way it is stored. In brief, when your data is tidy, each column is a variable, and each row is an observation. Tidy data is important because the consistent structure lets you focus your struggle on questions about the data, not fighting to get the data into the right form for different functions.

Once you have tidy data, a common first step is to transform it. Transformation includes narrowing in on observations of interest (like all people in one city, or all data from the last year), creating new variables that are functions of existing variables (like computing speed from distance and time), and calculating a set of summary statistics (like counts or means). Together, tidying and transforming are called wrangling, because getting your data in a form that’s natural to work with often feels like a fight!

Once you have tidy data with the variables you need, there are two main engines of knowledge generation: visualisation and modelling. These have complementary strengths and weaknesses so any real analysis will iterate between them many times.

Visualisation is a fundamentally human activity. A good visualisation will show you things that you did not expect, or raise new questions about the data. A good visualisation might also hint that you’re asking the wrong question, or you need to collect different data. Visualisations can surprise you, but don’t scale particularly well because they require a human to interpret them.

Models are complementary tools to visualisation. Once you have made your questions sufficiently precise, you can use a model to answer them. Models are a fundamentally mathematical or computational tool, so they generally scale well. Even when they don’t, it’s usually cheaper to buy more computers than it is to buy more brains! But every model makes assumptions, and by its very nature a model cannot question its own assumptions. That means a model cannot fundamentally surprise you.

The last step of data science is communication, an absolutely critical part of any data analysis project. It doesn’t matter how well your models and visualisation have led you to understand the data unless you can also communicate your results to others.

Surrounding all these tools is programming. Programming is a cross-cutting tool that you use in every part of the project. You don’t need to be an expert programmer to be a data scientist, but learning more about programming pays off because becoming a better programmer allows you to automate common tasks, and solve new problems with greater ease.

Reproducibility

Reproducibility

Exploration is useless if you don’t draw a map to repeat your steps

Reproducibility is the key!

• Someone questions your conclusions.

• One year later, you want to re-run the analysis with new data.

• One year later, you want to slightly modify the analysis.

• You are collaborating with someone else.

The three flavours of reproducibility

• Repeatability

• Reproducibility

• Replicability

Repeatability / Reproducibility / Replicability

Repeatability (Same team, same experimental setup): The measurement can be obtained with stated precision by the same team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same location on multiple trials. For computational experiments, this means that a researcher can reliably repeat her own computation.

Source: https://www.acm.org/publications/policies/artifact-review-and-badging-current

Definitions of the ACM (Association of Computing Machinery)

Repeatability / Reproducibility / Replicability

Reproducibility (Different team, same experimental setup): The measurement can be obtained with stated precision by a different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using the author’s own artifacts.

Source: https://www.acm.org/publications/policies/artifact-review-and-badging-current

Definitions of the ACM (Association of Computing Machinery)

Repeatability / Reproducibility / Replicability

Replicability (Different team, different experimental setup): The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently.

Source: https://www.acm.org/publications/policies/artifact-review-and-badging-current

Definitions of the ACM (Association of Computing Machinery)

Repeatability / Reproducibility / Replicability

• Repeatability: same team, same setup

• Reproducibility: different team, same setup

• Replicability: different team, different setup

You want your results to be at least reproducible.

Attention: if you are not careful enough,
you may have something that is
not even repeatable!

It is important to aim at replicability
from the very beginning.
It cannot be an afterthought!

How to achieve the 3 Rs?

• Take notes!

• Save all the information about your inputs

• Save all the processing you do on your data

• Snapshot your programming environment

• Keep everything under version control

Some practical information

About me

• Ozan Kahramanoğulları
• email: okahramanogullari@unibz.it
• office: B1.5.22 (NOI)

About the course

• Lectures
  • Monday: 10:00 - 12:00
  • Thursday: 08:00 - 10:00
• Lab
  • Thursday: 15:30 - 17:30
• Office hours: schedule by email
  • Wednesday: 14:00 - 16:00

.pull-right.larger[

Overlap with Programming for Data Analytics

It’s a feature, not a bug! :-)

]

About the exam

You should complete at least 60% of your weekly assignments.

Your final grade is:

About the material

• Slides are available online in OLE.

• There are three free books on the internet.

  • Data Visualization. A practical introduction. Haley.
  • Fundamentals of Data Visualization. Wilke.
  • R for Data Science. Wickham.

• We will also refer to this paper:

  • A layered grammar of graphics. Wickham.

• I will do a lot of live coding:

  • you are welcome to tag along and try things live.

Some questions for you

• Which programming languages do you know?

• Which version control systems do you know?

• Do you have prior experience with visualization?