Data wrangling verbs

The native pipe |> and the verbs in the dplyr package

  • filter()
  • summarize()
  • group_by()
  • mutate()
  • arrange()
  • inner_join()
  • select()

The pipe operator

A large pipe discharging water into a channel, the picture behind the name of the pipe operator

|>

Take this data frame, then… filter the data, then with those results… summarize

|> |>

  • “Pipe” a data frame into a “verb” command
  • “Chain” the results from one “verb” command into another
  • Think of it as the word “then”

filter()

Pull out just the rows you want in a data set, based on some criteria

Cheat sheet crop, Subset Observations (Rows): a table of five rows becomes a table of three rows, with the rows that are kept highlighted

filter()

Example

The Subset Observations diagram again: some rows are kept and the rest are dropped

the data frame

on_time_flights <- flights |>
  filter(arr_delay < 30)

the verb (function) what column (variable) to use for filtering

only keep rows where the column meets this criterion

filter()

Another example

The Subset Observations diagram again: some rows are kept and the rest are dropped

the data frame

newark_flights <- flights |>
  filter(origin == "EWR")

the verb (function) what column (variable) to use for filtering

only keep rows where the column meets this criterion

Comparisons: == != < > <= >=, combined with & (and) or | (or)

summarize()

Take a column of data from a data frame and reduce it down to a single summary statistic

Cheat sheet crop: a column of five cells passes through a wedge labeled summary function and comes out as a single cell

Cheat sheet crop, Summarise Data: a table of four columns and three rows becomes a table of one row of summaries

summarize()

Example

The summary function diagram again: a column of cells reduced to a single value

the data frame

flights |>
  summarize(av_dist = mean(distance))

the verb (function) what to call the result the column to summarize

what summary statistic to calculate

group_by() then summarize()

Take a column of data and reduce it down to a summary statistic, by some grouping variable

Cheat sheet crop in three steps: a plain table, then the same table with its rows colored into two groups, then one row per group

group_by() then summarize()

Example

The grouped summary diagram again: rows sorted into groups, then one row per group

the data frame

flights |>
  group_by(origin) |>
  summarize(av_dist = mean(distance))

the verb (function) what to call the result the column to summarize

group_by(), then what to group by

mutate()

Add a new column to a data frame, computed from the columns already there

Cheat sheet crop, Make New Variables: a table of four columns becomes a table of five, the new column added on the right

mutate()

Example

The Make New Variables diagram again: a new column added on the right of the table

the data frame

flights |>
  mutate(dist_km = distance * 1.609)

the verb (function) what to call the new column (new variable) what to put in the new column

arrange()

Sort the rows of a data frame by the values in one of its columns

arrange()

Example

the data frame

freq_dest |>
  arrange(desc(num_flights))

the verb (function) which column to sort by, largest first

freq_dest has one row per destination and its number of flights, made with group_by(dest) and summarize().

inner_join()

Combine two data frames by matching rows on a key column, keeping only the keys found in both

Data frame x has ids 1, 2, and 3 with values x1, x2, and x3; data frame y has ids 1, 2, and 4 with values y1, y2, and y4. Their inner join keeps only ids 1 and 2, the ids in both, each with its x value and its y value side by side in columns val.x and val.y

Diagram adapted from Garrick Aden-Buie’s tidyexplain

inner_join()

Example

the data frame

flights_joined <- flights |>
  inner_join(airlines, by = "carrier")

the verb (function) the second data frame the key column both share

select()

Pull out just the columns you want in a data set, based on the column names

Cheat sheet crop, Subset Variables (Columns): a table of five columns becomes a table of three, with the columns that are kept highlighted

select()

Example

The Subset Variables diagram again: some columns are kept and the rest are dropped

the data frame

flights_sub <- flights |>
  select(arr_delay, dep_delay)

the verb (function) what columns (variables) to keep