Data manipulation using the Tidyverse

Session 2 practical exercises

Have you heard about Murphy law?

Yesterday, you were instructed to become familiar with IMD data from the department, as your supervisor is planning your first surveillance project around it. During the weekly department meeting, Kassandra, the head of department for infectious diseases epidemiology and surveillance, informed the team about an email shared by one of the regions, R6, expressing concern about a recent rise in IMD cases among female young adults.

They asked your department for assistance in evaluating this signal, especially for adding national context and background information about other regions. Your colleagues start organizing a small task force, and Kassandra immediately appoints you to the team so you can get some first-hand experience at your new Institute. Your task will be to analyze IMD surveillance data from previous years of the affected region, and a few otherregions for comparison. It’s a cleaned, ready-to-use dataframe that epidemiologists in the unit know very well, with annual reports published already.

The perfect task for a brand new EPIET and EUPHEM fellow.

Before you start

Yesterday, you learned how to set up an R Project, start a script, use packages, and apply functions to your data. To continue with today’s session, you should open the saved project IntroToRCourse.Rproj.

Action — Start a new script and save it as S2_tidyverse.

Action — Prepare the script for today’s work: add initial notes, activate libraries and load data.

Remember you will need libraries for managing file paths and for importing data to R.
We will add a new library from now on: the tidyverse.
Today’s dataframe is found in the data/clean folder: IMD_Sample_Clean.xlsx. Load it and assign it to a new object called imd. Once loaded, click on your imd object in the Environment window to open the data viewer and explore the data for a while: familiarize yourself with the variables, the structure and some of the content.

💡 Show solution — only after trying yourself!

# Data import
imd <- import(here("data", "clean", "IMD_Sample_Clean.xlsx"))

Now, what is this tidyverse we keep talking about??

Part 1 · The pipe — a new way to think about code

Filter: retaining the meaningful

To start your task, you first need to think about the instructions you were given and how to translate them into code. This is the Grammar Model we are practicing. Remember the signal that arrived:

Increase in cases in R6 among female young adults

This is a sequence of three steps that are needed to explore the cases of concern. Let’s start by working slowly.

The first thing we need to do is keep only the rows that match the criteria from the signal. The function we need for this task is filter(), the row selection function from tidyverse. This is how the functions works:

# General use of a filter function
filter(data, condition)

The function needs as first argument the data it will use, and then the logical condition that rows need to fulfill in order to be retained, for example, cases from the region R6.

Action — Filter imd cases to only those happening in region R6. You can do this using this code:

filter(imd, region_id == "R6")

Let’s break down the code:

The data argument is imd, our imported working dataframe.
The condition argument uses the variable region_id from the dataframe. This is the variable we want to check.
- for the logical condition, we used == which is the logical operator for “is equal to”
- to select our region of interest, we used "R6" with quotes because it is a character value in the region_id variable.

Action — copy the filter() code in your script and execute it. What happens?

Yeah, you are right! We did not assign the function to any object, therefore the results only got printed in the console and, therefore, was lost to the R Void forever.

Action — assign the filter operation to a new object imd_filtered.

Cool, but that was only the first of three steps, we have two more to go! Let’s do it together in a simple way:

imd_filtered         <- filter(imd, region_id == "R6")
imd_filtered_f       <- filter(imd_filtered, sex == "Female")
imd_filtered_f_adult <- filter(imd_filtered_f, age_years >= 18)

And again, let’s break it down:

As a sequence, every step relied on the previous result: first we filtered the region, then that new dataframe was filtered for sex, and the resulting data for age. Every new step required the creation of a new object, and using it in the next function.
Each filtering used the adequate variable: region_id, sex and age_years and a logical operator.

Working in such a step-wise way is useful: anyone reading your code can very quickly get the idea of the transformations that are happening, and that is always a relevant aspect to consider when writing code. Of course, it has downside of requiring more text, more objects, and unnecessary repetition that can lead to mistakes.

Steps can be chained together. Before Tidy, the only way of chaining commands was called nesting, which consist in using the data argument of the function to write another function instead of direct data. You could then build matrioshka code like this one:

filter(filter(filter(imd, region_id == "R6"), sex == "Female"), age_years >= 18)

Now when you read this, your head needs a few minutes to understand what is happening. To really come up with the sequence, you need to start reading in the middle where imd is called for the first time, then it becomes a series of nested functions, arranged inside-out. Not beautiful, but does the work in a single command, and saves you three objects.

The pipe as “and then”

Now read this code:

imd %>%
  filter(region_id == "R6") %>%
  filter(sex == "Female") %>%
  filter(age_years >= 18)

Same result. But now the data flows downward, one step at a time, exactly in the order the steps happen. This is the logic of the pipe operator %>%

%>% has one job: take what is on the left and pass it as the first argument to what is on the right. Read it as “and then”:

Take imd, and then keep only cases from R6, and then keep only female cases, and then keep only the adults.

Three properties of this style that will save you time:

The dataframe is declared once, at the top. It is never repeated.
Variable names can be called directly within tidyverse functions, not requiring quotation or $ like base R.
Code reads top to bottom, like a recipe, each step acts on the result of the step above.

The pipeline is a transformation chain

Think of the pipe as an assembly line. The raw material (imd) enters at the top, passes through a series of stations (the verbs), and exits the other end transformed. Each station receives whatever came out of the previous one. Nothing is modified in place, a new object only exists if you assign it with <-.

Your first pipeline

Action — In your script, write the following and run it:

imd %>%
  filter(region_id == "R6")

Look at the result in the console. How many rows does it have? It should be fewer than the full dataset — you have just retained only cases from R6.

Action — Now assign the result to a new object and inspect it:

imd_region <- imd %>%
  filter(region_id == "R6")

The original data is untouched

Running a pipeline does not modify imd. Until you use <- to assign the result to an object, the transformation exists only as output in the console — it is printed, not saved. This is intentional: your raw data stays intact, and you build on it without risk.

Action — Confirm this by running nrow(imd) after creating imd_region.

What does the %>% operator do?

It modifies the dataframe in place, updating it with the result of each step It takes the result on its left and passes it as input to the function on its right It runs two functions simultaneously and combines their output It saves the result of the previous step to a new object automatically

In a tidy pipeline, how many times do you write the name of the dataframe?

Once, at the top of the pipeline — it is then passed automatically to each step In every function call, so each step knows which data to use Only in the last step, where the final result is produced It does not need to be declared — tidyverse finds it automatically

After running imd_female <- imd %>% filter(sex == "Female"), what happens to the original imd object?

The pipeline modifies imd directly, so imd now contains only female cases imd is unchanged — the result only exists in imd_female because you assigned it with <- Both imd and imd_female are updated, since they reference the same data The pipeline discards the result unless you print it with print()

Exercise Summary

Now you have a grasp of what actually the pipe %>% operator means when working with a Tidy workflow. In the following exercises you will learn more about its importance, and the huge advantages of working with it. We are just getting started!!

This is what you learned:

Function What it does

Function	What it does
`%>%`	Pipe operator, chains tidy functions together sequentially, passing modified data to the next action Shortcut: Win: Ctrl + Shift + M Mac: Cmd + Shift + M
`filter()`	Retains rows matching a condition

%>%

Pipe operator, chains tidy functions together sequentially, passing modified data to the next action

Shortcut:

Win: Ctrl + Shift + M
Mac: Cmd + Shift + M

filter() Retains rows matching a condition

--- title: "Data manipulation using the Tidyverse" subtitle: "Session 2 practical exercises" --- ```{r} #| include: false library(webexercises) library(pacman) pacman::p_load(tidyverse, rio, here) imd <- import(here("data", "clean", "IMD_Sample_Clean.xlsx")) ``` **Have you heard about Murphy law?** Yesterday, you were instructed to become familiar with IMD data from the department, as your supervisor is planning your first surveillance project around it. During the weekly department meeting, Kassandra, the head of department for infectious diseases epidemiology and surveillance, informed the team about an email shared by one of the regions, `R6`, expressing concern about a recent rise in IMD cases among female young adults. They asked your department for assistance in evaluating this signal, especially for adding national context and background information about other regions. Your colleagues start organizing a small task force, and Kassandra immediately appoints you to the team so you can get some first-hand experience at your new Institute. Your task will be to analyze IMD surveillance data from previous years of the affected region, and a few otherregions for comparison. It's a cleaned, ready-to-use dataframe that epidemiologists in the unit know very well, with annual reports published already. *The perfect task for a brand new EPIET and EUPHEM fellow.* ## Before you start Yesterday, you learned how to set up an R Project, start a script, use packages, and apply functions to your data. To continue with today's session, you should open the saved project `IntroToRCourse.Rproj`. **Action** — Start a new script and save it as `S2_tidyverse`. **Action** — Prepare the script for today's work: add initial notes, activate libraries and load data. 1. Remember you will need libraries for managing file paths and for importing data to R. 2. We will add a new library from now on: the `tidyverse`. 3. Today's dataframe is found in the `data/clean` folder: `IMD_Sample_Clean.xlsx`. Load it and assign it to a new object called `imd`. Once loaded, click on your `imd` object in the Environment window to open the data viewer and explore the data for a while: familiarize yourself with the variables, the structure and some of the content. ::: {.callout-tip collapse="true"} ## 💡 Show solution — only after trying yourself! ``` r # Data import imd <- import(here("data", "clean", "IMD_Sample_Clean.xlsx")) ``` ::: Now, what is this `tidyverse` we keep talking about?? ## Part 1 · The pipe — a new way to think about code ## Filter: retaining the meaningful  To start your task, you first need to think about the instructions you were given and how to translate them into code. This is the Grammar Model we are practicing. Remember the signal that arrived: - Increase in cases in `R6` among `female` `young adults` This is a ***sequence*** of three steps that are needed to explore the cases of concern. Let's start by working slowly. The first thing we need to do is keep only the rows that match the criteria from the signal. The function we need for this task is `filter()`, the row selection function from `tidyverse`. This is how the functions works: ``` r # General use of a filter function filter(data, condition) ``` The function needs as first argument the data it will use, and then the logical condition that rows need to fulfill in order to be retained, for example, cases from the region `R6`. **Action** — Filter `imd` cases to only those happening in region `R6`. You can do this using this code: ``` r filter(imd, region_id == "R6") ``` Let's break down the code: - The data argument is `imd`, our imported working dataframe. - The condition argument uses the variable `region_id` from the dataframe. This is the variable we want to check. - for the logical condition, we used `==` which is the logical operator for "is equal to" - to select our region of interest, we used `"R6"` with quotes because it is a character value in the `region_id` variable. **Action** — copy the `filter()` code in your script and execute it. ***What happens?*** Yeah, you are right! We did not assign the function to any object, therefore the results only got printed in the console and, therefore, was lost to the *R Void* forever. **Action** — assign the filter operation to a new object `imd_filtered`. Cool, but that was only the first of three steps, we have two more to go! Let's do it together in a simple way: ``` r imd_filtered <- filter(imd, region_id == "R6") imd_filtered_f <- filter(imd_filtered, sex == "Female") imd_filtered_f_adult <- filter(imd_filtered_f, age_years >= 18) ``` And again, let's break it down: - As a sequence, every step relied on the previous result: first we filtered the region, then that new dataframe was filtered for sex, and the resulting data for age. Every new step required the creation of a new object, and using it in the next function. - Each filtering used the adequate variable: `region_id`, `sex` and `age_years` and a logical operator. Working in such a step-wise way is useful: anyone reading your code can very quickly get the idea of the transformations that are happening, and that is always a relevant aspect to consider when writing code. Of course, it has downside of requiring more text, more objects, and unnecessary repetition that can lead to mistakes. Steps can be chained together. Before `Tidy`, the only way of chaining commands was called **nesting**, which consist in using the *data* argument of the function to write another function instead of direct data. You could then build matrioshka code like this one: ``` r filter(filter(filter(imd, region_id == "R6"), sex == "Female"), age_years >= 18) ``` Now when you read this, your head needs a few minutes to understand what is happening. To really come up with the sequence, you need to start reading in the middle where `imd` is called for the first time, then it becomes a series of nested functions, arranged inside-out. Not beautiful, but does the work in a single command, and saves you three objects. ### The pipe as "and then" Now read this code: ``` r imd %>% filter(region_id == "R6") %>% filter(sex == "Female") %>% filter(age_years >= 18) ``` Same result. But now the data flows *downward*, one step at a time, exactly in the order the steps happen. This is the logic of the **pipe operator** `%>%` `%>%` has one job: take what is on the left and pass it as the first argument to what is on the right. Read it as **"and then"**: > Take `imd`, **and then** keep only cases from R6, **and then** keep only female cases, **and then** keep only the adults. Three properties of this style that will save you time: - The dataframe is declared ***once***, at the top. It is never repeated. - Variable names can be called ***directly*** within tidyverse functions, not requiring quotation or `$` like base R. - Code reads ***top to bottom***, like a recipe, each step acts on the result of the step above. ::: callout-note ### The pipeline is a transformation chain Think of the pipe as an assembly line. The raw material (`imd`) enters at the top, passes through a series of stations (the verbs), and exits the other end transformed. Each station receives whatever came out of the previous one. Nothing is modified in place, a new object only exists if you assign it with `<-`. ::: ### Your first pipeline **Action** — In your script, write the following and run it: ``` r imd %>% filter(region_id == "R6") ``` Look at the result in the console. How many rows does it have? It should be fewer than the full dataset — you have just retained only cases from R6. **Action** — Now assign the result to a new object and inspect it: ``` r imd_region <- imd %>% filter(region_id == "R6") ``` ::: callout-important ### The original data is untouched Running a pipeline does **not** modify `imd`. Until you use `<-` to assign the result to an object, the transformation exists only as output in the console — it is printed, not saved. This is intentional: your raw data stays intact, and you build on it without risk. ::: **Action** — Confirm this by running `nrow(imd)` after creating `imd_region`. ------------------------------------------------------------------------ ```{r} #| echo: false opts1 <- c( "It modifies the dataframe in place, updating it with the result of each step", answer = "It takes the result on its left and passes it as input to the function on its right", "It runs two functions simultaneously and combines their output", "It saves the result of the previous step to a new object automatically" ) ``` **What does the `%>%` operator do?** `r longmcq(opts1)` ```{r} #| echo: false opts2 <- c( answer = "Once, at the top of the pipeline — it is then passed automatically to each step", "In every function call, so each step knows which data to use", "Only in the last step, where the final result is produced", "It does not need to be declared — tidyverse finds it automatically" ) ``` **In a tidy pipeline, how many times do you write the name of the dataframe?** `r longmcq(opts2)` ```{r} #| echo: false opts3 <- c( "The pipeline modifies `imd` directly, so `imd` now contains only female cases", answer = "`imd` is unchanged — the result only exists in `imd_female` because you assigned it with `<-`", "Both `imd` and `imd_female` are updated, since they reference the same data", "The pipeline discards the result unless you print it with `print()`" ) ``` **After running `imd_female <- imd %>% filter(sex == "Female")`, what happens to the original `imd` object?** `r longmcq(opts3)` ------------------------------------------------------------------------ ## Exercise Summary Now you have a grasp of what actually the pipe `%>%` operator means when working with a Tidy workflow. In the following exercises you will learn more about its importance, and the huge advantages of working with it. We are just getting started!! This is what you learned: +---------------+------------------------------------------------------------------------------------------------------+ | Function | What it does | +===============+======================================================================================================+ | `%>%` | Pipe operator, chains tidy functions together sequentially, passing modified data to the next action | | | | | | Shortcut: | | | | | | - Win: Ctrl + Shift + M | | | - Mac: Cmd + Shift + M | +---------------+------------------------------------------------------------------------------------------------------+ | `filter()` | Retains rows matching a condition | +---------------+------------------------------------------------------------------------------------------------------+ ```{=html} <script> document.addEventListener("DOMContentLoaded", function() { var radiogroups = document.getElementsByClassName("webex-radiogroup"); for (var i = 0; i < radiogroups.length; i++) { radiogroups[i].onchange = radiogroups_func; } }); </script> ```