Logical conditions and Tidy

Session 2 practical exercises

You already know how to build a pipeline and are starting to grasp the logic of tidy data. We introduced the filter() function in the previous exercise in a simple manner, without entering in its more complicated aspects. And, partially, that happens because the function itself is not they key aspect of row selection. That role belongs to logical statements — the true heart not only of filter(), but also of the whole programming experience.

Now you need to learn how to tell R which rows you want — and that means learning the language of logical conditions. A single condition is rarely enough in real epidemiological work, so you will also need to learn to combine them.

To do that precisely, you need logical operators — the symbols R uses to evaluate whether a condition is true or false. Here is the full set you will be using today:

Operator	Meaning
`==`	Exactly equal to, meaning absolute equivalence; “identical”
`!=`	Not equal to; “different”
`>`, `>=`	Greater than / greater than or equal to
`<`, `<=`	Less than / less than or equal to
`&`	AND — both conditions must be true at the same time
`\|`	OR — at least one condition must be true
`!`	NOT — negates the condition that follows; “opposite”
`%in%`	Present in a set of values; “is part of”
`is.na()`	Identifies missing values (`NA`)

Keep this table in view. You will use every operator today.

Part 2 · Logical operators

Equality and comparison

The simplest conditions to check are whether a variable equals a specific value, or falls above or below a numeric threshold. You have already seen those a couple times in this course, for example when we filtered only cases from R6.

imd_region <- imd %>%
  filter(region_id == "R6")

Filters need to be defined using the appropriate variable type. Here, region_id is a character (text) variable, therefore the value "R6" is written in quotes. Try running the same code without the quotes to see the error message.

From now on, this is the format of code we want to start emulating: calling the data and then using functions.

Action — Now filter for cases where age_years is 18 or older. Assign it to an object imd_adults.

Action — Filter for cases where age_years is strictly less than 18. Assign it to an object imd_kids. Do the two results together add up to the total rows in imd?

As we saw with R6, quotation marks indicate that a value is a character variable in R. Now try the same with age_years: use "18" instead of the numeric value 18 to see the error message. Take home message: Make sure the value you use in the logical condition has the appropriate type you are using or it won’t work.

Note that we will take a deep dive into variable classes tomorrow, learning how to check and change it

Hint - counting rows

Counting the number of rows of filtered dataframes is a wise action when working with large amounts of data. Since we are talking about logical operators, they can also be used outside functions, as pure condition checks. For example, you can type:

nrow(imd) == nrow(imd_adults) + nrow(imd_kids)

as a control to ask “Is the sum of the rows in the two filtered dataframes exactly equal to the original number of rows in imd?“

Well, turns out we have missing rows. But, how is that even possible? If the age logical conditions mean that someone is either younger than 18, or 18 and older, shouldn’t all cases be covered by the < and the >= conditions. The answer to that question is:

Action — Might there be cases with missing age_year information? We can check by creating imd_age_missing and filtering using the is.na() operator. Then, check again if all rows are covered across the three objects.

Checking rows is your first quality control

After every filter(), glance at the number of rows in the output. Does it go down when you expect it to? By roughly the amount you would expect? A filter that retains the same number of rows as the original — or drops to zero — is almost always a sign that something went wrong: a typo in the value, the wrong variable name, or a condition that is never true. There are many ways of checking this number: in the Environment window, opening the data viewer, calling the object in the console, using nrow() or other functions.

Combining conditions

Most real filters combine more than one condition. The & and | operators let you do this — but they behave very differently, and confusing them is one of the most common sources of silent errors in data analysis.

Action — Filter imd to keep only female cases from R6, combining both conditions with &. Assign the result to imd_r6_f and type below the row count.

When using multiple conditions inside filter(), the default behavior is treating them as AND (&) operators. You can either separate the conditions with commas, like arguments; create a single condition using the & operator; or pipe two filters together. And the result would be the same:

# Remember, you can achieve the same result with these three codes
# Comma-separated conditions
imd %>% filter(condition1, condition2)

# Conditions combined with the AND operator
imd %>% filter(condition1 & condition2)

# Two consequtive filters
imd %>% 
  filter(condition1) %>% 
  filter(condition2)

And it makes sense! With &, R evaluated both conditions together. With comma-separated conditions or two consecutive filters, the same two conditions are applied, so the result is the same. First retaining only cases from R6 and then, among these retained only Female cases. That is only valid for AND (&) statements. If we try the same with other operators, the results change slightly.

Action — Filter cases that are either from R6 or Female using the | (OR) operator, creating the object imd_r6_or_f and compare the resulting number of rows with imd_r6_f.

Can you understand why the difference?

| is wider than you think

With |, a row is retained if either condition is true — it does not need to satisfy both. This means your result will include all female cases from every region, plus all cases from R6 regardless of sex. In most surveillance contexts, this is not what you want. When in doubt, reach for &.

Selecting from a set with `%in%`

Suppose you need to filter for cases from several specific regions — say, R1, R6, and R9. You could write:

imd_regions <- imd %>% 
  filter(region_id == "R1" | region_id == "R6" | region_id == "R9")

This works, but the code grows unwieldy fast. The %in% operator is the clean solution: it checks whether a value is present anywhere in a set you define. Depending on the complexity and number of options, you can define the values directly using a vector in the filter function, or you could first create an object to serve as intermediary.

Therefore, you could write any of the following:

imd %>% 
  filter(variable1 %in% c("value1", "value2", "value3"))

# or: 
values <- c("value1", "value2", "value3")
imd %>% 
  filter(variable1 %in% values)

To create a vector with multiple values, R uses the c() function, where c stands for combine, i.e. c("value1", "value2", "value3").

Action — Filter imd to keep cases from regions R1, R6, and R9 using %in%. Assign the result to imd_regions.

Action — Now add a condition to also keep only female cases aged 18 or older. Assign the final result to imd_filtered. How many rows are left?

Hint

You can chain all conditions in a single filter() using commas, or write them as separate filter() steps in the same pipeline — both work.

Missing values

You have already used is.na() to find cases with missing age information. But there is something deeper going on with missing values that is worth understanding — because it will save you from a very common and very silent mistake.

The critical thing to know: you cannot use == to find NA. Try it in your console:

NA == NA

You might expect TRUE. R returns NA. The logic is sound: if a value is unknown, you cannot confirm it is equal to another unknown value. Two unknowns are not necessarily the same unknown. So R does not guess — it returns “I don’t know”. is.na() exists precisely to solve this. It does not compare — it simply asks “is this value missing?”, and returns TRUE or FALSE.

Action — Check how many cases in imd have a missing value in the serogroup variable? Create the imd_serogroup_na object for that and type the number below:

Finding missing values is a common task when doing data cleaning, but that is tomorrow’s topic. For now, we are only interested in discarding cases with NA values in some of their variables, something you will also do quite frequently in the future - believe me.

For that, we can introduce the last of today’s operators, the ! - opposite or negation operator. When placed right before another condition, it tells R to do the exact opposite. Therefore, if we want to keep rows NOT having NA values in some column, we would do something like:

imd %>% 
  filter(!is.na(variable1))

Action — Create a new object imd_age_na that contains only cases with missing age_years values. How many cases are there?

Action — Rather than an object with missing age_years cases, we want a dataframe without them this time. Call it imd_age_clean. Use the same filter with ! to achieve that. How many rows are left?

Optional — How can you use nrow() to check that imd_age_clean has effectively filtered the adequate number of rows? This one can be trickier, but give it a try!

What does filter(age_years > 18 | sex == "Female") return?

It keeps rows where both conditions are true It keeps rows where at least one condition is true It keeps rows where neither condition is true It keeps rows where exactly one condition is true, but not both

Which of the following correctly filters cases from regions R1, R6, and R9?

region_id == "R1" | region_id == "R6" | region_id == "R9" region_id == c("R1", "R6", "R9") region_id %in% c("R1", "R6", "R9") region_id & c("R1", "R6", "R9")

Why can’t you use == NA to find missing values in filter()?

Because NA == NA returns NA, not TRUE — R cannot confirm two unknown values are equal Because NA is stored as a number and cannot be compared with == Because filter() automatically removes NA before evaluating conditions Because == only works with character variables, not missing values

Which expression correctly keeps only rows where serogroup is not missing?

filter(is.na(serogroup) == FALSE) filter(serogroup != NA) filter(is.na(serogroup) == 0) filter(!is.na(serogroup))

Exercise summary

Logical operators are the engine behind filter() — and behind most of the decisions your code will ever make. You now know how to select rows by equality, by range, by set membership (using %in%), and how to handle the special case of missing values. Combined with the pipe, filter() gives you precise control over which data you are actually working with at any point in your analysis.

But this is only the tip of the Tidy iceberg, so let’s continue to the verb of creation: mutate().

💡 Show solution — only after trying yourself!

# Filter cases from Region 6
imd_region <- imd %>% filter(region_id == "R6")

# Filter adults and kids cases
imd_adults <- imd %>% filter(age_years >= 18)
imd_kids <- imd %>% filter(age_years < 18)

# Check the row sums of the filtered vs original data
nrow(imd) == nrow(imd_kids) + nrow(imd_adults)    # what is happening?

# Are there any missing age cases?
imd_age_missing <- imd %>% filter(is.na(age_years))

# Check again
nrow(imd) == nrow(imd_kids) + nrow(imd_adults) + nrow(imd_age_missing) # now it works!

# Filter both Region 6 AND Female cases
imd_r6_f <- imd %>%
  filter(region_id == "R6" & sex == "Female") # order of conditions doesn't matter

# Filter Region 6 OR Female cases
imd_r6_or_f <- imd %>%
  filter(region_id == "R6" | sex == "Female")

# Filter cases from any of R1, R6 or R9 regions
imd_regions <- imd %>% 
  filter(region_id == "R1" | region_id == "R6" | region_id == "R9")
  
# better option using %in%
imd_regions <- imd %>%
  filter(region_id %in% c("R1", "R6", "R9"))

# Combine all the different filters together
imd_filtered <- imd %>%
  filter(region_id %in% c("R1", "R6", "R9"),
         age_years >= 18,
         sex == "Female")

# Is there any missing case in serogroup?
imd_serogroup_na <- imd %>%
  filter(is.na(serogroup))

# Missing age cases
imd_age_na <- imd %>% 
  filter(is.na(age_years))

# Dataset without missing age_year cases
imd_age_clean <- imd %>% 
  filter(!is.na(age_years))

# We can check the difference in rows is correct
nrow(imd_age_clean) == (nrow(imd) - nrow(imd_age_na))

--- title: "Logical conditions and Tidy" subtitle: "Session 2 practical exercises" --- ```{r} #| include: false library(webexercises) library(pacman) pacman::p_load(tidyverse, rio, here) imd <- import(here("data", "clean", "IMD_Sample_Clean.xlsx")) ``` ```{=html}  ``` You already know how to build a pipeline and are starting to grasp the logic of tidy data. We introduced the `filter()` function in the previous exercise in a simple manner, without entering in its more complicated aspects. And, partially, that happens because the function itself is not they key aspect of row selection. That role belongs to logical statements — the true heart not only of `filter()`, but also of the whole programming experience. Now you need to learn how to tell R *which rows you want* — and that means learning the language of logical conditions. A single condition is rarely enough in real epidemiological work, so you will also need to learn to combine them. To do that precisely, you need **logical operators** — the symbols R uses to evaluate whether a condition is true or false. Here is the full set you will be using today: | Operator | Meaning | |------------------|------------------------------------------------------| | `==` | Exactly equal to, meaning absolute equivalence; "*identical*" | | `!=` | Not equal to; "*different*" | | `>`, `>=` | Greater than / greater than or equal to | | `<`, `<=` | Less than / less than or equal to | | `&` | AND — both conditions must be true at the same time | | `|` | OR — at least one condition must be true | | `!` | NOT — negates the condition that follows; "*opposite*" | | `%in%` | Present in a set of values; "*is part of*" | | `is.na()` | Identifies missing values (`NA`) | Keep this table in view. You will use every operator today. ## Part 2 · Logical operators ### Equality and comparison The simplest conditions to check are whether a variable equals a specific value, or falls above or below a numeric threshold. You have already seen those a couple times in this course, for example when we filtered only cases from `R6`. ``` r imd_region <- imd %>% filter(region_id == "R6") ``` ::: column-margin Filters need to be defined using the appropriate variable type. Here, `region_id` is a character (text) variable, therefore the value `"R6"` is written in quotes. Try running the same code without the quotes to see the error message. ::: From now on, this is the format of code we want to start emulating: calling the data *and then* using functions. **Action** — Now filter for cases where `age_years` is *18 or older*. Assign it to an object `imd_adults`. **Action** — Filter for cases where `age_years` is strictly *less than* 18. Assign it to an object `imd_kids`. Do the two results together add up to the total rows in `imd`? ::: column-margin As we saw with `R6`, quotation marks indicate that a value is a character variable in R. Now try the same with `age_years`: use `"18"` instead of the numeric value `18` to see the error message. Take home message: Make sure the value you use in the logical condition has the appropriate type you are using or it won't work. Note that we will take a deep dive into variable classes tomorrow, learning how to check and change it :::  ::: {.callout-tip collapse="true" appearance="simple" icon="false"} ## Hint - counting rows Counting the number of rows of filtered dataframes is a wise action when working with large amounts of data. Since we are talking about logical operators, they can also be used outside functions, as pure condition checks. For example, you can type: ``` r nrow(imd) == nrow(imd_adults) + nrow(imd_kids) ``` as a control to ask "*Is the sum of the rows in the two filtered dataframes exactly equal to the original number of rows in `imd`?"* ::: Well, turns out we have missing rows. But, how is that even possible? If the age logical conditions mean that someone is either younger than 18, or 18 and older, shouldn't all cases be covered by the `<` and the `>=` conditions. The answer to that question is: **Action** — Might there be cases with missing `age_year` information? We can check by creating `imd_age_missing` and filtering using the `is.na()` operator. Then, check again if all rows are covered across the three objects. ::: callout-important ### Checking rows is your first quality control After every `filter()`, glance at the number of rows in the output. Does it go down when you expect it to? By roughly the amount you would expect? A filter that retains the same number of rows as the original — or drops to zero — is almost always a sign that something went wrong: a typo in the value, the wrong variable name, or a condition that is never true. There are many ways of checking this number: in the Environment window, opening the data viewer, calling the object in the console, using `nrow()` or other functions. ::: ### Combining conditions Most real filters combine more than one condition. The `&` and `|` operators let you do this — but they behave very differently, and confusing them is one of the most common sources of silent errors in data analysis. **Action** — Filter `imd` to keep only female cases from R6, combining both conditions with `&`. Assign the result to `imd_r6_f` and type below the row count. `r fitb(323)` When using multiple conditions inside `filter()`, the default behavior is treating them as `AND` (`&`) operators. You can either separate the conditions with commas, like arguments; create a single condition using the `&` operator; or pipe two filters together. And the result would be the same: ``` r # Remember, you can achieve the same result with these three codes # Comma-separated conditions imd %>% filter(condition1, condition2) # Conditions combined with the AND operator imd %>% filter(condition1 & condition2) # Two consequtive filters imd %>% filter(condition1) %>% filter(condition2) ``` And it makes sense! With `&`, R evaluated both conditions together. With comma-separated conditions or two consecutive filters, the same two conditions are applied, so the result is the same. First retaining only cases from `R6` and then, among these retained only `Female` cases. That is only valid for `AND` (`&)` statements. If we try the same with other operators, the results change slightly. **Action** — Filter cases that are either from `R6` or `Female` using the `|` (OR) operator, creating the object `imd_r6_or_f` and compare the resulting number of rows with `imd_r6_f`. Can you understand why the difference? ::: {.callout-important collapse="true" appearance="simple"} ### `|` is wider than you think With `|`, a row is retained if **either** condition is true — it does not need to satisfy both. This means your result will include all female cases from every region, *plus* all cases from R6 regardless of sex. In most surveillance contexts, this is not what you want. When in doubt, reach for `&`. ::: ### Selecting from a set with `%in%` Suppose you need to filter for cases from several specific regions — say, R1, R6, and R9. You could write: ``` r imd_regions <- imd %>% filter(region_id == "R1" | region_id == "R6" | region_id == "R9") ``` This works, but the code grows unwieldy fast. The `%in%` operator is the clean solution: it checks whether a value is present anywhere in a set you define. Depending on the complexity and number of options, you can define the values directly using a vector in the filter function, or you could first create an object to serve as intermediary. Therefore, you could write any of the following: ``` r imd %>% filter(variable1 %in% c("value1", "value2", "value3")) # or: values <- c("value1", "value2", "value3") imd %>% filter(variable1 %in% values) ``` To create a vector with multiple values, R uses the `c()` function, where `c` stands for *combine*, i.e. `c("value1", "value2", "value3")`. **Action** — Filter `imd` to keep cases from regions R1, R6, and R9 using `%in%`. Assign the result to `imd_regions`. **Action** — Now add a condition to also keep only female cases aged 18 or older. Assign the final result to `imd_filtered`. How many rows are left? `r fitb(257)` ::: {.callout-tip collapse="true" appearance="simple"} ## Hint You can chain all conditions in a single `filter()` using commas, or write them as separate `filter()` steps in the same pipeline — both work. ::: ### Missing values You have already used `is.na()` to find cases with missing age information. But there is something deeper going on with missing values that is worth understanding — because it will save you from a very common and very silent mistake. The critical thing to know: **you cannot use `==` to find `NA`**. Try it in your console: ``` r NA == NA ``` You might expect `TRUE`. R returns `NA`. The logic is sound: if a value is unknown, you cannot confirm it is equal to another unknown value. Two unknowns are not necessarily the same unknown. So R does not guess — it returns "I don't know". `is.na()` exists precisely to solve this. It does not compare — it simply asks "is this value missing?", and returns `TRUE` or `FALSE`. **Action** — Check how many cases in `imd` have a missing value in the `serogroup` variable? Create the `imd_serogroup_na` object for that and type the number below: `r fitb(0)` Finding missing values is a common task when doing data cleaning, but that is tomorrow's topic. For now, we are only interested in discarding cases with `NA` values in some of their variables, something you will also do quite frequently in the future - believe me. For that, we can introduce the last of today's operators, the `!` - ***opposite or negation*** operator. When placed right before another condition, it tells R to do the exact opposite. Therefore, if we want to keep rows NOT having `NA` values in some column, we would do something like: ``` r imd %>% filter(!is.na(variable1)) ``` **Action** — Create a new object `imd_age_na` that contains only cases with missing `age_years` values. How many cases are there? `r fitb(112)` **Action** — Rather than an object with missing `age_years` cases, we want a dataframe *without* them this time. Call it `imd_age_clean`. Use the same filter with `!` to achieve that. How many rows are left? `r fitb(2515)` **Optional** — How can you use `nrow()` to check that `imd_age_clean` has effectively filtered the adequate number of rows? This one can be trickier, but give it a try! ```{=html}  ``` ------------------------------------------------------------------------ ```{r} #| echo: false opts1 <- c( "It keeps rows where both conditions are true", answer = "It keeps rows where at least one condition is true", "It keeps rows where neither condition is true", "It keeps rows where exactly one condition is true, but not both" ) ``` **What does `filter(age_years > 18 | sex == "Female")` return?** `r longmcq(opts1)` ```{r} #| echo: false opts2 <- c( '`region_id == "R1" | region_id == "R6" | region_id == "R9"`', '`region_id == c("R1", "R6", "R9")`', answer = '`region_id %in% c("R1", "R6", "R9")`', '`region_id & c("R1", "R6", "R9")`' ) ``` **Which of the following correctly filters cases from regions R1, R6, and R9?** `r longmcq(opts2)` ```{r} #| echo: false opts3 <- c( answer = "Because `NA == NA` returns `NA`, not `TRUE` — R cannot confirm two unknown values are equal", "Because `NA` is stored as a number and cannot be compared with `==`", "Because `filter()` automatically removes `NA` before evaluating conditions", "Because `==` only works with character variables, not missing values" ) ``` **Why can't you use `== NA` to find missing values in `filter()`?** `r longmcq(opts3)` ```{r} #| echo: false opts4 <- c( "`filter(is.na(serogroup) == FALSE)`", "`filter(serogroup != NA)`", "`filter(is.na(serogroup) == 0)`", answer = "`filter(!is.na(serogroup))`" ) ``` **Which expression correctly keeps only rows where `serogroup` is not missing?** `r longmcq(opts4)` ------------------------------------------------------------------------ ## Exercise summary Logical operators are the engine behind `filter()` — and behind most of the decisions your code will ever make. You now know how to select rows by equality, by range, by set membership (using `%in%`), and how to handle the special case of missing values. Combined with the pipe, `filter()` gives you precise control over which data you are actually working with at any point in your analysis. But this is only the tip of the Tidy iceberg, so let's continue to the verb of creation: `mutate()`. ::: {.callout-tip collapse="true"} ## 💡 Show solution — only after trying yourself! ``` r # Filter cases from Region 6 imd_region <- imd %>% filter(region_id == "R6") # Filter adults and kids cases imd_adults <- imd %>% filter(age_years >= 18) imd_kids <- imd %>% filter(age_years < 18) # Check the row sums of the filtered vs original data nrow(imd) == nrow(imd_kids) + nrow(imd_adults) # what is happening? # Are there any missing age cases? imd_age_missing <- imd %>% filter(is.na(age_years)) # Check again nrow(imd) == nrow(imd_kids) + nrow(imd_adults) + nrow(imd_age_missing) # now it works! # Filter both Region 6 AND Female cases imd_r6_f <- imd %>% filter(region_id == "R6" & sex == "Female") # order of conditions doesn't matter # Filter Region 6 OR Female cases imd_r6_or_f <- imd %>% filter(region_id == "R6" | sex == "Female") # Filter cases from any of R1, R6 or R9 regions imd_regions <- imd %>% filter(region_id == "R1" | region_id == "R6" | region_id == "R9") # better option using %in% imd_regions <- imd %>% filter(region_id %in% c("R1", "R6", "R9")) # Combine all the different filters together imd_filtered <- imd %>% filter(region_id %in% c("R1", "R6", "R9"), age_years >= 18, sex == "Female") # Is there any missing case in serogroup? imd_serogroup_na <- imd %>% filter(is.na(serogroup)) # Missing age cases imd_age_na <- imd %>% filter(is.na(age_years)) # Dataset without missing age_year cases imd_age_clean <- imd %>% filter(!is.na(age_years)) # We can check the difference in rows is correct nrow(imd_age_clean) == (nrow(imd) - nrow(imd_age_na)) ``` ::: ```{=html} <script> document.addEventListener("DOMContentLoaded", function() { var radiogroups = document.getElementsByClassName("webex-radiogroup"); for (var i = 0; i < radiogroups.length; i++) { radiogroups[i].onchange = radiogroups_func; } }); </script> ```