Tables of things you cannot count

Session 4 practical exercises

Before you start

Keep working in the same script. Your data is already imported and your factors are already in place, so there is nothing to set up, which is exactly the point of everything you did in the first two exercises.

An email arrives just before lunch:

“The tables are great, thank you. One last thing before the meeting: the group wants to know whether this is hitting the elderly harder. Cases per age group, how many of them died, and the fatality ratio. Oh, and they always ask about Serogroup C, include it.

Thanks
K”

Read the request again and see the indicators she is asking for. Four, maybe five, all broken down by age group. Every tool you have used so far gives you one number per cell. Today you learn to put several side by side.

Part 6 · Counting things that are not rows

count() and tabyl() count rows. That is their whole job, and it is why they cannot help you here: Kassandra is not asking how many rows are in each age group, she is asking how many of those rows meet a condition — died, or belonged to serogroup C.

To count conditions you need a different mechanism, which is simpler than you expect - and you even know already. But now I want to show you how it actually works: start by copying this code and looking at the output

Action — Run these two lines and look carefully at the output of the first one.

imd$death == "Yes"

sum(imd$death == "Yes", na.rm = TRUE)

The first line does not return deaths. It returns one TRUE or FALSE per case, in the same order as your rows. It is a mask laid over the dataset: this one yes, this one no, this one yes. Remember in session 2 when we insisted so much about the importance of logical operators in R.

The second line then exploits something R does quietly: when a logical vector meets an arithmetic function, TRUE becomes 1 and FALSE becomes 0. So adding up the mask adds up the TRUEs — and the number of TRUEs is the number of deaths.

Why na.rm = TRUE is not optional here

If death is missing for a single case, the comparison NA == "Yes" does not return FALSE. It returns NA, because R refuses to guess. And a single NA inside sum() turns the entire result into NA.

Action — See it happen: copy this code and execute it: sum(c(TRUE, FALSE, NA))

One unknown value, and your count of deaths disappears. na.rm = TRUE tells sum() to leave the unknowns out and add up the rest. Get into the habit of writing it every time — it costs you twelve characters and saves you an afternoon.

Data Cleaning - NAs

When doing your data cleaning in the future, always remember to double check not only whether NAs are present in the data or not - but what does those NAs values mean for each variable. Is it only missing information? Should it be there and can be accepted? Shall you replace for another value?

In the case of death the question in meaningful: if the value is missing, should we assume the case is alive? A report with the wrong number of dead cases is a serious flaw, so always check your assumption and how that affects the output. Here, we will assume NA == "No"

One row, many numbers

You already know how to collapse a dataset into a summary: group_by() and summarise(). What you have not done yet is ask summarise() for more than one thing at a time. And you can. Each argument you write inside summarise() becomes a new column, and they are all calculated on the same group, in the order you write them.

Action — Group by age_group and build a summary with the number of cases and the number of deaths. Call it imd_indicators.

Now the ratio. The case fatality ratio is deaths divided by cases, expressed as a percentage — and both of those numbers are already sitting in your summary.

Action — Add a third column, cfr, calculated from the two you just created. Round it to one decimal.

Hint - round() function

The function round(x, 1) will take a vector of values as first argument x and round it to the given number of decimals from the second argument. At this point of the course you know perfectly that the place of x can also be used to call another function, or write down an operation between existing variables, like a ratio between counts and a multiplication…

Action — Finish the table Kassandra asked for: add n_serogC with the number of serogroup C cases, and perc_C with the percentage they represent within each age group, rounded to one decimal.

Trust, then verify

Your table now contains five numbers per age group, none of which you calculated by hand, all of which you are about to send to a meeting. Before you do that, check one of them.

Action — Pick the oldest age group. Using filter() and nrow(), count how many cases it has, and how many of those died. Compare both numbers with the row in imd_indicators. Then divide one by the other and compare with your cfr.

Hint

filter() accepts several conditions separated by commas, and applies all of them at once.

The habit worth keeping

This takes ninety seconds and you will not always do it. But do it the first time you write any new summary, and do it again whenever a number surprises you.

Code that runs without errors is not the same as code that answers the question you asked. summarise() will happily calculate the wrong thing with perfect syntax — a misplaced grouping variable, a condition matching a label that does not exist, a denominator that is not what you thought. None of those throw an error. The only thing standing between you and a wrong number in a meeting document is a check you can do with two functions you learned in Session 2.

Exercise Summary

This exercise added one function to your vocabulary and one idea to your head. The function was round() to reformat percentages to less decimals, as R by default generates too many of them. The idea is bigger: from now on, a summary table is something you design, deciding column by column what each number should be, instead of something a function hands to you.

That freedom is also the risk, which is why the verification step matters more than any of the code around it.

💡 Show solution — only after trying yourself!

# The logic behind conditional counting
imd$death == "Yes"

sum(imd$death == "Yes", na.rm = TRUE)

sum(c(TRUE, FALSE, NA))

# Multi-indicator summary table
imd_indicators <- imd %>% 
  group_by(age_group) %>% 
  summarise(
    n_cases   = n(),
    n_deaths  = sum(death == "Yes", na.rm = TRUE),
    cfr       = round(n_deaths / n_cases * 100, 1),
    n_serogC  = sum(serogroup == "C", na.rm = TRUE),
    perc_C    = round(n_serogC / n_cases * 100, 1)
  )

# Manual verification of one group
imd %>% 
  filter(age_group == "65+") %>% 
  nrow()

imd %>% 
  filter(age_group == "65+", death == "Yes") %>% 
  nrow()

# Rename, format and export
imd_table_3 <- imd_indicators %>% 
  rename("Age group"        = age_group,
         "Cases (n)"        = n_cases,
         "Deaths (n)"       = n_deaths,
         "CFR (%)"          = cfr,
         "Serogroup C (n)"  = n_serogC,
         "Serogroup C (%)"  = perc_C) %>% 
  flextable() %>% 
  autofit()

save_as_docx(imd_table_3, path = here("outputs", "Table_3.docx"))

--- title: "Tables of things you cannot count" subtitle: "Session 4 practical exercises" --- ```{r} #| include: false library(webexercises) library(pacman) p_load(rio, here, tidyverse, janitor, flextable) imd <- import(here("data", "clean", "IMD_Sample_Clean.rds")) ``` ## Before you start Keep working in the same script. Your data is already imported and your factors are already in place, so there is nothing to set up, which is exactly the point of everything you did in the first two exercises. An email arrives just before lunch: *"The tables are great, thank you. One last thing before the meeting: the group wants to know whether this is hitting the elderly harder. Cases per age group, how many of them died, and the fatality ratio. Oh, and they always ask about Serogroup C, include it.* *Thanks\ K"* Read the request again and see the indicators she is asking for. Four, maybe five, all broken down by age group. Every tool you have used so far gives you **one** number per cell. Today you learn to put several side by side. ## Part 6 · Counting things that are not rows `count()` and `tabyl()` count **rows**. That is their whole job, and it is why they cannot help you here: Kassandra is not asking how many rows are in each age group, she is asking how many of those rows meet a *condition* — died, or belonged to serogroup C. To count conditions you need a different mechanism, which is simpler than you expect - and you even know already. But now I want to show you how it actually works: start by copying this code and looking at the output **Action** — Run these two lines and look carefully at the output of the first one. ``` r imd$death == "Yes" sum(imd$death == "Yes", na.rm = TRUE) ``` The first line does not return deaths. It returns one `TRUE` or `FALSE` **per case**, in the same order as your rows. It is a mask laid over the dataset: *this one yes, this one no, this one yes*. Remember in session 2 when we insisted so much about the importance of logical operators in R. The second line then exploits something R does quietly: when a logical vector meets an arithmetic function, `TRUE` becomes 1 and `FALSE` becomes 0. So adding up the mask adds up the `TRUE`s — and the number of `TRUE`s is the number of deaths. ::: callout-caution ## Why `na.rm = TRUE` is not optional here If `death` is missing for a single case, the comparison `NA == "Yes"` does not return `FALSE`. It returns `NA`, because R refuses to guess. And a single `NA` inside `sum()` turns the **entire result** into `NA`. **Action** — See it happen: copy this code and execute it: `sum(c(TRUE, FALSE, NA))` One unknown value, and your count of deaths disappears. `na.rm = TRUE` tells `sum()` to leave the unknowns out and add up the rest. Get into the habit of writing it every time — it costs you twelve characters and saves you an afternoon. ::: ::: callout-important ## Data Cleaning - NAs When doing your data cleaning in the future, always remember to double check not only whether NAs are present in the data or not - but what does those NAs values mean for each variable. Is it only missing information? Should it be there and can be accepted? Shall you replace for another value? In the case of `death` the question in meaningful: if the value is missing, should we assume the case is alive? A report with the wrong number of dead cases is a serious flaw, so always check your assumption and how that affects the output. Here, we will assume `NA == "No"` ::: ### One row, many numbers You already know how to collapse a dataset into a summary: `group_by()` and `summarise()`. What you have not done yet is ask `summarise()` for **more than one thing at a time**. And you can. Each argument you write inside `summarise()` becomes a new column, and they are all calculated on the same group, in the order you write them. **Action** — Group by `age_group` and build a summary with the number of cases and the number of deaths. Call it `imd_indicators`. Now the ratio. The case fatality ratio is deaths divided by cases, expressed as a percentage — and both of those numbers are already sitting in your summary. **Action** — Add a third column, `cfr`, calculated from the two you just created. Round it to one decimal. ::: callout-tip ## Hint - `round()` function The function `round(x, 1)` will take a vector of values as first argument `x` and round it to the given number of decimals from the second argument. At this point of the course you know perfectly that the place of `x` can also be used to call another function, or write down an operation between existing variables, like a ratio between counts and a multiplication... ::: **Action** — Finish the table Kassandra asked for: add `n_serogC` with the number of serogroup C cases, and `perc_C` with the percentage they represent within each age group, rounded to one decimal. ### Trust, then verify Your table now contains five numbers per age group, none of which you calculated by hand, all of which you are about to send to a meeting. Before you do that, check one of them. **Action** — Pick the oldest age group. Using `filter()` and `nrow()`, count how many cases it has, and how many of those died. Compare both numbers with the row in `imd_indicators`. Then divide one by the other and compare with your `cfr`. ::: {.callout-tip collapse="true" appearance="simple" icon="false"} ## Hint `filter()` accepts several conditions separated by commas, and applies all of them at once. ::: ::: callout-important ## The habit worth keeping This takes ninety seconds and you will not always do it. But do it **the first time you write any new summary**, and do it again whenever a number surprises you. Code that runs without errors is not the same as code that answers the question you asked. `summarise()` will happily calculate the wrong thing with perfect syntax — a misplaced grouping variable, a condition matching a label that does not exist, a denominator that is not what you thought. None of those throw an error. The only thing standing between you and a wrong number in a meeting document is a check you can do with two functions you learned in Session 2. ::: ### Nicely formatting the export table In the last exercise you turned a table sideways with `pivot_wider()` before presenting it. Look at `imd_indicators` and decide whether you need to do that again. ::: callout-note ## When *not* to pivot You do not. And understanding why is more useful than the pivot itself. `pivot_wider()` spreads **the categories of one variable** across columns: one column per year, one column per sex. In the last exercise your long table had a `year` column whose values became column headers. `imd_indicators` has nothing like that. Its columns are not categories of anything — they are five **different measures**: a count, another count, a percentage, another count, a ratio. There is no variable to spread, because each column already answers a different question. The table came out of `summarise()` already in the shape a reader needs: one row per group, one column per number. Reach for `pivot_wider()` when a variable's values should become headers, not out of habit. ::: So there is only presentation left. The column names, though, are yours: `n_cases`, `perc_C`, `cfr`. They are short and convenient because you have been typing them, but nobody at that meeting has seen your script. **Action** — Rename the columns to something a reader understands, then turn the result into a flextable and adjust the widths. ::: {.callout-tip collapse="true" appearance="simple" icon="false"} ## Hint `rename()`, from Session 3, takes `new_name = old_name`. If the new name contains spaces, wrap it in quotes: `rename("Age group" = age_group)`. ::: Notice where the renaming goes: **at the very end**, immediately before `flextable()`. The same rule you met with the `adorn_*` functions applies here. `Age group` and `Cases (n)` are readable, but they are also awkward to type and need quotes or backticks in every function that touches them afterwards. Keep the convenient names while you are working, and switch to the readable ones at the last possible moment. **Action** — Export the table to a Word document in your `outputs` folder, using `save_as_docx()` and `here()` as you did in the previous exercise. *Attach it to the reply and go for lunch.* ------------------------------------------------------------------------ ```{r} #| echo: false opts_1 <- c( "Because `sum()` counts how many values in a variable are not missing", answer = "Because the comparison returns `TRUE`/`FALSE`, and `sum()` treats `TRUE` as 1 and `FALSE` as 0", "Because `sum()` recognises text values and counts the ones that match", "Because `==` already counts the matching cases and `sum()` only displays the result" ) ``` **Why does `sum(death == "Yes")` return the number of deaths?** `r longmcq(opts_1)` ```{r} #| echo: false opts_2 <- c( "The missing cases would be counted as deaths", "The missing cases would be counted as survivors", answer = "The whole result would be `NA`, because a single `NA` propagates through `sum()`", "Nothing would change — `sum()` ignores missing values by default" ) ``` **What would happen to `sum(death == "Yes")` if `death` had missing values and you did not write `na.rm = TRUE`?** `r longmcq(opts_2)` ```{r} #| echo: false opts_3 <- c( "Because the table has too few rows for pivoting to make a difference", "Because `pivot_wider()` does not work on the output of `summarise()`", answer = "Because its columns are different measures, not the categories of one variable", "Because the table was already pivoted when `group_by()` collapsed the rows" ) ``` **Why does `imd_indicators` not need `pivot_wider()`?** `r longmcq(opts_3)` ## Exercise Summary This exercise added one function to your vocabulary and one idea to your head. The function was `round()` to reformat percentages to less decimals, as R by default generates too many of them. The idea is bigger: from now on, a summary table is something you **design**, deciding column by column what each number should be, instead of something a function hands to you. That freedom is also the risk, which is why the verification step matters more than any of the code around it. ::: {.callout-tip collapse="true"} ## 💡 Show solution — only after trying yourself! ``` r # The logic behind conditional counting imd$death == "Yes" sum(imd$death == "Yes", na.rm = TRUE) sum(c(TRUE, FALSE, NA)) # Multi-indicator summary table imd_indicators <- imd %>% group_by(age_group) %>% summarise( n_cases = n(), n_deaths = sum(death == "Yes", na.rm = TRUE), cfr = round(n_deaths / n_cases * 100, 1), n_serogC = sum(serogroup == "C", na.rm = TRUE), perc_C = round(n_serogC / n_cases * 100, 1) ) # Manual verification of one group imd %>% filter(age_group == "65+") %>% nrow() imd %>% filter(age_group == "65+", death == "Yes") %>% nrow() # Rename, format and export imd_table_3 <- imd_indicators %>% rename("Age group" = age_group, "Cases (n)" = n_cases, "Deaths (n)" = n_deaths, "CFR (%)" = cfr, "Serogroup C (n)" = n_serogC, "Serogroup C (%)" = perc_C) %>% flextable() %>% autofit() save_as_docx(imd_table_3, path = here("outputs", "Table_3.docx")) ``` ::: ```{=html} <script> document.addEventListener("DOMContentLoaded", function() { var radiogroups = document.getElementsByClassName("webex-radiogroup"); for (var i = 0; i < radiogroups.length; i++) { radiogroups[i].onchange = radiogroups_func; } }); </script> ```