Data Management in R

Introduction

In this article, I would like to guide you through data management in R. If you have worked with R before, you have probably noticed how powerful this tool is—and sometimes a little intimidating, too. But don’t worry, we’ll take it step by step: with practical examples and a touch of humor.

We’ll talk about reading in data, preparing and transforming it, handling missing values, filtering and coding, as well as storing and exporting data in R. That may sound like a lot, but believe me: once you get the hang of it, you’ll wonder how you ever managed without these options.

R is open source, meaning it is free, and offers a wide range of packages (libraries) that make your life easier. The best part: you can cover your entire data workflow from A to Z in R—from importing and cleaning your data all the way to your final analyses and visualizations.

Whether it’s CSV files, Excel sheets, databases, or even web APIs – R helps you bring everything together. And because you can document everything in scripts or R Markdown documents, your approach is reproducible. That’s not only practical for you (keyword: “I did something similar last year…”), but also for collaborating with others.

Basics & Setup

Important Packages

Before we really get started, I’d like to briefly introduce the packages I use most often for data management. There are countless packages in R, but dplyr and readr (both part of tidyverse) are true all-rounders. If you work with Excel files, readxl is often helpful. To tidy up and transform data, you really can’t do without tidyr either.

An R package (short for package) is a collection of functions, datasets, and documentation developed for a specific purpose. When you install a package (install.packages("paketname")) and load it (library(paketname)), its functions become available to you in the R environment.

RStudio Projects & Directories

I personally recommend setting up your own RStudio project. This is great because it gives you a clear, separate workspace where all your code, data, and results are stored. That way, you’ll be well organized from the start.

• Create a project: In RStudio, click the button in the top right (or go to “File -> New Project…”).
• Directory structure: I often use folders such as data/ for raw data, R/ for scripts, output/ for charts and tables, and docs/ for reports.

When you read in your data, avoid absolute paths (e.g., C:\Users\You…) – instead, work with relative paths. This makes your project portable.

Reading in data

Now to the main point: How do we get our data into R? Here are a few common approaches.

Reading in CSV/TXT files

The most common source is probably CSV files (Comma-Separated Values), or TSV (Tab-Separated Values). Thanks to the readr package (part of the tidyverse), this can be done in no time:

# Ganz klassisch:
meine_daten <- read.csv("data/mein_datensatz.csv")

# Mit readr-Paket:
library(readr)
meine_daten <- read_csv("data/mein_datensatz.csv")

The advantage of read_csv() is that it can be a little faster and also offers some useful options, such as using col_types to specify exactly which data type is expected in each column.

Tip: Adjust the separator. Sometimes a semicolon ; is used instead of a comma. So, if necessary, look out for parameters such as sep = ";" (already set as the default in the base function read.csv2()).

Reading in Excel files

If you have Excel files, you can use the readxl package. It is also very easy to use:

library(readxl)
excel_daten <- read_excel("data/meine_exceldatei.xlsx",
                          sheet = "Tabelle1")

This reads in the desired “sheet” and returns a data frame. So you do not first have to manually convert the file from Excel to CSV, which saves time and reduces the risk of errors.

Other formats

• JSON: You can read in JSON files using packages such as jsonlite.
• SQL databases: With DBI and RSQLite (or other drivers), you can access databases directly.
• Web APIs: The httr package helps you query REST APIs (e.g. JSON responses).

In short: Whatever format you have, there is probably a suitable package in R.

Data preparation and transformation

Basic operations with dplyr

dplyr is a package that wrangles your data—in other words, structures and prepares it. The five most important “helpers” in dplyr are:

1. filter() – Select rows based on a condition
2. select() – Select or rename columns
3. mutate() – Create new columns or recode variables
4. arrange() – Sort data sets
5. summarise() + group_by() – Summarise grouped data

Data wrangling refers to all the steps needed to transform “raw” data into a format suitable for analysis or visualisation. This includes, for example, cleaning up columns, renaming variables, merging individual tables, and so on.

Examples of transformation

Let’s assume you have a data set called meine_daten with the following columns:

• id (simply a sequential number)
• geschlecht (values: “m”, “f”)
• alter (age in years)
• einkommen (in euros)

Now let’s take a look at dplyr functions:

library(dplyr)

# 1) Filter: Nur Personen über 30
df_ueber30 <- meine_daten %>%
  filter(alter > 30)

# 2) Select: Nur Spalten id, alter und einkommen
df_einkommen <- meine_daten %>%
  select(id, alter, einkommen)

# 3) Mutate: Neue Variable 'alter_gruppe' anhand von alter
df_mutiert <- meine_daten %>%
  mutate(alter_gruppe = if_else(alter < 18, "Kind/Jugend",
                                if_else(alter < 65, "Erwachsen", 
                                        "Senior")))

Here you can see how to turn a continuous variable (age) into categorical groups. This is “coding”: In other words, we create a category from a number.

Filtering and sorting

After filtering, you may want to sort the data:

df_sortiert <- meine_daten %>%
  filter(!is.na(einkommen)) %>%    # Zeilen mit NA in einkommen ausschließen
  arrange(desc(einkommen))         # absteigend nach Einkommen sortieren

desc(einkommen) sorts in descending order. With arrange(einkommen), it would be in ascending order.

Recoding variables

In R, you can recode variables very flexibly. Suppose gender is coded as “m” or “w” – you want to turn these into factors, e.g. “Male” and “Female”:

df_kodiert <- meine_daten %>%
  mutate(geschlecht_factor = factor(geschlecht,
                                    levels = c("m", "w"),
                                    labels = c("Männlich", "Weiblich")))

A factor in R is a special data structure for storing categories. Unlike character variables, each category has its own level. This is helpful for statistical analyses (e.g. linear models with categorical predictors).

Recoding can also mean adding new levels, combining values, or dividing numeric variables into classes.

Handling missing values

What are missing values (NA)?

In R, missing values are marked as NA (Not Available). So, if you have a dataset in which certain entries are missing in one column, R will usually mark them as NA.

NA stands for missing information in R. It means that the actual value is unknown or undefined.

Identifying NA

# Gibt TRUE aus, wenn Wert fehlt
is.na(meine_daten$einkommen)

# Anzahl fehlender Werte in Spalte einkommen:
sum(is.na(meine_daten$einkommen))

Handling NA in calculations

For example, if you calculate the mean:

mean(meine_daten$einkommen)
# [1] NA

R returns NA because it does not know what to do with the missing values. You can use parameters such as na.rm = TRUE:

mean(meine_daten$einkommen, na.rm = TRUE)

This causes missing values to be ignored in the calculation. However, be sure to consider why they are missing and whether ignoring them makes sense.

Strategies for missing values

• Simple removal (na.omit() or drop_na()): You remove rows that contain NAs. Be careful—you could lose important data.
• Imputation: Missing values are estimated, e.g. using the mean, median, or more complex methods (multiple imputation). This is a more advanced method that we will not cover in greater detail in this introductory course.
• Specific handling: Sometimes, a separate code (e.g. 9999) is entered in datasets to represent missing values. In that case, you need to actively convert it to NA in R.

Data export and storage

Once you have finished wrangling your data, you may want to save it in another format or send your final dataset to colleagues.

Export as CSV

write.csv(df_kodiert, "output/ergebnis.csv", row.names = FALSE)

Using row.names = FALSE prevents R from creating an additional column for row names.

Export as Excel

There are several ways to do this. With the writexl package (lightweight and without a Java dependency), you can save a data frame as follows:

library(writexl)
write_xlsx(df_kodiert, "output/ergebnis.xlsx")

Alternatively, the openxlsx or xlsx packages offer similar functions. In those packages, too, you can specify which sheet you want to store the data in.

R-specific formats: RDS and RData

• RDS stores a single object in a file.

saveRDS(df_kodiert, file = "output/ergebnis.rds")
# Laden:
mein_obj <- readRDS("output/ergebnis.rds")

• RData stores multiple objects in a file.

save(df_kodiert, df_sortiert, file = "output/meine_objekte.RData")
# Laden:
load("output/meine_objekte.RData")

These formats are practical because they preserve data types and attributes (e.g. factors) directly. The CSV format handles these more rudimentarily.

Best Practices & Tips

Project structure & Documentation

• Folder structure: Keep your data, scripts, functions, and outputs separate.
• Naming: Avoid spaces and special characters in file names. Better: my_data_2023-01-01.csv.
• Comments: Comment on what you are doing, but avoid redundant statements such as “This is an average…”. Short, focused explanations are very helpful when you look at the code again six months from now.

First data quality checks

• summary() gives you a rough overview, including minima, maxima, means, and so on.
• str() tells you which data types you have and how many rows and columns there are.
• head() and tail() to display an excerpt of the data.
• Plots, e.g. histograms or boxplots, are also extremely valuable for detecting outliers.

An example of code for a quick plot:

plot(meine_daten$alter, meine_daten$einkommen,
     xlab = "Alter",
     ylab = "Einkommen (Euro)",
     main = "Streudiagramm Alter vs. Einkommen")

Or, more modern with ggplot2:

library(ggplot2)

ggplot(meine_daten, aes(x = alter, y = einkommen)) +
  geom_point() +
  labs(title = "Alter vs. Einkommen",
       x = "Alter",
       y = "Einkommen (Euro)")

Questions & Answers (Q&A)

Below, I will give you a short question-and-answer game so that you can check whether you have understood the key points. I will write the question and provide a sample answer that you can use as a guide.

1. Question: How do I read a CSV file called daten.csv into R if it is located in the raw_data subfolder?
   Answer (example): You can first make sure that your working directory is correct (or that you are in an RStudio project). Then, for example, you can use meine_daten <- read.csv("raw_data/daten.csv", sep = ",", header = TRUE) Or with the readr package:library(readr) meine_daten <- read_csv("raw_data/daten.csv") It is important to specify the correct separator (sep) if necessary, and to use header = TRUE if you have column names in the first row.
2. Question: What are NA values, and how can I exclude them when calculating a mean?
   Answer (example): In R, NA stands for missing values. If, for example, you have missing values in my_data$income, you need to specify this when calculating the mean:mean(my_data$income, na.rm = TRUE) This way, you ignore all missing values and get a mean based on the available data.
3. Question: How can I split a numeric variable into categories (e.g., “younger than 18,” “18–64,” “65 and older”)?
   Answer (example): There are several ways to do this in R. One classic approach is to use if_else in dplyr:library(dplyr) new_data <- old_data %>% mutate(age_group = case_when( age < 18 ~ "Child/Youth", age < 65 ~ "Adult", TRUE ~ "Senior" )) This creates a new column called age_group based on predefined conditions.
4. Question: How can I export my processed data as an Excel file?
   Answer (example): First, I load the writexl package:install.packages("writexl") # if not already installed library(writexl) write_xlsx(my_data, "output/my_data.xlsx") This creates an Excel file in the output folder containing the data frame.
5. Question: What is the advantage of saveRDS() compared with write.csv()?
   Answer (example): saveRDS() saves the object in an R-specific binary format and preserves all information about data types (e.g., factors). You can load it with readRDS(). With write.csv(), you partially lose metadata (e.g., factor-level information). However, CSVs are naturally more universal if you want to share data with people who do not use R.
6. Question: What should I do if I have huge datasets that fit in memory but everything is running somewhat slowly?
   Answer (example): I could try the data.table package, which is highly efficient for filtering and transformation operations. I could also convert my datasets to a more performant format such as Parquet (using the arrow package) or, if the dataset is really large, use a database connection via DBI and RSQLite.
7. Question: What are the concrete benefits of an RStudio project?
   Answer (example): A project helps you keep your files organized in one place, reduces the hassle of dealing with the ‘Working Directory’ (everything is relative to the project folder), and makes it easier to integrate version control (Git). It is essentially your own small container for code, data, and results.
8. Question: When should I remove missing values, and when is it better to replace them?
   Answer (example): It depends on the context. If data are genuinely missing at random and you have enough observations, deleting them may be fine. However, if there are systematic reasons for the missingness (e.g., certain groups), this could lead to bias. In that case, it is worth considering imputation methods, such as mean, median, or more complex multiple imputation.

Conclusion

Data management in R may seem overwhelming at first, but if you proceed step by step, you will find that a few functions from readr, dplyr, readxl (and perhaps writexl) are enough to handle 90% of your everyday tasks.

The key to success is to keep trying out new datasets from time to time and build yourself a kind of “library” of code snippets (code examples). At some point, you will be able to copy and paste them together almost blindly and have your data organized in no time.

With that in mind: Have fun tinkering and experimenting! And if you ever get stuck, there are always numerous online communities (Stack Overflow, RStudio Community) where you can get help.

---

Q&A (Questions from learners and sample solutions)

Q: “I have a table in which some rows appear twice. How can I remove duplicates or at least identify them?”
A (sample solution):

You can use distinct() from the dplyr package, which identifies and removes duplicates based on all or selected columns. For example:

library(dplyr)
df_eindeutig <- distinct(dein_dataframe)

This removes duplicate rows. If you want to do this based on just one column, specify it as a parameter.

---

Q: “Is it better to delete or replace NA values?”
A (sample solution):

It depends on your use case. If you have only a few NAs and they are truly randomly distributed, you can often simply omit them. However, if there are many of them and they follow a pattern, omitting them can lead to systematic biases. In that case, you need to consider imputation methods.

---

Q: “What help can I get if, for example, I don’t understand read_csv?”
A (sample solution):

In R or RStudio, you can simply enter ?read_csv or help("read_csv") to read the help text. In addition, many packages provide vignette(), e.g. vignette("readr"), which gives you a detailed explanation.