Fundamentals of Probability Theory: An Introduction to the World of

Probability theory forms the basis of many statistical methods. It helps us deal with uncertainty and random events by quantifying the likelihood that certain events will occur. In this blog post, I explain the basics of probability theory so that you have a solid foundation for delving further into statistics.

What is probability?

Imagine rolling a fair die. The probability of rolling a six is 1/6. But what exactly does that mean? The probability of an event is a number between 0 and 1 that indicates how likely it is that the event will occur. A probability of 0 means that the event will never occur, while a probability of 1 means that it will occur with certainty.

Laplace probability

One of the simplest models in probability theory is the so-called Laplace probability, which is used when all possible outcomes of an experiment are equally likely. For an event $A$ (e.g., rolling a six), the probability $P(A)$ is given by:

$P(A) = \frac{|A|}{|\Omega|}$

Here, $|A|$ is the number of favorable outcomes (e.g., 1 for rolling a six), and $|\Omega|$ is the number of possible outcomes (e.g., 6 for a die).

Example: Simple dice calculation in R

When you roll a die, you may want to calculate the probability of specific events, such as rolling a six or an odd number. Here is an example in R:

# All possible outcomes of a die roll
omega <- 1:6

# Probability of rolling a six
P_six <- 1/length(omega)

# Probability of rolling an odd number
P_odd <- length(omega[omega %% 2 == 1]) / length(omega)

P_six
P_odd

This simple example shows how you can calculate probabilities in a Laplace space.

Empirical probability

In practice, you often do not deal with ideal Laplace probabilities. Instead, you have to estimate probabilities based on observations. This is called the empirical probability. It indicates how often an event has occurred in a large number of trials.

The formula is:

$P(A) \approx \frac{\text{Number of times } A \text{ occurs}}{\text{Total number of trials}}$

Example: Simulating coin tosses in R

Suppose you want to find out how often heads appears in 1,000 tosses of a fair coin. In R, you can simulate this as follows:

# Simulation of 1,000 coin tosses
set.seed(123)  # For reproducibility
tosses <- sample(c("Heads", "Tails"), size = 1000, replace = TRUE)

# Frequency of heads
heads_frequency <- sum(tosses == "Heads") / length(tosses)
heads_frequency

This example shows you how to calculate probabilities based on repeated experiments.

Conditional probability

We are often interested not only in the probability of an event, but also in how it changes when we already know that another event has occurred. This is called conditional probability. The probability that event $A$ occurs given that $B$ has already occurred is calculated as follows:

$P(A|B) = \frac{P(A \cap B)}{P(B)}$

A classic example is the probability that someone is ill given a positive test result.

Example: Calculating conditional probabilities in R

Suppose you have the following probabilities: The probability that someone is ill is 0.01, and the probability that a test is positive when the person is ill is 0.99. The probability that the test is also positive for healthy people is 0.05. You want to know how likely it is that a person is actually ill when the test result is positive.

# Probabilities
P_krank <- 0.01
P_test_pos_krank <- 0.99
P_test_pos_gesund <- 0.05
P_gesund <- 1 - P_krank

# Overall probability of a positive test
P_test_pos <- P_test_pos_krank * P_krank + P_test_pos_gesund * P_gesund

# Conditional probability (Bayes' theorem)
P_krank_test_pos <- (P_test_pos_krank * P_krank) / P_test_pos
P_krank_test_pos

Independence of events

Two events $A$ and $B$ are independent if the occurrence of $A$ has no influence on the probability of $B$, and vice versa. Mathematically, this means:

$P(A \cap B) = P(A) \cdot P(B)$

An example of independent events would be rolling two dice. The result of one die does not influence the result of the other.

Bayes’ theorem

Bayes’ theorem is one of the most important theories in probability theory and is used to update probabilities when new information becomes available. It states:

$P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}$

This theorem is often used in fields such as medicine and data science to make decisions under uncertainty.

Conclusion

Probability theory is at the heart of many statistical procedures. From calculating simple probabilities and conditional probabilities to applying Bayes’ theorem, it provides us with tools for dealing with uncertainty. With the examples and R code presented here, you can try out and apply the basic concepts yourself.