Sample Size & Simulation – by Roopam

A few years ago, I did a day-long workshop on Statistical Thinking for a large German shipping & cargo company in Mumbai. During the Q&A session, the Vice President of operations asked a tricky question: what is a good sample size to achieve precision and accuracy for your analysis? He was looking for a one-size-fits-all answer and I wish it were that simple.

On the surface, this might seem to be an extremely fundamental and basic question but in practice, so many professionals struggle to answer this. In this article, I will try to answer this question.

Sample Size

Like ingredients are at the core of taste of the food, sample size and data quality are at the core of the analysis. With bad ingredients, even a master chef can’t produce a good dish. Hence, representative sample and sample size are the two most important aspects on which every data scientist needs to keep a cautious eye.

One of the most intuitive ways to identify the right sample size is through data simulation. In this article, we will simulate coin tosses to understand the significance of sample size. As you know tossing a coin is one of the easiest and fastest experiments with random binary outcomes.

Before you brush off coin tosses as too elementary and textbook-ish please check out the following practical events with random binary outcomes similar to coin tosses.

• Creating triggers for fraudulent credit card transactions
• Identifying customers for cross/up sell opportunities
• Detecting malignant cancer cells
• Measuring credit risk of a borrower
• Autoflagging spam emails
• Predicting employees’ likelihood for attrition

Data Simulation and Sample Size

In this section, we will toss a fair coin many times to estimate the probability of heads. This coin is fair because it has the equal probability of both heads and tails i.e. 0.5. We will use different sample sizes to observe at what point we get a precise value for the probability of heads. We could either toss the coin ourselves or make our computer run a simulation of coin tosses. I will choose the latter. Let’s say we tossed the coin 5 times and got the following results

Heads, Tails, Heads, Heads, Tails

With just the first result (Heads), i.e. sample size = 1, we get the sample probability of heads equal to 1. With the sample size of 2 (Heads, Tails) the sample probability of heads is 1/2 or 0.5. With the sample size of 5, we will get the sample probability of heads = 3/5.

Now to extend the above experiment, I will perform 10 experiments with 500 tosses each and estimate probability at each step. I will simulate these experiments for the obvious reason that I don’t want to toss a coin 5000 times (i.e. 10 × 500 times). The following are the results of these simulations (R code for these simulations).

In this graph, the orange line is the actual probability i.e. 0.5, and the blue line is the estimated probability for each step. As expected the estimated probability is getting closer to actual probability with the increase in sample size (left to right).

Now the question is how good is 500 as the sample size to estimate the probability of heads? For this, we will simulate 1000 experiments each with the sample size of 500, and estimate the probability of heads after each experiment. This histogram shows the estimated probability of 1000 experiments (R code for this simulation)

The histogram looks like a normal distribution or bell curve as would have expected from the central limit theorem (CLT) – we will come back to CLT in the next segment.

The summary statistics for this data:

Both mean and median for 1000 experiments (sample size = 500) is close to 0.5. The standard deviation of these experiments is 0.0226. This standard deviation for multiple experiments is also known as the standard error of the estimate as we will learn later in this article.

We could have estimated the same values for mean and standard error through the theory suggested by the central limit theorem.

Central Limit Theorem & Sample Size

Let’s try to simplify the central limit theorem for our purpose of coin tossing. It states that mean of all the probabilities of heads estimated through all the experiments is equal to the actual probability of heads. In mathematical form, this is stated as:

We know that the actual probability of heads or π is 50%. Unlike this problem, for most practical problems the value of π is unknown. In such cases, we assume the probability obtained from experiment (p) as the expected probability.

For our one thousand experiments, we got the mean as 0.5006. Had we performed an infinite number of experiments we will have got an exact value for actual probability i.e. 0.5. But none of us has the time to do an infinite number of experiments. Only mathematicians can do these experiments in their heads but for all practical purposes, most of us can rely on simulations.

Another part of the central limit theorem states that standard error for estimation of proportions or probabilities is:

Here, π is the actual probability of heads for the fair coin which is 0.5. Hence, the standard error for a fair coin is:

In our simulated experiments, we got the standard error or standard deviation of estimated probabilities of heads:

The simulated results gave almost the same result as the theory suggested.

Identify the Right Sample Size

Now, let us come back to our original question for this article of identification of sample size. In the previous section, we have found the theoretical definition for standard error as:

We can reorganize the above equation to measure the value of n which is sample size.

Now, if you are OK with standard error as 10% of the actual probability of head (i.e. 10% of 0.5 i.e. 0.05). In this case, 95% of your results will be in the confidence interval of 0.4 to 0.6. This is calculated through 2 × standard error on either side of 0.5. A sample size of 100 will be good enough to achieve this confidence as shown in the calculation

On the other hand, if you want standard error as 0.5% of the actual probability of head (i.e. 0.005). In this case, 95% of your results will be in the confidence interval range of 0.495 to 0.505. However, to achieve this high level of precision the requirement of sample size is whopping 40000 tosses.

Hence, the entire sampling strategy is a trade-off between sample size and standard error. This table shows sample size requirement for the desired standard error as a percentage of probability of heads for a fair coin.

When we plot a sample size vs standard error plot, because of the inverse square relationship between them, toward the right side of the plot standard error doesn’t decrease as drastically as towards the left. This is an important concept to keep in mind while choosing the sample size for your analysis.

Sign-off Note

We have done the entire calculation of sample size based on the assumption that means is the measure that you want to estimate for a population. This is usually a good assumption for most analysis. However, it is always good for you to ask : is mean the right measure for your analysis? The right measure for your analysis depends on the question you are trying to answer.

For example, consider these two questions:

1. What is the salinity of the Pacific Ocean?
2. Is there another planet with intelligent life in the Universe?

In terms of population size, the number of drops in the ocean and planets in the Universe is similar. A couple of drops of water are enough to answer the first question since the salinity of oceans is fairly constant. On the other hand, the second question is a black swan problem. You may need to visit every single planet to rule our possibility of an intelligent form of life.  Hence the sample size depends on the degree of similarity or homogeneity of the population in question.