Information about Law Of Large Numbers

| The law of large numbers (LLN) is a theorem in probability that describes the long-term stability of a random variable. Given a sample of independent and identically distributed random variables with a finite population mean and variance, the average of these observations will eventually approach and stay close to the population mean.

The LLN can easily be illustrated using the rolls of a die. That is, outcomes of a multinomial distribution in which the numbers 1, 2, 3, 4, 5, and 6 are equally likely to be chosen. The population mean (or "expected value") of the outcomes is:

(1 + 2 + 3 + 4 + 5 + 6) / 6 = 3.5.

The following graph plots the results of an experiment of rolls of a die. In this experiment we see that the average of die rolls deviates wildly at first. As predicted by LLN the average stabilizes around the expected value of 3.5 as the number of observations become large.



The law of large numbers works equally well for proportions. Given repeated flips of a fair coin, the frequency of heads (or tails) will approach 50% over a large number of trials. However, note that the absolute difference in the number of heads and tails won't necessarily get smaller. For example, we may see 520 heads after 1000 flips and 5096 heads after 10000 flips. While the difference has increased from 20 to 96, the average has moved from .52 to .5096, closer to the true 50%.

The LLN is important because it "guarantees" stable long-term results for random events. For example, while a casino may lose money in a single spin of the American Roulette wheel, it will almost certainly gain very close to 5.3% of all gambled money over thousands of spins. Any winning streak by a player will eventually be overcome by the parameters of the game. It is important to remember that the LLN only applies (as the name indicates) when a large number of observations are considered. There is no principle that a small number of observations will converge to the expected value or that a streak of one value will immediately be "balanced" by the others. See the Gambler's fallacy.

History

Jacob Bernoulli first described the LLN as so simple that even the stupidest man instinctively knows it is true. ^[1] Despite this, it took him over 20 years to develop a sufficiently rigorous mathematical proof which was published in Ars Conjectandi'' (The Art of Conjecturing) in 1713. He named this his "Golden Theorem" but it became generally known as "Bernoulli's Theorem" (not to be confused with the Law in Physics with the same name.) In 1835, S.D. Poisson further described it under the name "La loi des grands nombres" (The law of large numbers)^[2]. Thereafter, it was known under both names, but the "Law of large numbers" is most frequently used.

After Bernoulli and Poisson published their efforts, other mathematicians also contributed to refinement of the law, including Chebyshev, Markov, Borel, Cantelli and Kolmogorov. These further studies have given rise to two prominent forms of the LLN. One is called the "weak" law and the other the "strong" law. These forms do not describe different laws but instead refer to different ways of describing the convergence of the observed or measured probability to the actual probability, and the strong form implies the weak.

Forms

Both versions of the law state that the sample average

converges towards the expected value

where X₁, X₂, ... an infinite sequence of i.i.d. random variables with finite expected value E(X₁)=E(X₂) = ... = µ < ∞.

An assumption of finite variance Var(X₁) = Var(X₂) = ... = σ² < ∞ is not necessary. Large or infinite variance will make the convergence slower, but LLN holds anyway. This assumption is often used because it makes the proofs easier and shorter.

The difference between the strong and the weak version is which kind of convergence we are talking about.

The weak law

The weak law of large numbers states that the sample average converges in probability towards the expected value

That is to say that for any positive number ε,

(Proof)

Interpreting the convergence in probability, the weak law essentially states that the average of many observations will eventually be close to the mean within any nonzero margin specified, no matter how small.

This version is called the weak law because convergence in probability is weak convergence of random variables.

A consequence of the weak LLN is the asymptotic equipartition property.

The strong law

The strong law of large numbers states that the sample average converges almost surely to the expected value

That is,

The proof is more complex than that of the weak law. This law justifies the intuitive interpretation of the expected value of a random variable as the "long-term average when sampling repeatedly".

This version is called the strong law because almost sure convergence is strong convergence of random variables. The strong law implies the weak law.

References

• Grimmett, G. R. and Stirzaker, D. R. (1992). Probability and Random Processes, 2nd Edition. Clarendon Press, Oxford. ISBN 0-19-853665-8. 
• Richard Durrett (1995). Probability: Theory and Examples, 2nd Edition. Duxbury Press. 
• Martin Jacobsen (1992). Videregående Sandsynlighedsregning (Advanced Probability Theory) 3rd Edition''. HCØ-tryk, Copenhagen. ISBN 87-91180-71-6. 

See also

• Central limit theorem
• Gambler's fallacy
• Law of averages

External links

• MathWorld: Weak Law of Large Numbers
• MathWorld: Strong Law of Large Numbers
• Laws of Chance Tables - used for testing claims of success greater than what can be attributed to random chance

Probability is the likelihood that something is the case or will happen. Probability theory is used extensively in areas such as statistics, mathematics, science and philosophy to draw conclusions about the likelihood of potential events and the underlying mechanics of
..... Click the link for more information.

A random variable is an abstraction of the intuitive concept of chance into the theoretical domains of mathematics, forming the foundations of probability theory and mathematical statistics.
..... Click the link for more information.

In probability theory, to say that two events are independent, intuitively means that the occurrence of one event makes it neither more nor less probable that the other occurs.
..... Click the link for more information.

expected value (or mathematical expectation, or mean) of a discrete random variable is the sum of the probability of each possible outcome of the experiment multiplied by the outcome value (or payoff).
..... Click the link for more information.

variance of a random variable (or somewhat more precisely, of a probability distribution) is one measure of statistical dispersion, averaging the squared distance of its possible values from the expected value.
..... Click the link for more information.

In mathematics and statistics, the arithmetic mean (or simply the mean) of a list of numbers is the sum of all the members of the list divided by the number of items in the list. The arithmetic mean is what students are taught very early to call the "average".
..... Click the link for more information.

Dice (the plural of die, from Old French dé, from Latin datum "something given or played" [1] ) are small polyhedral objects, usually cubical, used for generating random numbers or other symbols.
..... Click the link for more information.

multinomial distribution is a generalization of the binomial distribution.

The binomial distribution is the probability distribution of the number of "successes" in n independent Bernoulli trials, with the same probability of "success" on each trial.
..... Click the link for more information.

fair coin. One for which the probability is not 1/2 is called a biased or unfair coin.

Fair results from a biased coin

If a cheater has altered a coin to prefer one side over another (a biased coin), surprisingly the coin can still be used for fair results by
..... Click the link for more information.

Roulette is a casino and gambling game named after the French word meaning "small wheel". In the game a croupier spins a wheel in one direction, then spins a ball in the opposite direction around a tilted circular surface running around the circumference of the wheel.
..... Click the link for more information.

The gambler's fallacy is a formal fallacy. It is the incorrect belief that the likelihood of a random event can be affected by or predicted from other, independent events.
..... Click the link for more information.

Bernoulli's equation redirects here; see Bernoulli differential equation for an unrelated topic in ordinary differential equations.

Bernoulli's Principle
..... Click the link for more information.

Pafnuty Chebyshev

Pafnuty Lvovich Chebyshev
Born May 16 [O.S. May 4] 1821
Borovsk, Kaluga, Russia
Died December 8 [O.S.
..... Click the link for more information.

Andrey Markov

Andrey Andreyevich Markov
Born May 14 1856(1856--) N.S.
..... Click the link for more information.

Félix Édouard Justin Émile Borel (January 7, 1871 – February 3, 1956) was a French mathematician and politician.

Borel was born in Saint-Affrique, France. Along with René-Louis Baire and Henri Lebesgue, he was among the pioneers of measure theory and its application to
..... Click the link for more information.

Francesco Paolo Cantelli (1875-1966) was an Italian mathematician. He was the founder of the Istituto Italiano degli Attuari for the applications of mathematics and probability to economics.

His early papers were on problems in astronomy and celestial mechanics.
..... Click the link for more information.

Andrey Kolmogorov

Born March 25 1903(1903--)
Tambov, Imperial Russia
Died September 20 1987 (aged 84)
Moscow, USSR
..... Click the link for more information.

In the absence of a more specific context, convergence denotes the approach toward a definite value, as time goes on; or to a definite point, a common view or opinion, or toward a fixed or equilibrium state.
..... Click the link for more information.

independent and identically distributed (i.i.d.) if each has the same probability distribution as the others and all are mutually independent.

The abbreviation i.i.d.
..... Click the link for more information.

in distribution, if

:

for every real number a at which F is continuous. Since F(a) = Pr(X ≤ a), this means that the probability that the value of X
..... Click the link for more information.

Theorem:

Proof using Chebyshev's inequality

This proof uses the assumption of finite variance (for all ). The independence of the random variables implies no correlation between them, and we have that

..... Click the link for more information.

In information theory the asymptotic equipartition property (AEP) is a general property of the output samples of a stochastic source. It is fundamental to the concept of typical set used in theories of compression.
..... Click the link for more information.

in distribution, if

:

for every real number a at which F is continuous. Since F(a) = Pr(X ≤ a), this means that the probability that the value of X
..... Click the link for more information.

In probability theory, an event happens almost surely (a.s.) if it happens with probability one. The concept is analogous to the concept of "almost everywhere" in measure theory.
..... Click the link for more information.

A central limit theorem is any of a set of weak-convergence results in probability theory. They all express the fact that any sum of many independent and identically-distributed random variables will tend to be distributed according to a particular "attractor distribution".
..... Click the link for more information.

The gambler's fallacy is a formal fallacy. It is the incorrect belief that the likelihood of a random event can be affected by or predicted from other, independent events.
..... Click the link for more information.

The law of averages is a lay term used to express a belief that outcomes of a random event will "even out" over a small sample.

As invoked in everyday life, the "law" usually reflects bad statistics or wishful thinking rather than any mathematical principle.
..... Click the link for more information.