Tutorial on Expectation and Variance Doron ZEILBERGER

IMPORTANT REMINDER (Probability) If you have a finite number of scenarios, say two, tossing a coin, where there are only two outcomes, Heads (H) and Tail (T), we say that the set of scenarios is the sample space. If it is a fair coin, then Heads is supposed, in the long run to come up roughly as often as Tails (in real life it is hardly ever exact), so we say that the probability of Heads and Probability of Tails are both \frac{1}{2}. In this case the sample space is {H, T } and

P (H) = \frac{1}{2} , P (T) = \frac{1}{2} ,

But if the coin is loaded, say in the long run, Heads only shows up 1 in ten times, then

P (H) = \frac{1}{10} , P (T) = \frac{9}{10}

If you roll a die, the sample space is:

{1, 2, 3, 4, 5, 6} .

The individual scenarios (namely, landing on 1, landing on 2, etc.) are called atomic events. If it is a fair coin then all the probabilities are the same, and hence

P (1) = \frac{1}{6} , P (2) = \frac{1}{6} , P (3) = \frac{1}{6} , P (4) = \frac{1}{6} , P (5) = \frac{1}{6} , P (6) = \frac{1}{6} .

But if it is a loaded coin, then the probabilities may be different. For example

P (1) = \frac{1}{12} , P (2) = \frac{1}{4} , P (3) = \frac{1}{20} , P (4) = \frac{17}{60} , P (5) = \frac{1}{18} , P (6) = \frac{5}{18} .

Important Check: The sum of all the probabilities of the atomic events must be 1. An event is any subset of the set of atomic scenarios, including that nothing happened, the empty set, ∅ whose probability is 0 and the Universal Set, whose probability is 1. To compute the probability of an event, you simply add up the probabilities of its atoms. Example I: With the above loaded coin, find

(a) The probability of the event {1, 3, 6} (b) The probability of the event {2, 6}

(c) The probability that it landed on an even face.

(d) The probability that it landed on an odd face. Solution to Example I:

(a) P (\{1, 3, 6\}) = P (1) + P (3) + P (6) = \frac{1}{12} + \frac{1}{20} + \frac{5}{18} = \frac{37}{90}

(b) P (\{2, 6\}) = P (2) + P (6) = \frac{1}{4} + \frac{5}{18} = \frac{19}{36}

(c) The even faces are {2, 4, 6}, so P (EvenF ace) = P (\{2, 4, 6\} = P (2) + P (4) + P (6) = \frac{1}{4} + \frac{17}{60} + \frac{5}{18} = \frac{73}{90}

(d) The long way is to do the same for {1, 3, 5}, but since the complement of the event “odd” is “even” we can just do P (OddF ace) = 1 − P (EvenF ace) = 1 − \frac{73}{90} = \frac{17}{90}. Do Right Now Exercise 1: A certain tetrahedral die (with four faces) labeled 1, 2, 3, 4, has the following probabilities for the atomic events

P (1) = \frac{1}{8} , P (2) = \frac{3}{8} , P (3) = \frac{1}{16} , P (4) = \frac{7}{16} .

Find

(a) The probability of the event {1, 4}

(b) The probability of the event {1, 3, 4}

(c) The probability that it landed on an even face.

(d) The probability that it landed on an odd face. Important Concept: Random Variable Given our “universal set” of all atomic events, a random variable (a stupid name) is really an assigning of a value to any atomic scenario. For example if we have a coin and you are promised that if it lands Heads you get $10 dollars, and if lands Tails you lose $5 dollars, then the “random variable” is “gain”, and we have

X(Heads) = 10 , X(T ail) = −5 .

Another example, in our family, the random variable is height (in inches) then X(Doron) = 71 , X(Jane) = 63 , X(Celia) = 63 , X(T amar) = 64 , X(Hadas) = 69 .

Example II: In a six-faced die, labeled {1, 2, 3, 4, 5, 6}, if you are promised 20 − i^{2} dollars if it lands on a face with i dots, spell out the random variable, let’s call it X, for all possible scenarios. Solution to Example II:

X(1) = 20−1^{2} = 19 , X(2) = 20−2^{2} = 16 , X(3) = 20−3^{2} = 11 , X(4) = 20−4^{2} = 4 ,

X(5) = 20 − 5^{2} = −5 , X(6) = 20 − 6^{2} = −16 , .

Do Right Now! Exercise 2: In a four-faced die, labeled {1, 2, 3, 4} if you are promised 10 − 3i dollars if it lands on a face with i dots, spell out the random variable, let’s call it X for all possible scenarios. Important note: Once you have a random variable X defined on your sample space, you get many new ones 2X, 3X + 2, X^{2}, X^{3} etc. Example III: In a six-faced die, labeled {1, 2, 3, 4, 5, 6} if you are promised 20 − i^{2} dollars if it lands on a face with i dots, let’s call it X. Spell out the random variables (a)2X (b) X^{2} (c) X^{3} Solution to Example III: (a)

(2X)(1) = 2 · 19 = 38 , (2X)(2) = 2 · 16 = 32 , (2X)(3) = 2 · 11 = 22 ,
(2X)(4) = 2 · 4 = 8 , (2X)(5) = 2 · (−5) = −10 , (2X)(6) = 2 · (−16) = −32 , .

(b)

X^{2}(1) = 19^{2} = 361 , X^{2}(2) = 16^{2} = 256 , X^{2}(3) = 11^{2} = 121 ,
X^{2}(4) = 4^{2} , X^{2}(5) = (−5)^{2} = 25 , X^{2}(6) = (−16)^{2} = 256 , .

(c)

X^{3}(1) = 19^{3} = 6859 , X^{3}(2) = 16^{3} = 4096 , X^{3}(3) = 11^{3} = 1331 ,
X^{3}(4) = 4^{3} = 64 , X^{3}(5) = (−5)^{3} = −125 , X^{3}(6) = (−16)^{3} = −4096 , .

Exercise 3: In a four-faced die, labeled {1, 2, 3, 4} if you are promised 10 − 3i dollars if it lands on a face with i dots, let’s call the random variable describing the gain X. Spell out the random variables (a)2X (b) X^{2} (c) X^{3}.

Important Concept: Expectation of random variable

Suppose that the sample space is \{H, T \} and it is a fair coin, P (H) = \frac{1}{2} and P (T) = \frac{1}{2}. You are promised $100 every time it lands on Heads and nothing if it lands on Tails. How much would you expect to gain in the long-run per toss? Since in the long-run it would, roughly lands Heads every other time, your expected gain (per toss) is $50, which is 100 · \frac{1}{2} = 50.

If it is a loaded coin, with P (H) = \frac{1}{10} and P (T) = \frac{9}{10}, then you would only get $100 one-tenth of the time, so he would get, roughly, in the long-run, 100 · \frac{1}{10} = 10 dollars.

If it is a loaded coin, with P (H) = \frac{1}{10} and P (T) = \frac{9}{10}, but now you have to pay 10 dollars if it lands Tails then you would only get $100 one-tenth of the time, but have to pay 10 dollars, nine-tenth of the time, so in the long-run you should expect, per toss, 100 · \frac{1}{10} − 10 · \frac{9}{10} = 10 − 9 = 1 dollars.

This motivates the following

VERY IMPORTANT DEFINITION (FORMULA)

If X is a random variable on a sample space whose atomic events are {x_{1}, x_{2}, …, x_{n}}, with respective probabilities P (x_{1}), …, P (x_{n}), and X is any random variable, then the expectation of X, denoted by E(X), (often also by μ) is given by (in sigma notation)

E(X) = ∑_{i=1}^{n} P (x_{i})X(x_{i})

In more concrete … notation

E(X) = P (x_{1})X(x_{1}) + P (x_{2})X(x_{2}) + … + P (x_{n})X(x_{n}) .

Example IV: Using the same probability distribution as Example I, namely

P (1) = \frac{1}{12} , P (2) = \frac{1}{4} , P (3) = \frac{1}{20} , P (4) = \frac{17}{60} , P (5) = \frac{1}{18} , P (6) = \frac{5}{18}

and the same random variable,X, as in Example II, compute E(X)

Solution to Example IV:

E(X) = \frac{1}{12} · 19 + \frac{1}{4} · 16 + \frac{1}{20} · 11+

\frac{17}{60} · 4 + \frac{1}{18} · (−5) + \frac{5}{18} · (−16) = 229/90 = 2.544 ….

Answer to Example IV: The expectation E(X), is 229/90 = 2.544 …. Do Right Now Exercise 4: Using the same probability distribution as Exercise 1, namely

P (1) = \frac{1}{8} , P (2) = \frac{3}{8}, P (3) = \frac{1}{16} , P (4) = \frac{7}{16}

and the same random variable,X, as in Exercise 2, namely if you are promised 10 − 3i dollars if it lands on a face with i dots, compute the expectation E(X). Example IV’: Using the same probability distribution as in Example I, namely

P (1) = \frac{1}{12} , P (2) = \frac{1}{4} , P (3) = \frac{1}{20} , P (4) = \frac{17}{60} , P (5) = \frac{1}{18} , P (6) = \frac{5}{18} ,

and the same random variable,X, as in Example II, compute the expectation of X^{2}, namely E(X^{2}) Solution to Example IV’:

E(X) = \frac{1}{12} · (19)^{2} + \frac{1}{4} · (16)^{2} + \frac{1}{20} · (11)^{2}+

\frac{17}{60} · 4^{2} + \frac{1}{18} · (−5)^{2} + \frac{5}{18} · (−16)^{2} = \frac{1063}{6} = 177.16667 ….

Answer to Example IV’: The expectation of X^{2}, E(X^{2}), is \frac{1063}{6} = 177.16667 … Do Right Now Exercise 4’: Using the same probability distribution as Exercise 1, namely

P (1) = \frac{1}{8} , P (2) = \frac{3}{8}, P (3) = \frac{1}{16} , P (4) = \frac{7}{16}

and the same random variable,X, as in Exercise 2, namely if you are promised 10 − 3i dollars if it lands on a face with i dots, compute the expectation of X^{2}, E(X^{2}). Important Concept: Variance The Variance of a random variable, denoted by V ar(X) (often also σ^{2}) is the expectation of the square of the “deviation from expectation”, namely, calling E(X) = μ

E((X − μ)^{2}) .

There are two ways to compute it, the first way is to first figure out, explicitly, the values of the new random variable (X − μ)^{2} on each of the atomic events, and then take the expectation of that, but a better way is via the Important Formula

V ar(X) = E(X^{2}) − E(X)^{2} .

(alias E(X^{2}) − μ^{2}) Note: E(X^{2}) is NOT the same as E(X)^{2}, so the difference is not 0! In E(X^{2}) you first square X getting a brand-new random variable, X^{2} (like we did above), and in E(X)^{2}, you first find the expectation E(X) (alias μ) and then square it. In the extreme case where X is the same on every member of our sample space, the expectation equals to that common value, and then the variance is 0. Proof of the Important Formula

V ar(X) = E((X − μ)^{2}) = E(X^{2} − 2Xμ + μ^{2}) = E(X^{2}) − E(2Xμ) + E(μ^{2}) =
E(X^{2}) − 2μE(X) + μ^{2}E(1) = E(X^{2}) − 2μ^{2} + μ^{2} ˙1 = E(X^{2}) − μ^{2} .

Example IV”: Using the same probability distribution as Example I, namely

P (1) = \frac{1}{12} , P (2) = \frac{1}{4} , P (3) = \frac{1}{20} , P (4) = \frac{17}{60} , P (5) = \frac{1}{18} , P (6) = \frac{5}{18} ,

and the same random variable,X, as in Example II, compute the variance of X, namely V ar(X) Solution to Example IV”: From Example IV, and Example IV’

E(X) = 229/90 , E(X^{2}) = \frac{1063}{6}

Hence

V ar(X) = E(X^{2}) − E(X)^{2} = \frac{1063}{6} − (\frac{229}{90})^{2} = \frac{1382609}{8100} = 170.6924691 …

Answer to Example IV”: The variance of X, V ar(X), is \frac{1382609}{8100} = 170.6924691 …. Do Right Now Exercise 4”: Using the same probability distribution as Exercise 1, namely

P (1) = \frac{1}{8} , P (2) = \frac{3}{8}, P (3) = \frac{1}{16} , P (4) = \frac{7}{16}

and the same random variable,X, as in Exercise 2, namely if you are promised 10 − 3i dollars if it lands on a face with i dots, compute the variance of X, V ar(X).

Note: You may use the answers that you got for Exercises 4 and 4’. Do Right Now Exercise 5: In our family, consider the random variable, X, height (in inches) then

X(Celia) = 63 , X(T amar) = 64 , X(Hadas) = 69 .

If you pick a girl uniformly at random (each with probability \frac{1}{3}) find the expectation and variance of the Zeilberger girls. Exercise 5’: In our family, consider the random variable, X, height (in inches) then

X(Celia) = 63 , X(T amar) = 64 , X(Hadas) = 69 .

If I pick Celia with probability 0.1, Tamar with probability 0.4 and Hadas with probability .5, find the expectation and variance of the height with this probability distribution.

Doron Zeilberger, Department of Mathematics, Rutgers University (New Brunswick), Hill Center-Busch Campus, 110 Frelinghuysen Rd., Piscataway, NJ 08854-8019, USA.

Email: zeilberg at math dot rutgers dot edu .