Prove for Sample Mean

Suppose we have variable $X$ following a distribution where $E (X) = μ$ . But we don't know $μ$ , instead we have multiple observations of $X$ , like $X_{1}, X_{2}, X_{3} \dots$

let

X = (\begin{matrix} X_{1} \\ X_{2} \\ ⋮ \\ X_{n} \end{matrix})

and the mean is

\hat{μ} = (\begin{matrix} μ \\ μ \\ ⋮ \\ μ \end{matrix})

We want to find a value of $μ$ that minimises $X - \hat{μ}$ . The residual vector (in red) should thus be minimised. Thus, $X - \hat{μ}$ should be perpendicular to $\hat{μ}$ . Using Pythagoras theorem

| | \hat{μ} | |^{2} + | | X - \hat{μ} | |^{2} = | | X | |^{2}

\begin{aligned} \sum^{N} μ^{2} + \sum_{i = 1}^{N} (X_{i} - μ)^{2} & = \sum_{i = 1}^{N} X_{i}^{2} \\ \sum_{i = 1}^{N} μ^{2} + X_{i}^{2} - 2 μ X_{i} + μ^{2} & = \sum_{i = 1}^{N} X_{i}^{2} \\ 2 N μ^{2} - 2 μ \sum_{i = 1}^{N} X_{i} & = 0 \\ 2 N μ^{2} & = 2 μ \sum_{i = 1}^{N} X_{i} \\ μ & = \frac{1}{N} \sum_{i = 1}^{N} X_{i} \end{aligned}