•
the Blog
Fisher information (with a cat)
By Guillaume Filion, filed under
Immanuel,
bias–variance trade-off,
Fisher information,
dialogue.
• 13 December 2022 •
It is still summer but the days are getting shorter (p < 0.05). Edgar and Sofia are playing chess, Immanuel purrs in a sofa next to them. Edgar has been holding his head for a while, thinking about his next move. Sofia starts:
“Something bothers me Immanuel. In the last post, you told us that Fisher information could be defined as a variance, but that is not what I remember from my classes of mathematical statistics.”
“What do you remember, Sofia?”
“Our teacher said it was the curvature of the log-likelihood function around the maximum. More specifically, consider a parametric model $(f(X;\theta))$ where $(X)$ is a random variable and $(\theta)$ is a parameter. Say that the true (but unknown) value of the parameter is $(\theta^*)$. The first terms of the Taylor expansion of the log-likelihood $(\log f(X;\theta))$ around $(\theta^*)$ are
$$\log f(X;\theta^*) + (\theta - \theta^*) \cdot \frac{\partial}{\partial \theta} \log f(X;\theta^*) + \frac{1}{2}(\theta - \theta^*)^2 \cdot \frac{\partial^2}{\partial \theta^2} \log f(X;\theta^*).$$
Now compute the expected value and obtain the approximation below. We call it $(\varphi(\theta))$ to emphasize that it is...
A gentle introduction to the Cramér-Rao lower bound (with a cat)
By Guillaume Filion, filed under
Immanuel,
dialogue,
Fisher information.
• 22 November 2021 •
It is summer, Edgar and Sofia are comfortably sitting on the terrace, watching the beautiful light of the end of the day. Edgar starts:
“Let’s play a game to see who is the better statistician! Immanuel my cat will give each of us a secret number strictly greater than zero. The other person will have to guess it.”
“How are we going to guess?”
“Let’s say that the secret numbers are the means of some Poisson variable. We generate samples at random. The one who gets the closest estimate by dinner time wins.”
“That sounds easy! Will Immanuel give us the same number?”
“What is the fun in that? Let’s ask him to give two different numbers. You know what to do. Just give me your first sample whenever you are ready and I will try to guess your secret number.”
Immanuel whispers something in the ear of Sofia and then does the same with Edgar. Sofia opens her laptop and after a few keystrokes she says “The first number I have for you is 1.”
“OK, I give up. You win.”
Sofia is puzzled at first, but then she notices how Immanuel is rolling...