words per minute
2
Typist_4524643
00:00
Speed
The log-likelihoodfunction of the GMM is hard to optimize because it contains the logarithm of a sum of terms which makes it really hard to solve analytically. Instead of doing this, we'll use the iterative EM algorithm which uses the posterior distribution of the latent variables given by $P(Z X,\Theta^0)$ to find the expectation of the complete data log-likelihood with respect to this posterior. This expectation is denoted $Q(\Theta,\Theta^0)$. By defining the posterior probability $\eta_{nk} := P(Z_n = k x_n,\Theta^0)$ and also $S_k=\sum_{n=1}^{N}\eta_{nk}$
we have by definition of log-likelyhood:
\begin{split}
\frac{\partial Q(\Theta,\Theta_0)}{\partial \mu_k} & = \sum_{n=1}^N \Big( 0 + \eta_{nk}\big(0 + 0 + \frac{1}{2}\Sigma_k^{-1}(x_n - \mu_k^*) \big) \Big) \text{ (using the identity $\nabla_x (x^TAx) = Ax$)} \\
0 & = \Sigma_k^{-1}( -S_k\mu_k^* + \sum_{n=1}^N \eta_{nk} x_n) \\
S_k\mu_k^* & = \sum_{n=1}^N \eta_{nk} x_n \text{ ($\Sigma_k^{-1}$ is invertible)} \\
\mu_k^* & = \frac{1}{S_k}\sum_{n=1}^N \eta_{nk} x_n
\; \llap{\mathrel{\boxed{\phantom{\mu_k^* = \frac{1}{S_k}\sum_{n=1}^N \eta_{nk} x_n }}}}
\end{split}