Calculate gradients for Word2Vec

Now that we have learned how to calculate gradients w.r.t. vectors in the last blog post, let’s see whether we can apply this technique to another model, the famous Word2vec.

Basically what Word2vec is trying to do is to predict every word around each word. It does this by calculating the probability of every possible choice in the vocabulary to be the context word for each center word.

Concretely, if $u_o$ is the vector representing the context word, and $v_c$ is the vector representing the center word, then the probability of $u_o$ being the correct context word is: $\hat{y}_o = p(o|c) = \frac{e^{u_o^T v_c}}{\sum_{w=1}^V e^{u_w^Tv_c}}$. Now we want this to be as high as possible, which means the cost function is defined as below:

$J = CE(y, \hat{y}) = -\sum_{i=1}^{V} y_i \log \hat{y}_i$

Let’s define $\theta_o = u_o^T v_c$. Then we already know that $\frac{\partial J}{\partial \theta} = \hat{y} - y$.

Now we go back.