# 举例分析Word2vec源码

### 初始化工作

Word a b c d e
Count 1 2 3 4 6

Word a b c d e
ID 1 2 3 4 5

#### 构建Huffman Tree

1. 词库中每个单词，对应到Huffman Tree的叶子节点，都有ID；
2. Huffman　Tree上，每个非叶子节点也有ID，是两套不同的ID编号，都从1开始，前者最大ID为单词数，后者最大ID为非叶子节点数，其实我们通过观察Huffman Tree，可以发现前者最大ID比后者大1，即叶子节点比非叶子节点多1.

### CBoW(Continuous Bag of Words)

#### Input Layer to Hidden Layer

$$syn0_{5*6}= \begin{bmatrix} syn0_{11} & \ldots & syn0_{16} \\ \vdots & \vdots & \vdots \\ syn0_{51} & \ldots & syn0_{56} \end{bmatrix} \tag{2.1.1}$$

$$neu0_{1*6}= \begin{bmatrix} 0 &1 & 1 & 0 & 0 \end{bmatrix} \cdot \begin{bmatrix} syn0_{11} & \ldots & syn0_{16} \\ \vdots & \vdots & \vdots \\ syn0_{51} & \ldots & syn0_{56} \end{bmatrix}= \begin{bmatrix} neu1_{11} & \ldots & neu1_{16} \end{bmatrix} \tag{2.1.2}$$

#### Hidden Layer to Output Layer

$$syn1_{6*4}= \begin{bmatrix} syn1_{11} & \ldots & syn1_{14} \\ \vdots & \vdots & \vdots \\ syn1_{61} & \ldots & syn1_{64} \end{bmatrix} \tag{2.2.1}$$

$$y_{1*4}= \begin{bmatrix} neu1_{11} & \ldots & neu1_{16} \end{bmatrix} \cdot \begin{bmatrix} syn1_{11} & \ldots & syn1_{14} \ \vdots & \vdots & \vdots \\ syn1_{61} & \ldots & syn1_{64} \end{bmatrix}= \begin{bmatrix} y_1 & y_2 & y_3 & y_4 \end{bmatrix} \tag{2.2.2}$$

$$y_4=\begin{bmatrix} neu1_{11} & \ldots & neu1_{16} \end{bmatrix} \cdot \begin{bmatrix} syn1_{14}\\ \vdots \\ syn1_{64} \end{bmatrix} \tag{2.2.3}$$

$$y_3=\begin{bmatrix} neu1_{11} & \ldots & neu1_{16} \end{bmatrix} \cdot \begin{bmatrix} syn1_{13}\\ \vdots \\ syn1_{63} \end{bmatrix} \tag{2.2.4}$$

### 关于Word2Vec中模型的训练–进阶

#### Train syn0

$$syn0’=syn0 + alpha * \begin{bmatrix} 0\\ 1\\ 1\\ 0\\ 0 \end{bmatrix} \cdot ( \begin{bmatrix} \gamma_1 & \gamma_1 & \gamma_1 & \gamma_1 \end{bmatrix} *neu1*(1-neu1)))\\ =syn0 + alpha* \begin{bmatrix} 0 & 0 & 0 & 0\\ t_1 & t_2 & t_3 & t_4\\ t_1 & t_2 & t_3 & t_4\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ \end{bmatrix},\ where\ t_i=\gamma_i * neu1_i$$

$$syn0_{2*}’=syn0_{2*} + alpha* \begin{bmatrix} \gamma_1 & \gamma_2 & \gamma_3 & \gamma_4 \end{bmatrix} *neu1*(1-neu1)\\ syn0_{3*}’=syn0_{3*} + alpha* \begin{bmatrix} \gamma_1 & \gamma_2 & \gamma_3 & \gamma_4 \end{bmatrix} *neu1*(1-neu1)$$

$$syn0_{2*}’=syn0_{2*} + alpha* \begin{bmatrix} \gamma_1 & \gamma_2 & \gamma_3 & \gamma_4 \end{bmatrix})$$