本論文著重於SemEval研討會的挑戰，其挑戰為對於推文的三類情緒分類問題，我們實作出一個辨識系統來處理極性分類問題。我們基於先前所提出的方法，針對多元文本資訊進行擷取，並對分類模型進行整合。於資料前處理上，我們首先對資料進行分析，並挑選出一些詞彙的種類進行縮寫還原與表情文字正規化處理，藉此使資料能夠降低文本複雜度及獲得更好的表示方式；於詞向量調適部分，我們使用skip-gram模型，並在模型的預測項目加入推文極性與情緒詞數量，使模型對於預先訓練好的詞向量GloVe進行調適，將一般目的的詞向量被調整成適合情緒分類問題的詞向量；此外，我們嘗試將調適後的詞向量與預先訓練的詞向量串聯，結合詞向量之不同特性。而我們發現分類模型有過擬合的現象，因此我們將針對這部分進行改善。模型架構上，我們選擇了類神經網路作為我們的分類器，其模型架構使用了1層的CNN與LSTM模型，當模型訓練完之後，我們考慮所有模型之間的不同特性，因此對模型進行整合的動作。我們以本論文所提出的方法，相對於比賽時的CNN與LSTM模型之分類結果，平均召回率分別提升了約2%。

In this paper, we participate in the task on SemEval, which is sentiment analysis in Twitter. We implement a recognition system for the polarity classification. The method described in this paper is based on our previous work. We try to capture information from multiple context and combine models. In data pre-processing, we first analysis data, and then do data preprocessing with abbreviations and emoticons in order to decrease the data complexity and make better representation of data. In word vectors adaptation, we use skip-gram model labelling with polarity and number of sentiment words to adapt the pre-trained word vector, GloVe, from general-purpose word vector to sentiment-specific word vector. Moreover, we concatenate sentiment-specific word vector with pre-trained word vector. While we find that our model has over-fitting issue, we try to fix the problem. In our models, we choose neural network as our classifiers, where model architectures are 1-layer CNN model and LSTM model. After model training, we consider different characteristics between models and combine them. The average recall of our CNN and LSTM models with proposed methods described in this paper increase about 2% respectively than the model in our previous work at SemEval 2017.

論文審定書 i
Acknowledgments ii
摘要 iii
ABSTRACT iv
Table of Contents v
List of Tables vii
List of Figures viii
Chapter 1 簡介 1
1.1 背景與研究動機 1
Chapter 2 相關研究 4
2.1 類神經網路 4
2.2 分散式表示法與詞嵌入 5
2.3 SemEval 7
2.3.1 問題描述與評估方式 7
2.3.2 相關研究 8
2.3.3 先前研究 9
Chapter 3 研究方法 14
3.1 資料集與詞向量 14
3.1.1 資料集簡介 14
3.1.2 資料前處理 15
3.1.3 詞向量調適 17
3.2 模型架構 20
3.2.1 模型整合 20
3.2.2 工具 20
Chapter 4 實驗結果 22
4.1 實驗敘述 22
4.1.1 實驗流程 22
4.1.2 實驗設定與基準實驗 23
4.1.3 整體實驗整理 23
4.2 資料前處理之實驗 25
4.3 詞向量調適之實驗 26
4.3.1 不同詞向量用於情緒分類問題之實驗 26
4.3.2 詞向量串聯之實驗 29
4.3.3 詞向量之相似性 30
Chapter 5 結論與未來展望 33
Bibliography 34

[1] J. Bollen, H. Mao, and X. Zeng, “Twitter Mood Predicts the Stock Market,” Journal of computational science, vol. 2, no. 1, pp. 1–8, 2011.

[2] B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up? Sentiment Classification using Machine Learning Techniques,” in Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pp. 79–86, Association for Computational Linguistics, 2002.

[3] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.

[4] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet Classification with Deep ConvolutionalNeural Networks,” in Advances in neural information processing systems, pp. 1097–1105, 2012.

[5] A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, and J. Schmidhuber, “A Novel Connectionist System for Unconstrained Handwriting Recognition,” IEEE transactions on pattern analysis and machine intelligence, vol. 31, no. 5, pp. 855–868, 2009.

[6] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech Recognition with Deep Recurrent Neural Networks,” in Acoustics, speech and signal processing (icassp), 2013 ieee international conference on, pp. 6645–6649, IEEE, 2013.

[7] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” in Advances in neural information processing systems, pp. 3104–3112, 2014.

[8] P. J. Werbos, “Backpropagation Through Time: What It Does and How to Do It,” Proceedings of the IEEE, vol. 78, no. 10, pp. 1550–1560, 1990.

[9] Y. Bengio, P. Simard, and P. Frasconi, “Learning Long-Term Dependencies with Gradient Descent Is Difficult,” IEEE transactions on neural networks, vol. 5, no. 2, pp. 157–166, 1994.

[10] S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.

[11] R. Pascanu, T. Mikolov, and Y. Bengio, “On the Difficulty of Training Recurrent Neural Networks.,” ICML (3), vol. 28, pp. 1310–1318, 2013.

[12] M. Schuster and K. K. Paliwal, “Bidirectional Recurrent Neural Networks,” IEEE Transactions on Signal Processing, vol. 45, no. 11, pp. 2673–2681, 1997.

[13] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling,” arXiv preprint arXiv:1412.3555, 2014.

[14] O. Delalleau and Y. Bengio, “Shallow vs. Deep Sum-Product Networks,” in Advances in Neural Information Processing Systems, pp. 666–674, 2011.

[15] R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio, “How to Construct Deep Recurrent Neural Networks,” arXiv preprint arXiv:1312.6026, 2013.

[16] T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khudanpur, “Recurrent Neural Network Based Language Model.,” in Interspeech, vol. 2, p. 3, 2010.

[17] X. Chen, X. Liu, Y. Qian, M. Gales, and P. C. Woodland, “CUED-RNNLM—An OpenSource Toolkit for Efficient Training and Evaluation of Recurrent Neural Network Language Models,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pp. 6000–6004, IEEE, 2016.

[18] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al., “Tensorflow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,” arXiv preprint arXiv:1603.04467, 2016.

[19] F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I. Goodfellow, A. Bergeron, N. Bouchard, D. Warde-Farley, and Y. Bengio, “Theano: New Features and Speed Improvements,” arXiv preprint arXiv:1211.5590, 2012.

[20] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.

[21] G. E. Hinton, J. L. McClelland, and D. E. Rumelhart, “Distributed Representations,” Rumelhart, D. E. and McClelland, J. L., editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations, MIT Press, Cam-
bridge, MA, pp. 77–109, 1986.

[22] Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, “A Neural Probabilistic Language Model,” Journal of machine learning research, vol. 3, no. Feb, pp. 1137–1155, 2003.

[23] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient Estimation of Word Representations in Vector Space,” arXiv preprint arXiv:1301.3781, 2013.

[24] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, “Natural Language Processing (Almost) from Scratch,” Journal of Machine Learning Research, vol. 12, no. Aug, pp. 2493–2537, 2011.

[25] D. Tang, F. Wei, N. Yang, M. Zhou, T. Liu, and B. Qin, “Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification,” in ACL (1), pp. 1555–1565, 2014.

[26] P. Nakov, A. Ritter, S. Rosenthal, F. Sebastiani, and V. Stoyanov, “SemEval-2016 Task 4: Sentiment Analysis in Twitter,” Proceedings of SemEval, pp. 1–18, 2016.

[27] J. Deriu, M. Gonzenbach, F. Uzdilli, A. Lucchi, V. De Luca, and M. Jaggi, “SwissCheese at SemEval-2016 Task 4: Sentiment Classification Using An Ensemble of Convolutional Neural Networks with Distant Supervision,” Proceedings of SemEval, pp. 1124–1128, 2016.

[28] X. Xu, H. Liang, and T. Baldwin, “UNIMELB at SemEval-2016 Tasks 4A and 4B: An Ensemble of Neural Networks and a Word2Vec Based Model for Sentiment Classification,” Proceedings of SemEval, pp. 183–189, 2016.

[29] M. Nabil, A. Atyia, and M. Aly, “CUFE at SemEval-2016 Task 4: A Gated Recurrent Model for Sentiment Classification,” in Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 2016.

[30] T.-H. Yang, T.-H. Tseng, and C.-P. Chen, “deepSA at SemEval-2017 Task 4: Interpolated Deep Neural Networks for Sentiment Analysis in Twitter,” Proceedings of the 11th International Workshop on Semantic Evaluation. Vancouver, Canada, 2017.

[31] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global Vectors for Word Representation.,” in EMNLP, vol. 14, pp. 1532–1543, 2014.

[32] Y. Kim, “Convolutional Neural Networks for Sentence Classification,” arXiv preprint arXiv:1408.5882, 2014.

[33] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.,” Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.

[34] M. Rouvier and B. Favre, “SENSEI-LIF at SemEval-2016 Task 4: Polarity Embedding Fusion for Robust Sentiment Analysis,” Proceedings of SemEval, pp. 202–208, 2016.