融合ALBERT与BiGRU-Attention-CRF模型的湖泊命名实体识别
1. 广东海洋大学, 电子与信息工程学院, 广东湛江 524088
2. 广东海洋大学, 化学与环境学院, 广东湛江 524088
| 摘 要： | 中国湖泊众多且分布广泛，全面掌握湖泊信息对推动重大水网工程实施、河湖生态环境修复和智慧水利建设等具有重要意义。该文运用深度学习模型对湖泊实体进行识别与提取，为从湖泊文本中挖掘有效湖泊信息提拱了参考。首先，自定义了湖泊命名实体识别(Named Entity Recognition, NER)方法和标注规范，并建立湖泊文本语料库。其次，考虑到自建的语料库规模较小，文本信息的稀疏性可能影响模型的性能和识别效果，引入轻量预训练模型(A Little Bidirectional Encoder Representations from Transformers, ALBERT)生成高质量词向量，作为双向门控循环单元-条件随机场（BiGRU-CRF）模型的输入层特征向量，并将注意力机制融入到该模型为实体的语义信息增加特征权重，提升实体特征提取效果。最后，利用自建的语料库完成大量验证试验，并对比4种深度学习模型的识别效果。结果表明：ALBERT-BiGRU-Attention-CRF模型对数据集整体的识别效果良好，准确率、召回率和F1分别达到91.26%、90.38%和90.81%。该模型对高频出现的4类实体识别的性能均优于其他对比模型。 |
| 关 键 词： | 命名实体识别; 预训练模型; 双向门控循环单元; 注意力机制; 条件随机场 |
| DOI： | 10.57237/j.se.2022.01.003 |
View:
Downloads:
收稿日期: 2022-09-29
接受日期: 2022-10-21
在线出版日期: 2022-11-01
Fusion of ALBERT and BiGRU-Attention-CRF Models for Named Entity Recognition in Lakes
1. School of Electronics and Information Engineering, Guangdong Ocean University, Zhanjiang 524088, China
2. School of Chemistry and Environment, Guangdong Ocean University, Zhanjiang 524088, China
| Abstract: | There are many lakes in China and they are widely distributed, so it is important to have comprehensive information on lakes to promote the implementation of major water network projects, ecological and environmental restoration of rivers and lakes, and smart water conservancy construction. This paper uses a deep learning model to identify and extract lake entities, and provides a reference for mining effective lake information from lake text. Firstly, the Named Entity Recognition (NER) method and annotation specification are customized, and a corpus of lake texts is established. Secondly, considering the small size of the self-built corpus and the sparsity of text information may affect the performance of the model and the recognition effect, a lightweight pre-training model (ALBERT) is introduced to generate high-quality word vectors as a two-way gating cycle The unit-conditional random field (BiGRU-CRF) model is used to generate the input layer feature vectors, and the attention mechanism is incorporated into the model to add feature weights to the semantic information of the entities to improve the entity feature extraction effect. Finally, a large number of validation experiments are completed, and the recognition effects of the four deep learning models are compared. The results show that the ALBERT-BiGRU-Attention-CRF model has good recognition effect on the dataset as a whole, with accuracy, recall and F1 reaching 91.26%, 90.38% and 90.81%, respectively. In addition, the model outperforms the other comparison models for the recognition of all four types of entities that occur at high frequencies. |
| Keywords: | NER; ALBERT; BiGRU; Attention; CRF |
基金
| 1. | 国家自然科学基金项目 (42176067) |
| 2. | 广东海洋大学强校工程项目 (Q18307) |
参考文献
| [1] | 第一次全国水利普查公报 [J]. 中华人民共和国水利部公报, 2013 (02): 53-57. |
| [2] | Rau L F. Extracting Company Names from Text [C] // The Seventh IEEE Conference on Artificial Intelligence Application. IEEE, 1991: 29-32. |
| [3] | 李军怀, 陈苗苗, 王怀军, 崔颖安, 张爱华. 基于ALBERT-BGRU-CRF的中文命名实体识别方法 [J]. 计算机工程, 2022, 48 (06): 89-94+106. |
| [4] | 巩敦卫, 张永凯, 郭一楠, 王斌, 樊宽鲁, 火焱. 融合多特征嵌入与注意力机制的中文电子病历命名实体识别 [J]. 工程科学学报, 2021, 43 (09): 1190-1196. |
| [5] | Chen Y, Xu L, Liu K, et al. Event extraction via dynamic multi-pooling convolutional neural networks [C] // Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015: 167-176. |
| [6] | Voorhees E, Harman DK (2006) Trec: experiment and evaluation in information retrieval. J Am Soc Inf Sci Technol 32 (4): 563–567. |
| [7] | Fader A, Zettlemoyer L, Etzioni O. Paraphrase-driven learning for open question answering [C] // Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2013: 1608-1618. |
| [8] | Yao X, Van Durme B. Information extraction over structured data: Question answering with freebase [C] // Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2014: 956-966. |
| [9] | 许宁. 面向旅游领域的智能问答系统设计与实现 [D]. 内蒙古大学, 2021. |
| [10] | Bunescu R, Mooney R. A shortest path dependency kernel for relation extraction [C] // Proceedings of human language technology conference and conference on empirical methods in natural language processing. 2005: 724-731. |
| [11] | Miwa M, Bansal M. End-to-end relation extraction using lstms on sequences and tree structures. In: ACL. 2016: 1105–1116. |
| [12] | Singh S, Riedel S, Martin B, Zheng J, McCallum A. Joint inference of entities, relations, and coreference. In: AKBC. 2013: 1–6. |
| [13] | Han S, Hao X, Huang H. An event-extraction approach for business analysis from online Chinese news [J]. Electronic Commerce Research and Applications, 2018, 28: 244-260. |
| [14] | Upadhyay S, Gupta N, Roth D. Joint multilingual supervision for cross-lingual entity linking [J]. arXiv preprint arXiv: 1809. 07657, 2018. |
| [15] | Li J, Bu C, Li P, et al. A coarse-to-fine collective entity linking method for heterogeneous information networks [J]. Knowledge-Based Systems, 2021, 228: 107286. |
| [16] | 冯钧, 杭婷婷, 陈菊, 王云峰, 王秉发, 张涛. 领域知识图谱研究进展及其在水利领域的应用 [J]. 河海大学学报(自然科学版), 2021, 49 (01): 26-34. |
| [17] | 段浩, 韩昆, 赵红莉, 蒋云钟, 李豪, 毛文山. 水利综合知识图谱构建研究 [J]. 水利学报, 2021, 52 (08): 948-958. |
| [18] | 刘婷, 张社荣, 王超, 李志竑, 关炜, 王枭华. 水利施工事故文本智能分析的BERT-BiLSTM混合模型 [J]. 水力发电学报, 2022, 1 (07): 1-12. |
| [19] | Cao P, Chen Y, Liu K, et al. Adversarial transfer learning for Chinese named entity recognition with self-attention mechanism [C] // Proceedings of the 2018 conference on empirical methods in natural language processing. 2018: 182-192. |
| [20] | Xu C, Wang F, Han J, Li C. Exploiting multiple embeddings for Chinese named entity recognition. In: Proceedings of the 28th ACM international conference on information and knowledge management. 2019: 2269-2272. |
| [21] | Greenberg N, Bansal T, V erga P, McCallum A. Marginal likelihood training of bilstm-crf for biomedical named entity recognition from disjoint label sets. In: Proceedings of the 2018 conference on empirical methods in natural language processing. 2018: 2824–2829. |
| [22] | Lan Z, Chen M, Goodman S, et al. Albert: A lite bert for self-supervised learning of language representations [J]. arXiv preprint arXiv:1909.11942, 2019. |
| [23] | http://www.lakesci.csdb.cn |
| [24] | Ju M, Miwa M, Ananiadou S. A neural layered model for nested named entity recognition. In: Proceedings of the 2018 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies, vol 1 (Long Papers). 2018: 1446–1459. |
| [25] | Chung J, Gulcehre C, Cho K H, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling [J]. arXiv preprint arXiv: 1412. 3555, 2014. |
| [26] | Zhu Q L, Li X L, Conesa A, et al. GRAM-CNN: a deep learning approach with local context for named entity recognition in biomedical text. Bioinformatics. 2018, 34 (9): 1547. |
| [27] | WOJEK C SCHIELE B. A dynamic conditional random field model for joint labeling of object and scene classes [C] // Proceedings of the 10th European Conference on Computer Vision, LNCS 5305. Berlin: Springer. 2008: 733-747. |
