摘要
多模态知识图谱的融合需要解决知识融合过程中的实体对齐问题。在多模态知识图谱中,多模态属性可以提供关键对齐信息来提升实体对齐的能力。本文提出一种基于多模态属性嵌入和图注意力网络的多模态知识图谱实体对齐方法。首先,根据多模态知识图谱中图像、文本和图谱结构信息,将多模态知识图谱划分成子图;其次,利用图注意力网络提取文本和图结构信息,利用视觉几何组(visual geometry group, VGG)网络提取图像特征信息;然后,将文本、图像和图结构特征生成嵌入表示到向量空间;最后,综合子图的多模态特征和图结构特征用于对齐。实验结果表明,在对齐任务中该模型相比于4种基线模型性能有明显提升(Hits@1、Hits@10和MRR提升了10.64%、5.60%和0.227)。
The fusion of multimodal knowledge graph requires addressing the entity alignment problem in knowledge fusion.In multimodal knowledge graph,multimodal attributes can provide key alignment information to improve entity alignment effectiveness.This paper proposes a method for entity alignment in multimodal knowledge graphs based on multimodal attribute embedding and graph attention network.First,the multimodal knowledge graph is divided into subgraphs according to image,text and graph structure information.Text and graph structure information are then extracted by graph attention network,while image information is extracted by visual geometry group(VGG)network.These multimodal attributes are embedded into vector space.Finally,the proposed method integrates the multimodal attributes and the graph structure of the subgraphs for alignment.Experimental results shows that the proposed model significantly improves performance,achieving increases of 10.64% on Hits@1, 5.60% on Hits@10, and 0.226 on MRR comparedto four baseline models for entity alignment.
作者
刘炜
徐辉
李卫民
LIU Wei;XU Hui;LI Weimin(School of Computer Engineering and Science,Shanghai University,Shanghai 200444,China;School of Artificial Intelligence,Shanghai University,Shanghai 200444,China)
出处
《应用科学学报》
CAS
CSCD
北大核心
2024年第6期1040-1051,共12页
Journal of Applied Sciences
基金
国家自然科学基金重大项目(No.61991410)
浦江国家实验室项目(No.P22KN00391)资助。
关键词
多模态知识图谱
实体对齐
多模态属性嵌入
图注意力网络
multimodal knowledge graph
entity alignment
multimodal attribute embedding
graph attention network
