摘要
多模态数据间交互式任务的兴起对于综合利用不同模态的知识提出了更高的要求,因此融合不同模态知识的多模态知识图谱应运而生.然而,现有多模态知识图谱存在图谱知识不完整的问题,严重阻碍对信息的有效利用.缓解此问题的有效方法是通过实体对齐进行知识图谱补全.当前多模态实体对齐方法以固定权重融合多种模态信息,在融合过程中忽略不同模态信息贡献的差异性.为解决上述问题,设计一套自适应特征融合机制,根据不同模态数据质量动态融合实体结构信息和视觉信息.此外,考虑到视觉信息质量不高、知识图谱之间的结构差异也影响实体对齐的效果,本文分别设计提升视觉信息有效利用率的视觉特征处理模块以及缓和结构差异性的三元组筛选模块.在多模态实体对齐任务上的实验结果表明,提出的多模态实体对齐方法的性能优于当前最好的方法.
The recent surge of interactive tasks involving multi-modal data brings a high demand for utilizing knowledge in different modalities.This facilitated the birth of multi-modal knowledge graphs,which aggregate multi-modal knowledge to meet the demands of the tasks.However,they are known to suffer from the knowledge incompleteness problem that hinders the utilization of information.To mitigate this problem,it is of great need to improve the knowledge coverage via entity alignment.Current entity alignment methods fuse multi-modal information by fixed weighting,which ignores the different contributions of individual modalities.To solve this challenge,we propose an adaptive feature fusion mechanism,that combines entity structure information and visual information via dynamic fusion according to the data quality.Besides,considering that low quality visual information and structural difference between knowledge graphs further impact the performance of entity alignment,we design a visual feature processing module to improve the effective utilization of visual information and a triple filtering module to ease structural differences.Experiments on multi-modal entity alignment indicate that our method outperforms the state-of-the-arts.
作者
郭浩
李欣奕
唐九阳
郭延明
赵翔
GUO Hao;LI Xin-Yi;TANG Jiu-Yang;GUO Yan-Ming;ZHAO Xiang(College of Systems Engineering,National University of Defense Technology,Changsha 410073)
出处
《自动化学报》
EI
CAS
CSCD
北大核心
2024年第4期758-770,共13页
Acta Automatica Sinica
基金
国家自然科学基金(62002373,61872446,71971212,U19B2024)资助。
关键词
多模态知识图谱
实体对齐
预训练模型
特征融合
Multi-modal knowledge graph
entity alignment
pre-trained model
feature fusion
