摘要
针对基于Transformer框架的图像标注任务中提取视觉特征容易引入噪声问题且为了进一步提高视觉的上下文信息,提出了一种基于综合几何关系稀疏自注意力机制的图像标注方法。首先通过结合图像区域的绝对位置、相对位置和空间包含关系提取详细全面的视觉表示,获取图像中潜在的上下文信息;其次提出了注意力层权重矩阵的稀疏化方法,该方法解决了Transformer忽略图像区域的局部性并引入噪声信息的问题;最后,采用了强化学习方法作为指导策略,实现模型在句子级别优化目标序列。通过在MS-COCO数据集上进行的对比实验结果表明,提出的方法在BLEU1、BLEU4、METEOR、ROUGE-L、CIDEr和SPICE指标上分别比基线模型提升了0.2、0.7、0.1、0.3、1.2和0.4,有效提升了图像自动标注的性能。
Aiming at the problem that extracting visual features in image annotation task based on Transformer framework is easy to introduce noise,and in order to further improve the visual context information,this paper proposed an image captioning method based on comprehensive geometric relationship sparse self attention mechanism.Firstly,it combined the absolute position,relative position and spatial inclusion relationship of the image region to obtain a detailed and comprehensive visual representation so as to obtain the potential context information in the image.Secondly,this paper proposed the sparse method of attention layer weight matrix,which solved the problem that Transformer ignored the locality of image region and introduced noise information.Finally,this paper used the reinforcement learning method as the guidance strategy to optimize the target sequence at the sentence level.The experimental results on MS-COCO dataset show that the proposed method improves the baseline model by 0.2,0.7,0.1,0.3,1.2 and 0.4 respectively in BLEU1,BLEU4,METEOR,ROUGE-L,CIDEr and SPICE,which effectively improves the performance of image automatic captioning.
作者
李艳
金小峰
Li Yan;Jin Xiaofeng(Intelligent Information Processing Laboratory,Yanbian University,Yanji Jilin 133002,China)
出处
《计算机应用研究》
CSCD
北大核心
2022年第4期1132-1136,共5页
Application Research of Computers
基金
延边大学世界一流学科建设培育项目(18YLPY14)
国家社会科学基金重大资助项目(18ZDA306)
延边大学外国语言文学世界一流学科建设攻关科研项目(18YLGG01)。
