摘要
传统的多模态分析首先为语言、图像和其他资源及符号间关系(如图文关系)建构抽象框架,然后用一些实例演示框架的应用。最近,随着大数据分析的多模态转向,多模态研究开始通过大数据分析来测试和验证其理论框架。由于计算模型的开发需要以多模态理论为基础,并使用大量经过分析的文本进行训练,所以本文提出将多模态框架与现有的大数据、云计算、自然语言处理、图像处理、视频处理和语境元数据计算模型进行融合。这种跨领域的融合能有效地改进计算工具和技术,也能将多模态理论用于理解多模态交际的前沿研究之中。为此,我们探索了如何将现有的计算工具和方法集成为一个多模态分析平台(MAP),用于搜索、存储和分析在线媒体中的文本、图像和视频,使结果可视化。在本文中,我们讨论了用大数据方法进行多模态话语分析的优势以及存在的局限性。我们认为要改进大数据分析的计算技术,需要将有关语言、图像、元数据和其他资源的知识整合为符号系统(而不是简单的符号和像素集)((1))。
The multimodal analysis traditionally involves conceptualising abstract frameworks for language,images,and other resources and their intersemiotic relations(e.g.text and image relations)and then demonstrating these frameworks with some examples.This scenario has changed with the recent move towards multimodal approaches to big data analytics which will involve empirically testing and validating multimodal theory and frameworks through the analysis of large data sets.However,large training sets of analysed texts are required to develop computational models based on multimodal theory.Therefore,an alternative approach which involves integrating multimodal frameworks with existing computational models for big data,cloud computing,natural language processing,image processing,video processing,and contextual metadata is proposed.The integration of these disparate fields has the potential to dramatically improve computational tools and techniques,thus placing multimodality at the forefront of research aimed at mapping and understanding multimodal communication.As a step forward in this direction,we explore how existing computational tools and approaches can be integrated into a multimodal analysis platform(MAP)with facilities for searching,storing and analysing text,images and videos in online media,together with dashboards for visualising the results.Preliminary analyses and classifications of text and images about COVID-19 and George Floyd in five online newspapers and Twitter postings show how media patterns can be studied using existing computational tools.The study highlights(a)the benefits and current limitations of big data approach to multimodal discourse analysis and(b)the need to incorporate knowledge about language,images,metadata,and other resources as semiotic systems(rather simply sets of symbols and pixels)to improve computational techniques for big data analytics.
出处
《当代修辞学》
CSSCI
北大核心
2023年第6期47-60,共14页
Contemporary Rhetoric
基金
上海市社科规划青年项目“庭审冲突过程的多模态建构研究”(项目编号:2020EYY005)
上海交通大学文科建设处团队项目(项目编号:WKCX007)。
关键词
多模态分析
新闻媒体
社交媒体
大数据
Multimodal analysis
News media
Social media
Twitter
COVID-19
George Floyd
