modality-discriminator

发表于 2022-10-20 更新于 2022-10-21 分类于 Paper ， MMKG 阅读次数：本文字数： 3k 阅读时长 ≈ 3 分钟

COLING 2022，代码。

作者认为，不是所有的social media post都需要多模态信息，可能有的post更适合单模态模型，如果加入多模态信息反而可能造成错误的后果。因此，作者基于强化学习，提出了一种可以把social post分为单模态集合和多模态集合的方法。

Recently, multimodal information extraction from social media posts has gained increasing attention in the natural language processing community. Despite their success, current approaches overestimate the significance of images. In this paper, we argue that different social media posts should consider different modalities for multimodal information extraction. Multimodal models cannot always outperform unimodal models. Some posts are more suitable for the multimodal model, while others are more suitable for the unimodal model. Therefore, we propose a general data splitting strategy to divide the social media posts into two sets so that these two sets can achieve better performance under the information extraction models of the corresponding modalities. Specifically, for an information extraction task, we first propose a data discriminator that divides social media posts into a multimodal and a unimodal set. Then we feed these sets into the corresponding models. Finally, we combine the results of these two models to obtain the final extraction results. Due to the lack of explicit knowledge, we use reinforcement learning to train the data discriminator. Experiments on two different multimodal information extraction tasks demonstrate the effectiveness of our method. The source code of this paper can be found in https://github.com/xubodhu/RDS.

1 Introduction

问题：不是所有的social post都适合于多模态信息抽取方法。比如下图：

对于(a)和(c)来说，加入图像信息能够辅助信息抽取，比如我们可以知道[Kolo MISC]是一条狗，而不是一个人；从图像中两个人拉着手，可以判断出Meghan Markle和Prince Harry是夫妻而不是同事。

但是对于(b)和(d)来说，加入图像信息反而可能导致错误判断。在(b)里可能因为看到图像中有个人像，然后判断[Nasa ORG]是个人；在(d)里因为看到很多人类似的服装，错误判断出Angel和Jesennia Rodriguez是同事而不是夫妻。

方法：作者期望能够设计一种把不同social post分类为适用于单模态和适用于多模态的方法。由于不存在这样的数据集，因此作者提出了一种基于强化学习的data discriminator。

2 Method

首先，作者会把训练集随机分为\(D_{model}\) (80%)和\(D_{split}\) (20%)。

\(D_{model}\)用来训练单模态和多模态模型，这两种模型可以是任意的方法，训练完毕之后，就freeze所有参数。

\(D_{split}\)是用来训练Data discriminator，Data discriminator会判断某个输入样本是否适用于多模态方法，根据Data discriminator的判断结果，检测单模态和多模态模型在判断的单模态集合和多模态集合上的表现效果。如果在划分的多模态集合中，多模态模型效果比单模态模型效果越好；在划分的单模态集合中，单模态模型效果比多模态模型效果越好，就证明这个Data discriminator的判断效果越准确。