Publications

Suggested Citation

Harzig, P., Brehm, S., Lienhart, R., Kaiser, C., & Schallner, R. (2018). Multimodal image captioning for marketing analysis. Proceedings of the IIEEE Conference on Multimedia Information Processing and Retrieval, Miami, FL, United States. https://doi.org/10.48550/arXiv.1802.01958

Year

2018

Authors
P. Harzig,
Prof. Dr. Rainer Lienhart,
S. Brehm,
Dr. Carolin Kaiser
Publication title
Multimodal image captioning for marketing analysis
Publication
Peer-reviewed

Multimodal image captioning for marketing analysis

Abstract:

Automatically captioning images with natural language sentences is an important research topic. State of the art models are able to produce human-like sentences. These models typically describe the depicted scene as a whole and do not target specific objects of interest or emotional relationships between these objects in the image. However, marketing companies require to describe these important attributes of a given scene. In our case, objects of interest are consumer goods, which are usually identifiable by a product logo and are associated with certain brands. From a marketing point of view, it is desirable to also evaluate the emotional context of a trademarked product, i.e., whether it appears in a positive or a negative connotation. We address the problem of finding brands in images and deriving corresponding captions by introducing a modified image captioning network. We also add a third output modality, which simultaneously produces real-valued image ratings. Our network is trained using a classification - aware loss function in order to stimulate the generation of sentences with an emphasis on words identifying the brand of a product. We evaluate our model on a dataset of images depicting interactions between humans and branded products. The introduced network improves mean class accuracy by 24.5 percent. Thanks to adding the third output modality, it also considerably improves the quality of generated captions fo r images depicting branded products. 

Authors

  • P. Harzig, Multimedia Computing and Computer Vision Lab, University of Augsburg, Augsburg, Germany
  • Prof. Dr. Rainer Lienhart, Universität Augsburg
  • S. Brehm, University of Augsburg
  • Dr. Carolin Kaiser, Head of Artificial Intelligence, NIM, carolin.kaiser@nim.org

Contact

Dr. Carolin Kaiser

Head of Artificial Intelligence

Share Publication
Scroll to top