JTAV: Jointly learning social media content representation by fusing textual, acoustic, and visual features

Hongru Liang; Haozheng Wang; Jun Wang; Shaodi You; Zhe Sun; Jin Mao Wei; Zhenglu Yang

JTAV: Jointly learning social media content representation by fusing textual, acoustic, and visual features

Hongru Liang, Haozheng Wang, Jun Wang, Shaodi You, Zhe Sun, Jin Mao Wei, Zhenglu Yang^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Citations (Scopus)

Abstract

Learning social media content is the basis of many real-world applications, including information retrieval and recommendation systems, among others. In contrast with previous works that focus mainly on single modal or bi-modal learning, we propose to learn social media content by fusing jointly textual, acoustic, and visual information (JTAV). Effective strategies are proposed to extract fine-grained features of each modality, that is, attBiGRU and DCRNN. We also introduce cross-modal fusion and attentive pooling techniques to integrate multi-modal information comprehensively. Extensive experimental evaluation conducted on real-world datasets demonstrates our proposed model outperforms the state-of-the-art approaches by a large margin.

Original language	English
Title of host publication	COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings
Editors	Emily M. Bender, Leon Derczynski, Pierre Isabelle
Publisher	Association for Computational Linguistics (ACL)
Pages	1269-1280
Number of pages	12
ISBN (Electronic)	9781948087506
Publication status	Published - 2018
Event	27th International Conference on Computational Linguistics, COLING 2018 - Santa Fe, United States Duration: 20 Aug 2018 → 26 Aug 2018

Publication series

Name	COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings

Conference

Conference	27th International Conference on Computational Linguistics, COLING 2018
Country/Territory	United States
City	Santa Fe
Period	20/08/18 → 26/08/18

Cite this

Liang, H., Wang, H., Wang, J., You, S., Sun, Z., Wei, J. M., & Yang, Z. (2018). JTAV: Jointly learning social media content representation by fusing textual, acoustic, and visual features. In E. M. Bender, L. Derczynski, & P. Isabelle (Eds.), COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings (pp. 1269-1280). (COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings). Association for Computational Linguistics (ACL).

Liang, Hongru ; Wang, Haozheng ; Wang, Jun et al. / JTAV : Jointly learning social media content representation by fusing textual, acoustic, and visual features. COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings. editor / Emily M. Bender ; Leon Derczynski ; Pierre Isabelle. Association for Computational Linguistics (ACL), 2018. pp. 1269-1280 (COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings).

@inproceedings{97ed85a0fcb74ba8968be9cd6b8991e2,

title = "JTAV: Jointly learning social media content representation by fusing textual, acoustic, and visual features",

abstract = "Learning social media content is the basis of many real-world applications, including information retrieval and recommendation systems, among others. In contrast with previous works that focus mainly on single modal or bi-modal learning, we propose to learn social media content by fusing jointly textual, acoustic, and visual information (JTAV). Effective strategies are proposed to extract fine-grained features of each modality, that is, attBiGRU and DCRNN. We also introduce cross-modal fusion and attentive pooling techniques to integrate multi-modal information comprehensively. Extensive experimental evaluation conducted on real-world datasets demonstrates our proposed model outperforms the state-of-the-art approaches by a large margin.",

author = "Hongru Liang and Haozheng Wang and Jun Wang and Shaodi You and Zhe Sun and Wei, {Jin Mao} and Zhenglu Yang",

note = "Publisher Copyright: {\textcopyright} 2018 COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings. All rights reserved.; 27th International Conference on Computational Linguistics, COLING 2018 ; Conference date: 20-08-2018 Through 26-08-2018",

year = "2018",

language = "English",

series = "COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1269--1280",

editor = "Bender, {Emily M.} and Leon Derczynski and Pierre Isabelle",

booktitle = "COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings",

address = "United States",

}

Liang, H, Wang, H, Wang, J, You, S, Sun, Z, Wei, JM & Yang, Z 2018, JTAV: Jointly learning social media content representation by fusing textual, acoustic, and visual features. in EM Bender, L Derczynski & P Isabelle (eds), COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings. COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings, Association for Computational Linguistics (ACL), pp. 1269-1280, 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, United States, 20/08/18.

JTAV: Jointly learning social media content representation by fusing textual, acoustic, and visual features. / Liang, Hongru; Wang, Haozheng; Wang, Jun et al.
COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings. ed. / Emily M. Bender; Leon Derczynski; Pierre Isabelle. Association for Computational Linguistics (ACL), 2018. p. 1269-1280 (COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - JTAV

T2 - 27th International Conference on Computational Linguistics, COLING 2018

AU - Liang, Hongru

AU - Wang, Haozheng

AU - Wang, Jun

AU - You, Shaodi

AU - Sun, Zhe

AU - Wei, Jin Mao

AU - Yang, Zhenglu

PY - 2018

Y1 - 2018

N2 - Learning social media content is the basis of many real-world applications, including information retrieval and recommendation systems, among others. In contrast with previous works that focus mainly on single modal or bi-modal learning, we propose to learn social media content by fusing jointly textual, acoustic, and visual information (JTAV). Effective strategies are proposed to extract fine-grained features of each modality, that is, attBiGRU and DCRNN. We also introduce cross-modal fusion and attentive pooling techniques to integrate multi-modal information comprehensively. Extensive experimental evaluation conducted on real-world datasets demonstrates our proposed model outperforms the state-of-the-art approaches by a large margin.

AB - Learning social media content is the basis of many real-world applications, including information retrieval and recommendation systems, among others. In contrast with previous works that focus mainly on single modal or bi-modal learning, we propose to learn social media content by fusing jointly textual, acoustic, and visual information (JTAV). Effective strategies are proposed to extract fine-grained features of each modality, that is, attBiGRU and DCRNN. We also introduce cross-modal fusion and attentive pooling techniques to integrate multi-modal information comprehensively. Extensive experimental evaluation conducted on real-world datasets demonstrates our proposed model outperforms the state-of-the-art approaches by a large margin.

UR - http://www.scopus.com/inward/record.url?scp=85119407421&partnerID=8YFLogxK

M3 - Conference contribution

T3 - COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings

SP - 1269

EP - 1280

BT - COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings

A2 - Bender, Emily M.

A2 - Derczynski, Leon

A2 - Isabelle, Pierre

PB - Association for Computational Linguistics (ACL)

Y2 - 20 August 2018 through 26 August 2018

ER -

Liang H, Wang H, Wang J, You S, Sun Z, Wei JM et al. JTAV: Jointly learning social media content representation by fusing textual, acoustic, and visual features. In Bender EM, Derczynski L, Isabelle P, editors, COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings. Association for Computational Linguistics (ACL). 2018. p. 1269-1280. (COLING 2018 - 27th International Conference on Computational Linguistics, Proceedings).

JTAV: Jointly learning social media content representation by fusing textual, acoustic, and visual features

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this