Guiding the long-short term memory model for image caption generation

Xu Jia; Efstratios Gavves; Basura Fernando; Tinne Tuytelaars

doi:10.1109/ICCV.2015.277

Guiding the long-short term memory model for image caption generation

Xu Jia, Efstratios Gavves, Basura Fernando, Tinne Tuytelaars

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

365 Citations (Scopus)

Abstract

In this work we focus on the problem of image caption generation. We propose an extension of the long short term memory (LSTM) model, which we coin gLSTM for short. In particular, we add semantic information extracted from the image as extra input to each unit of the LSTM block, with the aim of guiding the model towards solutions that are more tightly coupled to the image content. Additionally, we explore different length normalization strategies for beam search to avoid bias towards short sentences. On various benchmark datasets such as Flickr8K, Flickr30K and MS COCO, we obtain results that are on par with or better than the current state-of-the-art.

Original language	English
Title of host publication	2015 International Conference on Computer Vision, ICCV 2015
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	2407-2415
Number of pages	9
ISBN (Electronic)	9781467383912
DOIs	https://doi.org/10.1109/ICCV.2015.277
Publication status	Published - 17 Feb 2015
Event	15th IEEE International Conference on Computer Vision, ICCV 2015 - Santiago, Chile Duration: 11 Dec 2015 → 18 Dec 2015

Publication series

Name	Proceedings of the IEEE International Conference on Computer Vision
Volume	2015 International Conference on Computer Vision, ICCV 2015
ISSN (Print)	1550-5499

Conference

Conference	15th IEEE International Conference on Computer Vision, ICCV 2015
Country/Territory	Chile
City	Santiago
Period	11/12/15 → 18/12/15

Access to Document

10.1109/ICCV.2015.277

Cite this

Jia, X., Gavves, E., Fernando, B., & Tuytelaars, T. (2015). Guiding the long-short term memory model for image caption generation. In 2015 International Conference on Computer Vision, ICCV 2015 (pp. 2407-2415). Article 7410634 (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2015 International Conference on Computer Vision, ICCV 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCV.2015.277

@inproceedings{d954776fbeac4a919f704493bce2ad1b,

title = "Guiding the long-short term memory model for image caption generation",

abstract = "In this work we focus on the problem of image caption generation. We propose an extension of the long short term memory (LSTM) model, which we coin gLSTM for short. In particular, we add semantic information extracted from the image as extra input to each unit of the LSTM block, with the aim of guiding the model towards solutions that are more tightly coupled to the image content. Additionally, we explore different length normalization strategies for beam search to avoid bias towards short sentences. On various benchmark datasets such as Flickr8K, Flickr30K and MS COCO, we obtain results that are on par with or better than the current state-of-the-art.",

author = "Xu Jia and Efstratios Gavves and Basura Fernando and Tinne Tuytelaars",

note = "Publisher Copyright: {\textcopyright} 2015 IEEE.; 15th IEEE International Conference on Computer Vision, ICCV 2015 ; Conference date: 11-12-2015 Through 18-12-2015",

year = "2015",

month = feb,

day = "17",

doi = "10.1109/ICCV.2015.277",

language = "English",

series = "Proceedings of the IEEE International Conference on Computer Vision",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "2407--2415",

booktitle = "2015 International Conference on Computer Vision, ICCV 2015",

address = "United States",

}

Jia, X, Gavves, E, Fernando, B & Tuytelaars, T 2015, Guiding the long-short term memory model for image caption generation. in 2015 International Conference on Computer Vision, ICCV 2015., 7410634, Proceedings of the IEEE International Conference on Computer Vision, vol. 2015 International Conference on Computer Vision, ICCV 2015, Institute of Electrical and Electronics Engineers Inc., pp. 2407-2415, 15th IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, 11/12/15. https://doi.org/10.1109/ICCV.2015.277

Guiding the long-short term memory model for image caption generation. / Jia, Xu; Gavves, Efstratios; Fernando, Basura et al.
2015 International Conference on Computer Vision, ICCV 2015. Institute of Electrical and Electronics Engineers Inc., 2015. p. 2407-2415 7410634 (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2015 International Conference on Computer Vision, ICCV 2015).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Guiding the long-short term memory model for image caption generation

AU - Jia, Xu

AU - Gavves, Efstratios

AU - Fernando, Basura

AU - Tuytelaars, Tinne

PY - 2015/2/17

Y1 - 2015/2/17

N2 - In this work we focus on the problem of image caption generation. We propose an extension of the long short term memory (LSTM) model, which we coin gLSTM for short. In particular, we add semantic information extracted from the image as extra input to each unit of the LSTM block, with the aim of guiding the model towards solutions that are more tightly coupled to the image content. Additionally, we explore different length normalization strategies for beam search to avoid bias towards short sentences. On various benchmark datasets such as Flickr8K, Flickr30K and MS COCO, we obtain results that are on par with or better than the current state-of-the-art.

AB - In this work we focus on the problem of image caption generation. We propose an extension of the long short term memory (LSTM) model, which we coin gLSTM for short. In particular, we add semantic information extracted from the image as extra input to each unit of the LSTM block, with the aim of guiding the model towards solutions that are more tightly coupled to the image content. Additionally, we explore different length normalization strategies for beam search to avoid bias towards short sentences. On various benchmark datasets such as Flickr8K, Flickr30K and MS COCO, we obtain results that are on par with or better than the current state-of-the-art.

UR - http://www.scopus.com/inward/record.url?scp=84973917813&partnerID=8YFLogxK

U2 - 10.1109/ICCV.2015.277

DO - 10.1109/ICCV.2015.277

M3 - Conference contribution

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 2407

EP - 2415

BT - 2015 International Conference on Computer Vision, ICCV 2015

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 15th IEEE International Conference on Computer Vision, ICCV 2015

Y2 - 11 December 2015 through 18 December 2015

ER -

Guiding the long-short term memory model for image caption generation

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this