Guided open vocabulary image captioning with constrained beam search

Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould

    Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

    134 Citations (Scopus)

    Abstract

    Existing image captioning models do not generalize well to out-of-domain images containing novel scenes or objects. This limitation severely hinders the use of these models in real world applications dealing with images in the wild. We address this problem using a flexible approach that enables existing deep captioning architectures to take advantage of image taggers at test time, without re-training. Our method uses constrained beam search to force the inclusion of selected tag words in the output, and fixed, pretrained word embeddings to facilitate vocabulary expansion to previously unseen tag words. Using this approach we achieve state of the art results for out-of-domain captioning on MSCOCO (and improved results for in-domain captioning). Perhaps surprisingly, our results significantly outperform approaches that incorporate the same tag predictions into the learning algorithm. We also show that we can significantly improve the quality of generated ImageNet captions by leveraging ground-truth labels.

    Original languageEnglish
    Title of host publicationEMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings
    PublisherAssociation for Computational Linguistics (ACL)
    Pages936-945
    Number of pages10
    ISBN (Electronic)9781945626838
    DOIs
    Publication statusPublished - 2017
    Event2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017 - Copenhagen, Denmark
    Duration: 9 Sept 201711 Sept 2017

    Publication series

    NameEMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings

    Conference

    Conference2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017
    Country/TerritoryDenmark
    CityCopenhagen
    Period9/09/1711/09/17

    Fingerprint

    Dive into the research topics of 'Guided open vocabulary image captioning with constrained beam search'. Together they form a unique fingerprint.

    Cite this