Learning a dual-language vector space for domain-specific cross-lingual question retrieval

Guibin Chen, Chunyang Chen, Zhenchang Xing, Bowen Xu

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

43 Citations (Scopus)

Abstract

The lingual barrier limits the ability of millions of nonEnglish speaking developers to make effective use of the tremendous knowledge in Stack Overow, which is archived in English. For cross-lingual question retrieval, one may use translation-based methods that first translate the nonEnglish queries into English and then perform monolingual question retrieval in English. However, translation-based methods suffer from semantic deviation due to inappropriate translation, especially for domain-specific terms, and lexical gap between queries and questions that share few words in common. To overcome the above issues, we propose a novel cross-lingual question retrieval based on word embeddings and convolutional neural network (CNN) which are the state-of-the-art deep learning techniques to capture wordand sentence-level semantics. The CNN model is trained with large amounts of examples from Stack Overow duplicate questions and their corresponding translation by machine, which guides the CNN to learn to capture informative word and sentence features to recognize and quantify semantic similarity in the presence of semantic deviations and lexical gaps. A uniqueness of our approach is that the trained CNN can map documents in two languages (e.g., Chinese queries and English questions) in a dual-language vector space, and thus reduce the cross-lingual question retrieval problem to a simple k-nearest neighbors search problem in the dual-language vector space, where no query or question translation is required. Our evaluation shows that our approach significantly outperforms the translation-based method, and can be extended to dual-language documents retrieval from different sources.

Original languageEnglish
Title of host publicationASE 2016 - Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering
EditorsSarfraz Khurshid, David Lo, Sven Apel
PublisherAssociation for Computing Machinery, Inc
Pages744-755
Number of pages12
ISBN (Electronic)9781450338455
DOIs
Publication statusPublished - 25 Aug 2016
Externally publishedYes
Event31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016 - Singapore, Singapore
Duration: 3 Sept 20167 Sept 2016

Publication series

NameASE 2016 - Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

Conference

Conference31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016
Country/TerritorySingapore
CitySingapore
Period3/09/167/09/16

Fingerprint

Dive into the research topics of 'Learning a dual-language vector space for domain-specific cross-lingual question retrieval'. Together they form a unique fingerprint.

Cite this