Rank Pooling for Action Recognition

Basura Fernando, Efstratios Gavves, M. Jose Oramas, Amir Ghodrati, Tinne Tuytelaars

Research output: Contribution to journalArticlepeer-review

221 Citations (Scopus)

Abstract

We propose a function-based temporal pooling method that captures the latent structure of the video sequence data - e.g., how frame-level features evolve over time in a video. We show how the parameters of a function that has been fit to the video data can serve as a robust new video representation. As a specific example, we learn a pooling function via ranking machines. By learning to rank the frame-level features of a video in chronological order, we obtain a new representation that captures the video-wide temporal dynamics of a video, suitable for action recognition. Other than ranking functions, we explore different parametric models that could also explain the temporal changes in videos. The proposed functional pooling methods, and rank pooling in particular, is easy to interpret and implement, fast to compute and effective in recognizing a wide variety of actions. We evaluate our method on various benchmarks for generic action, fine-grained action and gesture recognition. Results show that rank pooling brings an absolute improvement of 7-10 average pooling baseline. At the same time, rank pooling is compatible with and complementary to several appearance and local motion based methods and features, such as improved trajectories and deep learning features.

Original languageEnglish
Article number7458903
Pages (from-to)773-787
Number of pages15
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume39
Issue number4
DOIs
Publication statusPublished - 1 Apr 2017
Externally publishedYes

Fingerprint

Dive into the research topics of 'Rank Pooling for Action Recognition'. Together they form a unique fingerprint.

Cite this