Temporal-Viewpoint Transportation Plan for Skeletal Few-Shot Action Recognition

Lei Wang; Piotr Koniusz

doi:10.1007/978-3-031-26316-3_19

Temporal-Viewpoint Transportation Plan for Skeletal Few-Shot Action Recognition

Lei Wang, Piotr Koniusz^*

^*Corresponding author for this work

School of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › peer-review

15 Citations (Scopus)

Abstract

We propose a Few-shot Learning pipeline for 3D skeleton-based action recognition by Joint tEmporal and cAmera viewpoiNt alIgnmEnt (JEANIE). To factor out misalignment between query and support sequences of 3D body joints, we propose an advanced variant of Dynamic Time Warping which jointly models each smooth path between the query and support frames to achieve simultaneously the best alignment in the temporal and simulated camera viewpoint spaces for end-to-end learning under the limited few-shot training data. Sequences are encoded with a temporal block encoder based on Simple Spectral Graph Convolution, a lightweight linear Graph Neural Network backbone. We also include a setting with a transformer. Finally, we propose a similarity-based loss which encourages the alignment of sequences of the same class while preventing the alignment of unrelated sequences. We show state-of-the-art results on NTU-60, NTU-120, Kinetics-skeleton and UWA3D Multiview Activity II.

Original language	English
Title of host publication	Computer Vision – ACCV 2022 - 16th Asian Conference on Computer Vision, Proceedings
Editors	Lei Wang, Juergen Gall, Tat-Jun Chin, Imari Sato, Rama Chellappa
Publisher	Springer Science+Business Media B.V.
Pages	307-326
Number of pages	20
Volume	13844
ISBN (Print)	9783031263156
DOIs	https://doi.org/10.1007/978-3-031-26316-3_19
Publication status	Published - 2 Mar 2023
Event	16th Asian Conference on Computer Vision, ACCV 2022 - Macao, China Duration: 4 Dec 2022 → 8 Dec 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13844 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	16th Asian Conference on Computer Vision, ACCV 2022
Country/Territory	China
City	Macao
Period	4/12/22 → 8/12/22

Access to Document

10.1007/978-3-031-26316-3_19

Cite this

Wang, L., & Koniusz, P. (2023). Temporal-Viewpoint Transportation Plan for Skeletal Few-Shot Action Recognition. In L. Wang, J. Gall, T.-J. Chin, I. Sato, & R. Chellappa (Eds.), Computer Vision – ACCV 2022 - 16th Asian Conference on Computer Vision, Proceedings (Vol. 13844, pp. 307-326). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13844 LNCS). Springer Science+Business Media B.V.. https://doi.org/10.1007/978-3-031-26316-3_19

Wang, Lei ; Koniusz, Piotr. / Temporal-Viewpoint Transportation Plan for Skeletal Few-Shot Action Recognition. Computer Vision – ACCV 2022 - 16th Asian Conference on Computer Vision, Proceedings. editor / Lei Wang ; Juergen Gall ; Tat-Jun Chin ; Imari Sato ; Rama Chellappa. Vol. 13844 Springer Science+Business Media B.V., 2023. pp. 307-326 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{e7f47e25256f45a9b8c9e74c73469dd9,

title = "Temporal-Viewpoint Transportation Plan for Skeletal Few-Shot Action Recognition",

abstract = "We propose a Few-shot Learning pipeline for 3D skeleton-based action recognition by Joint tEmporal and cAmera viewpoiNt alIgnmEnt (JEANIE). To factor out misalignment between query and support sequences of 3D body joints, we propose an advanced variant of Dynamic Time Warping which jointly models each smooth path between the query and support frames to achieve simultaneously the best alignment in the temporal and simulated camera viewpoint spaces for end-to-end learning under the limited few-shot training data. Sequences are encoded with a temporal block encoder based on Simple Spectral Graph Convolution, a lightweight linear Graph Neural Network backbone. We also include a setting with a transformer. Finally, we propose a similarity-based loss which encourages the alignment of sequences of the same class while preventing the alignment of unrelated sequences. We show state-of-the-art results on NTU-60, NTU-120, Kinetics-skeleton and UWA3D Multiview Activity II.",

author = "Lei Wang and Piotr Koniusz",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 16th Asian Conference on Computer Vision, ACCV 2022 ; Conference date: 04-12-2022 Through 08-12-2022",

year = "2023",

month = mar,

day = "2",

doi = "10.1007/978-3-031-26316-3\_19",

language = "English",

isbn = "9783031263156",

volume = "13844",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science+Business Media B.V.",

pages = "307--326",

editor = "Lei Wang and Juergen Gall and Tat-Jun Chin and Imari Sato and Rama Chellappa",

booktitle = "Computer Vision – ACCV 2022 - 16th Asian Conference on Computer Vision, Proceedings",

address = "Netherlands",

}

Wang, L & Koniusz, P 2023, Temporal-Viewpoint Transportation Plan for Skeletal Few-Shot Action Recognition. in L Wang, J Gall, T-J Chin, I Sato & R Chellappa (eds), Computer Vision – ACCV 2022 - 16th Asian Conference on Computer Vision, Proceedings. vol. 13844, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13844 LNCS, Springer Science+Business Media B.V., pp. 307-326, 16th Asian Conference on Computer Vision, ACCV 2022, Macao, China, 4/12/22. https://doi.org/10.1007/978-3-031-26316-3_19

Temporal-Viewpoint Transportation Plan for Skeletal Few-Shot Action Recognition. / Wang, Lei; Koniusz, Piotr.
Computer Vision – ACCV 2022 - 16th Asian Conference on Computer Vision, Proceedings. ed. / Lei Wang; Juergen Gall; Tat-Jun Chin; Imari Sato; Rama Chellappa. Vol. 13844 Springer Science+Business Media B.V., 2023. p. 307-326 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13844 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › peer-review

TY - GEN

T1 - Temporal-Viewpoint Transportation Plan for Skeletal Few-Shot Action Recognition

AU - Wang, Lei

AU - Koniusz, Piotr

PY - 2023/3/2

Y1 - 2023/3/2

N2 - We propose a Few-shot Learning pipeline for 3D skeleton-based action recognition by Joint tEmporal and cAmera viewpoiNt alIgnmEnt (JEANIE). To factor out misalignment between query and support sequences of 3D body joints, we propose an advanced variant of Dynamic Time Warping which jointly models each smooth path between the query and support frames to achieve simultaneously the best alignment in the temporal and simulated camera viewpoint spaces for end-to-end learning under the limited few-shot training data. Sequences are encoded with a temporal block encoder based on Simple Spectral Graph Convolution, a lightweight linear Graph Neural Network backbone. We also include a setting with a transformer. Finally, we propose a similarity-based loss which encourages the alignment of sequences of the same class while preventing the alignment of unrelated sequences. We show state-of-the-art results on NTU-60, NTU-120, Kinetics-skeleton and UWA3D Multiview Activity II.

AB - We propose a Few-shot Learning pipeline for 3D skeleton-based action recognition by Joint tEmporal and cAmera viewpoiNt alIgnmEnt (JEANIE). To factor out misalignment between query and support sequences of 3D body joints, we propose an advanced variant of Dynamic Time Warping which jointly models each smooth path between the query and support frames to achieve simultaneously the best alignment in the temporal and simulated camera viewpoint spaces for end-to-end learning under the limited few-shot training data. Sequences are encoded with a temporal block encoder based on Simple Spectral Graph Convolution, a lightweight linear Graph Neural Network backbone. We also include a setting with a transformer. Finally, we propose a similarity-based loss which encourages the alignment of sequences of the same class while preventing the alignment of unrelated sequences. We show state-of-the-art results on NTU-60, NTU-120, Kinetics-skeleton and UWA3D Multiview Activity II.

UR - https://www.scopus.com/pages/publications/85151059345

U2 - 10.1007/978-3-031-26316-3_19

DO - 10.1007/978-3-031-26316-3_19

M3 - Conference Paper

SN - 9783031263156

VL - 13844

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 307

EP - 326

BT - Computer Vision – ACCV 2022 - 16th Asian Conference on Computer Vision, Proceedings

A2 - Wang, Lei

A2 - Gall, Juergen

A2 - Chin, Tat-Jun

A2 - Sato, Imari

A2 - Chellappa, Rama

PB - Springer Science+Business Media B.V.

T2 - 16th Asian Conference on Computer Vision, ACCV 2022

Y2 - 4 December 2022 through 8 December 2022

ER -

Wang L, Koniusz P. Temporal-Viewpoint Transportation Plan for Skeletal Few-Shot Action Recognition. In Wang L, Gall J, Chin TJ, Sato I, Chellappa R, editors, Computer Vision – ACCV 2022 - 16th Asian Conference on Computer Vision, Proceedings. Vol. 13844. Springer Science+Business Media B.V. 2023. p. 307-326. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-26316-3_19

Temporal-Viewpoint Transportation Plan for Skeletal Few-Shot Action Recognition

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Temporal-Viewpoint Transportation Plan for Skeletal Few-Shot Action Recognition

Cite this