A Dataset To Evaluate The Representations Learned By Video Prediction Models (Moving Symbols)

Abstract

We present a parameterized synthetic dataset called Moving Symbols to support the objective study of video prediction networks. Using several instantiations of the dataset in which variation is explicitly controlled, we highlight issues in an existing state-of-the-art approach and propose the use of a performance metric with greater semantic meaning to improve experimental interpretability. Our dataset provides canonical test cases that will help the community better understand, and eventually improve, the representations learned by such networks in the future.

[ arXiv | dataset code ]

Acknowledgements

This project was completed as part of an internship at Toyota Research Institute.

Publications

A Dataset To Evaluate The Representations Learned By Video Prediction Models Ryan Szeto, Simon Stent, German Ros, and Jason J. Corso International Conference on Learning Representations (Workshop Track), 2018 [ BibTeX ] @inproceedings{szeto2018dataset, author={Szeto, Ryan and Stent, Simon and Ros, German and Corso, Jason J.}, title={A Dataset To Evaluate The Representations Learned By Video Prediction Models}, booktitle={International Conference on Learning Representations (Workshop Track)}, month={Apr}, year={2018} }