Abstract: Online action detection plays very important role in many applications such as security and human-computer interaction. This mission requires that the system can detect the action when it just started, instead of waiting for the entire action comes to an end. Since in online action detection problem models can only make judgments based on the observed part of the video, so compared to other tasks such as action recognition and action detection, the model needs to dig out more from history information to assist decision-making for current frame. Based on the long short-term memory (LSTM) model commonly used in online action detection problems, this paper constructs a two-stream LSTM model called 2S-LSTM, and introduces the self-supervised learning idea, which is widely used in the image field, into the online action detection problem. First, the two-stream network 2S-LSTM model uses LSTM to model the temporal information of RGB flow and optical flow respectively. Moreover, based on the idea of self-supervised learning we construct two new loss functions:temporal similarity loss and optical flow stability loss for training. Experiments show that, compared with the past online motion detection methods such as RED, TRN, and IDN, our model in has achieved better results on both the TVSeries and THUMOS’14 datasets.

Key words: self-supervised learning, two-stream LSTM networks(2S-LSTM), online action detection, temporal similarity loss, optical flow stability loss