This might be off on a tangent, But with the increase in memory and processing power I wonder if it would be advantageous to slice the content of a film into Pixel vectors instead of frame vectors, i.e. the data would consist of a pixels worth of data blocks, each containing a vector of that pixel's RGB value from from frame 0 to final frame. Currently machine learning on video data consists of blocks of data containing a frame of pixels, not a 'pixel of frames'