One of the main task for deep sea submersible is for human-machine collaborative scientific exploration, e.g., human ourselves drive the submersible and monitor cameras around the submersible to observe new species fish or strange topography in a tedious way. In this paper, by defining novel marine animals or any extreme events as novel events, we design a new deep sea novel visual event analysis framework to improve the efficiency of human-machine collaboration and improve the accuracy simultaneously. Specifically, our visual framework concerns diverse functions than most state-of-the-arts, including novel event detection, tracking and summarization. Due to the power and computation resource limitation of the submersible, we design an efficient deep learning based visual saliency method for novel event detection and propose an online object tracking strategy as well. All the experiments are depending on Chinese Jiaolong, the manned deep sea submersible, which mounts several PanCtiltCzoom (PTZ) camera and static cameras. We build a new novel deep sea event dataset and the results justify that our human-machine collaborative visual observation framework can automatically detect, track and summarize the novel deep sea event.