Rocks are one of the major surface features of Mars. The accurate characterization of the chemical and mineralogical composition of Martian rocks would yield significant evolutionary information about relevant geological processes and exobiological exploration. Many existing rock recognition systems generally assume that all testing classes are known during training. Over real planetary surfaces, the autonomous recognition system is likely to encounter an unknown category of rock that is crucial to the performance of the rock classification task. Therefore, we develop an open-set Martian rock-type classification framework based on their spectral signatures, with the subgoal of new/unknown rock-type recognition and category-incremental learning for expanding the recognition model. First, the spectral signatures of rock samples are captured to characterize their mineralogical compositions and physical properties, which serves as the input of the developed framework. To further produce the highly discriminative feature representation from the original spectral signatures, a Transformer architecture integrated with contrastive learning is constructed and trained in an end-to-end manner to force instances of the same class to remain close-by while pushing those of a dissimilar class farther apart. Following this, according to the extreme value theorem (EVT), category-specific distance distribution analysis is conducted to detect and identify new/unknown types of rock samples due to the isolated characteristics of new/unknown rock samples in the latent feature space. Finally, the recognition model is incrementally updated to learn these identified "unknown" samples without forgetting previously known categories when the associated labels are progressively obtained. The multispectral camera, a duplicated payload of the counterpart onboard the Zhurong rover, is used as the multispectral sensor for capturing the spectral information of the laboratory rock dataset shared by the National Mineral Rock and Fossil Specimens Resource Center for both qualitative and quantitative evaluation. Experimental results indicate the effectiveness and robustness of the developed in situ analysis framework. IEEE