We propose an approach for enhancing pedestrian detection in thermal infrared images using paired visible-thermal images in training. Recently, approaches that retrieve the corresponding visible features from thermal features using a key-value memory network have been proven effective for improving detection results. However, for memory networks storing thermal-visible features, random initialization and end-to-end training may not be ideal, as this can reduce the diversity of memory slots. Also, the retrieved visible features have different reliability as the overall similarities between key slots in the memory network and thermal features differ. These motivate us to propose a DIscrepancy Preserving (DIP) Memory that is updated manually to prevent convergence of key-value memory slots. We also evaluate the reliability of each retrieved visible feature and adjust the training protocol of the detection head. Experiment results on two visible-infrared pedestrian detection datasets demonstrate the superiority of our framework.