Recent studies on visual neural encoding and decoding have made significant progress,benefiting from the latest advances in deep neural networks having powerful representations. However,two challenges remain. First,the current decoding algorithms based on deep generative models always struggle with information losses,which may cause blurry reconstruction. Second,most studies model the neural encoding and decoding processes separately,neglecting the inherent dual relationship between the two tasks. In this paper,we propose a novel neural encoding and decoding method with a two-stage flow-based invertible generative model to tackle the above issues. First,a convolutional auto-encoder is trained to bridge the stimuli space and the feature space. Second,an adversarial cross-modal normalizing flow is trained to build up a bijective transformation between image features and neural signals,with local and global constraints imposed on the latent space to render cross-modal alignment. The method eventually achieves bi-directional generation of visual stimuli and neural responses with a combination of the flow-based generator and the auto-encoder. The flow-based invertible generative model can minimize information losses and unify neural encoding and decoding into a single framework. Experimental results on different neural signals containing spike signals and functional magnetic resonance imaging demonstrate that our model achieves the best comprehensive performance among the comparison models.