In this paper, we present a partial sparsification scheme for the marginalization of visual inertial odometry (VIO) systems. Sliding window optimization is widely used in VIO systems to guarantee constant complexity by optimizing over a set of recent states and marginalizing out past ones. The marginalization step introduces fill-in between variables incident to the marginalized ones, and most VIO systems discard measurements targeted at active landmark points to maintain sparsity of the marginalized information matrix, at the expense of potential information loss. The scheme is to first retain the dense prior from the marginalization excluding visual measurements, followed by a dense marginalization step that connects landmarks. The dense marginalization prior is then partially sparsified to extract pseudo factors that maintain the overall sparsity while minimizing the information loss. The proposed scheme is tested on public datasets and achieves appreciable results compared with several state-of-the-art approaches. The test also demonstrates that our scheme is applicable to real-time operations.