COVID-19 pandemic continues to rampage in the world. Before the achievement of global herd immunity, non-pharmacological interventions(NPIs) are crucial to mitigate the pandemic. Although various NPIs have been put into practice, there are many concerns about the impacts and effectiveness of these NPIs. COVID-19 modelling study (CMS) in epidemiology can provide evidence to solve the aforementioned concerns. It is time-consuming to collect evidence manually when dealing with the vast amount of CMS papers. Accordingly, we seek to accelerate evidence collection by developing an information extraction model to automatically identify evidence from CMS papers. This work presents a novel COVID-19 Non-pharmacological Interventions Evidence (CNPIE) Corpus, which contains 597 abstracts of COVID-19 modelling study with richly annotated entities and relations of the impacts of NPIs. We design a semi-supervised document-level information extraction model (SS-DYGIE++) which can jointly extract entities and relations. Our model outperforms previous baselines in both entity recognition and relation extraction tasks by a large margin. The proposed work can be applied towards automatic evidence extraction in the public health domain for assisting the public health decision-making of the government.