Document-grounded conversations are designed to generate and engage in conversations based on specific documents or texts provided as context. The ability to incorporate documents into these conversations enables a deeper understanding of the subject matter, fostering more informed and meaningful discussions. However, prior approaches were predominantly rooted in auto-regressive models, overlooking the need for a comprehensive global perspective and the refinement of responses. In this paper, we introduce an innovative Multi-Pass Decoding (MPD) architecture, which iteratively updates background knowledge and enhances responses in document-grounded conversations. During each iteration, it starts by adaptively combining semantics derived from the context, documents, and previous responses. To address the issue of inadequate response quality, we have also developed two modules dedicated to identifying and refining inappropriate words or phrases in responses generated during the previous iteration. Furthermore, MPD is model-agnostic, enabling seamless integration with conventional sequence-to-sequence frameworks. Our empirical experiments on three document-grounded conversation datasets demonstrate that our methods facilitate the production of more contextually accurate and coherent responses.