Fact verification aims to automatically judge the veracity of a claim according to several pieces of evidence. Due to the manual con- struction of datasets, spurious correlations be- tween claim patterns and its veracity (i.e., bi- ases) inevitably exist. Recent studies show that models usually learn such biases instead of understanding the semantic relationship be- tween the claim and evidence. Existing debi- asing works can be roughly divided into data- augmentation-based and weight-regularization- based pipeline, where the former is inflexible and the latter relies on the uncertain output on the training stage. Unlike previous works, we propose a novel method from a counterfactual view, namely CLEVER, which is augmentation- free and mitigates biases on the inference stage. Specifically, we train a claim-evidence fusion model and a claim-only model independently. Then, we obtain the final prediction via sub- tracting output of the claim-only model from output of the claim-evidence fusion model, which counteracts biases in two outputs so that the unbiased part is highlighted. Compre- hensive experiments on several datasets have demonstrated the effectiveness of CLEVER.