Fairness perceptions may be affected by counterfactual comparisons. Although certain studies using a two player ultimatum game (UG) have shown that comparison with the proposers influences the responders' fairness perceptions in a gain context, the effect of counterfactual comparison in a UG with multiple responders or proposers remains unclear, especially in a loss context. To resolve these issues, this study used a modified three-player UG with multiple responders in Experiment 1 and multiple proposers in Experiment 2 to examine the influence of counterfactual comparison on fairness-related decision-making in gain and loss contexts. The two experiments consistently showed that regardless of the gain or loss context, the level of inequality of the offer and counterfactual comparison influenced acceptance rates (ARs), response times (RTs), and fairness ratings (FRs). If the offers that were received were better than the counterfactual offers, unequal offers were more likely to be accepted than equal offers, and participants were more likely to report higher FRs and to make decisions more quickly. In contrast, when the offers they received were worse than the counterfactual offers, participants were more likely to reject unequal offers than equal offers, reported lower FRs, and made decisions more slowly. These results demonstrate that responders' fairness perceptions are influenced by not only comparisons of the absolute amount of money that they would receive but also specific counterfactuals from other proposers or responders. These findings improve our understanding of fairness perceptions.