Five kinds of carboxyl-functionalized quaternary ammonium ionic liquids (ILs), tricaprylmethylammonium succinate ([A336]Suc), tricaprylmethylammonium aspartate ([A336]Asp), tricaprylmethylammonium glutamate ([A336]Glu), tricaprylmethylammonium trifluoroacetate ([A336]TFA), and tricaprylmethylammonium phthalate ([A336]Pha), with strong hydrophobicity and hydrogen bond basicity were used to separate methacrylic acid (MAA) and acetic acid (HAc) from an aqueous solution. [A336]Suc shows better extraction performance than the other ILs because of the stronger hydrogen bonding basicity. Even though HAc shows the antagonistic effect on the extraction of MAA, the selectivity of MAA to HAc is achieved 54.70 for [A336]Suc at the optimized extraction conditions. The molecular mechanism of the extraction by IL is revealed by combining FT-IR and quantum chemical calculations. The results indicate that multiple hydrogen bonds are presented in the IL-acid complex which plays an important role in the acid extraction. Meanwhile, the computational studies demonstrate that the preferential extraction of MAA to HAc by [A336]Suc originated from the difference in the strength of the hydrogen bonding interaction between carboxylate group of ILs and acid.