Novel carbon-based nc-MoC/a-C(Al) nanocomposite coating was fabricated by co-incorporation of metallic molybdenum and aluminum via multi-functional magnetron sputtering system. The composition and structure of as-fabricated coating were investigated by X-ray photoelectron spectrometer (XPS), X-ray diffraction (XRD) and high-resolution transmission electron microscope (HRTEM). It confirmed that the nc-MoC/a-C(Al) coating is composed of 72.0 at.% C, 17.3 at.% Mo, 6.7 at.% Al and 4.0 at.% O, respectively, and it presents a typical nanocomposite microstructure where Mo carbide nanoparticles are uniformly dispersed in the a-C matrix. Nano-indenter, interferometer and ball-on-disk tribometer were carried out to evaluate the hardness, internal stress and tribological properties. The results showed that the carbon-based nc-MoC/a-C(Al) nanocomposite coating owns superior combined mechanical properties with relatively low internal stress and high hardness as well as strong adhesion strength. In particular, this novel carbon-based coating can achieve low-friction and anti-wear performances with a friction coefficient of 0.06 and wear rate of 2.1 × 10^− 16 m³/N·m, indicating that it has a great potential in engineering applications.