The optical modulation format conversion scheme of converting star quadrature amplitude modulation signals to four-stage pulse amplitude modulation signals is theoretically realized by combining phase-sensitive amplification in a silicon-p-toluene sulphonate hybrid waveguide. By optimizing the waveguide structure, a phase mismatch bandwidth from 1310 to 1890 nm and an ultra-high nonlinear coefficient of 6.966 × 103 m−1 W−1 are obtained. Constellation plots, error vector magnitude, and bit error rate (BER) are used to evaluate the conversion scheme. Simulation results show that the scheme not only ensures the integrity of the signal information before and after conversion but also achieves nearly 12 dB performance improvement in the converted signal with a BER threshold of 10−3. It is further shown that the scheme has great potential for applications such as long-short distance interconnections and hierarchical modulation of advanced format signals. © 2023 Society of Photo-Optical Instrumentation Engineers (SPIE)