All-fiber integrated phase shifters and optical switches have important applications in photonic devices, such as optical controlling, optical fiber sensing, and signal processing. In this Letter, for the first time to the best of our knowledge, we integrated the photothermal effect of a nanomaterial based on gold nanorods (GNRs) and a microfiber interferometer to realize a compact all-optical fiber phase shifter. GNRs surrounding the microfiber were excited by near-infrared light via the evanescent interaction, subsequently releasing the heat through the photothermal effect. Then, the refractive index around the microfiber was varied to shift the interference dips in a reversible manner. Experimentally, a spectral shift efficiency of 0.16 nm/mW near the wavelength of 1550 nm was obtained using an excitation laser at the wavelength of 808 nm. The device also provided an all-optical switching with the modulation depth of 76.4%. The proposed GNR-based all-fiber device can provide high potentials in all-optical signal control applications. © 2019 Optical Society of America.