Close-orbit low-mass X-ray binaries (LMXBs), radio binary millisecond pulsars (BMSPs) with extremely low-mass helium white dwarfs (ELM He WDs) and ultra-compact X-ray binaries (UCXBs) are all part of the same evolutionary sequence. It is therefore of uttermost importance to understand how these populations evolve from one specie to another. Moreover, UCXBs are important gravitational wave (GW) sources and can be detected by future space-borne GW observatories. However, the formation and evolutionary link between these three different populations of neutron star (NS) binaries are not fully understood. In particular, a peculiar fine-tuning problem has previously been demonstrated for the formation of these systems. In this investigation, we test a newly suggested magnetic braking prescription and model the formation and evolution of LMXB5. We compute a grid of binary evolution models and present the initial parameter space of the progenitor binaries which successfully evolve all the way to produce UCXBs. We find that the initial orbital period range of LMXB5, which evolve into detached NS + ELM He WD binaries and later UCXBs, becomes significantly wider compared to evolution with a standard magnetic braking prescription, and thus helps to relieve the fine-tuning problem. However, we also find that formation of wide-orbit BMSPs is prohibited for strong versions of this new magnetic braking prescription, which therefore calls for a revision of the prescription. Finally, we present examples of the properties of UCXBs as Galactic GW sources and discuss their detection by the LISA, TianQin, and Taiji observatories.