Despite the progress in regular scene text spotting, how to detect and recognize irregular text with efficiency and accuracy remains a challenging task. In this work, we propose a novel Corner and Character Assisted Network (CCANet) which exploits pixel-wise semantics to learn explicit text corner and character center positions with low computational cost. Concretely, in the detection stage, we develop a pixel-level Corner Rectification Branch to refine the inaccurately regressed text corners; in the recognition stage, we design another pixellevel Character Enhancement Branch which generates a Gaussian-like character center heatmap to provide attention guidance for the decoding process. To overcome the reliance of character-level annotations, we adopt an iterative approach to generate pseudo-GT label for the character heatmap, which regards the attention peak position of the attention-based recognizer as the true character center. The extensive experiments conducted on two irregular text benchmarks, TotalText and CTW1500, demonstrate that the proposed CCANet achieves competitive and even new state-of-the-art performance.