Few-shot learning attempts to identify novel categories by exploiting limited labeled training data, while the performances of existing methods still have much room for improvement. Thanks to a very low cost, many recent methods resort to additional unlabeled training data to boost performance, known as semi-supervised few-shot learning (SSFSL). The general idea of SSFSL methods is to first generate pseudo labels for all unlabeled data and then augment the labeled training set with selected pseudo-labeled data. However, almost all previous SSFSL methods only take supervision signal from pseudo-labeling, ignoring that the distribution of training data can also be utilized as an effective unsupervised regularization. In this paper, we propose a simple yet effective SSFSL method named TENET, which takes low-rank feature reconstruction as the unsupervised objective function and pseudo labels as the supervised constraint. We provide several theoretical insights on why TENET can mitigate overfitting on low-quality training data, and why it can enhance the robustness against inaccurate pseudo labels. Extensive experiments on four popular datasets validate the effectiveness of TENET.