In this study we propose an explicit formulation of the learning behavior of deep neural networks (DNNs) trained with noisy labels in image classification. Specifically, we show theoretically and experimentally that the training process can be divided into two phases: a learning phase in which the outputs of DNNs converge to a hidden noisy label distribution; and a memorization phase in which DNNs start to overfit until the output for each sample converges to its corresponding noisy label. This two-phase formulation enables us to resolve a common pitfall of existing methods for robust training against noisy labels based on the small-loss assumption, namely clean samples have smaller losses than noisy samples in the early training phase. We show that these methods fail when the noise transition matrix is not column diagonally maximal and that this pitfall can be fixed by a simple modification of the small-loss assumption.