Several studies employing the visual world paradigm have shown that people will look at visual objects having a semantic relationship with spoken words. However, it remains unclear how eye movements are controlled in the visual world paradigm. The present study examined how listeners guide their eyes in the visual world paradigm by distinguishing the conceptual overlap hypothesis and the uncertainty reduction hypothesis. The uncertainty of the spoken words and the degree of conceptual overlap between visual objects and spoken words were manipulated by varying the spoken words across different hierarchical levels. The result showed that participants looked at the target objects more often when there was greater conceptual overlap between visual objects and spoken words, suggesting that the degree of conceptual overlap between spoken words and visual objects is the major factor that determines the probability of looking at a target object.