In this study, we assess the internal and external validity of Gelfand et al.'s (2011) recently developed measure of cultural tightness-looseness (CTL). Our study is composed of six countries (China, Mexico, Netherlands Russia, Spain, U.S.) with three subsamples (business professionals, K-12 teachers, college students) per country. For these 18 subsamples, confirmatory factor analyses failed to support the unidimensional structure of the 6-item CTL measure. Exploratory factor analyses provided further evidence that the 6-item CTL measure does not have a unidimensional structure across cultures. Additionally, inter-rater agreement analyses did not support the use of aggregated scores to construct country-level scores for the CTL index. We also found that country rankings of CTL scores (in total and for subsamples) were substantively different from those reported by Gelfand et al. (2011). Further country-level correlation analyses yielded mixed support for the external validity of the CTL scores. We conclude with a commentary on the implications of our study for cross-cultural research.