Various lineages of ammonia-oxidizing archaea (AOA) are present in deep waters, but the mechanisms that determine ecotype formation are obscure. We studied 18 high-quality genomes of the marine group I AOA lineages (alpha, gamma and delta) from the Mariana and Ogasawara trenches. The genomes of alpha AOA resembled each other, while those of gamma and delta lineages were more divergent and had even undergone insertion of some phage genes. The instability of the gamma and delta AOA genomes could be partially due to the loss of DNA polymerase B (polB) and methyladenine DNA glycosylase (tag) genes responsible for the repair of point mutations. The alpha AOA genomes harbour genes encoding a thrombospondin-like outer membrane structure that probably serves as a barrier to gene flow. Moreover, the gamma and alpha AOA lineages rely on vitamin B-12-independent MetE and B-12-dependent MetH, respectively, for methionine synthesis. The delta AOA genome contains genes involved in uptake of sugar and peptide perhaps for heterotrophic lifestyle. Our study provides insights into co-occurrence of cladogenesis and anagenesis in the formation of AOA ecotypes that perform differently in nitrogen and carbon cycling in dark oceans.