This article presents the experimental investigation on instabilities of thermocapillary- buoyancy convection in the transition process in an open rectangular liquid layer subject to a horizontal temperature gradient. In the experimental run, an infrared thermal imaging system was constructed to observe and record the surface wave of the rectangular liquid layer. It was found that there are distinct convection longitudinal rolls in the flow field in the thermocapillary- buoyancy convection transition process. There are different wave characterizations for liquid layers with different thicknesses. For sufficiently thin layers, oblique hydrothermal waves are observed, which was predicted by the linear-stability analysis of Smith & Davis in 1983. For thicker layers, the surface flow is distinct and intensified, which is because the buoyancy convection plays a dominant role and bulk fluid flow from hot wall to cold wall in the free surface of liquid layers. In addition, the spatiotemporal evolution analysis has been carried out to conclude the rule of the temperature field destabilization in the transition process.