Fitting conics from images is a preliminary step for its plentiful applications. It is a common sense that geometric distance based fitting methods are better than algebraic distance based ones. However, for a long time, there has not been a geometric distance between a point and a general conic that allows easy computation and achieves high accuracy simultaneously. Though Sampson distance is widely accepted, it is only a first-order approximation. For other geometric distances, the computations are too complex to be popular in practice. In this paper, we derive a new geometric distance between a point and a general conic, called Polar-N-Direction distance. The distance can be adapted to a projective transformation because it is computed along the normal direction of the polar line of the point, making conic fitting more robust. Moreover, Polar-N-Direction distance is accurate and simultaneously still analytical in an explicit representation, which is quite easy to be implemented. Then, based on the distance, a new cost function is constructed. The conic fitting optimization by minimizing this cost function has all the merits of the geometric distance based methods and simultaneously avoids their limitations. Experiments show that the conic fitting method is greatly efficient.