The performance of autonomous robots in varying environments needs to be improved. For such incremental improvement, here we propose an incremental learning framework based on Q-learning and the adaptive kernel linear (AKL) model. The AKL model is used for storing behavioral policies that are learned by Q-learning. Both the structure and parameters of the AKL model can be trained using a novel L2-norm kernel recursive least squares (L2-KRLS) algorithm. AKL model initially without nodes and gradually accumulates content. The proposed framework allows to learn new behaviors without forgetting the previous ones. A novel local -greedy policy is proposed to speed the convergence rate of Q-learning. It calculates the exploration probability of each state for generating and selecting more important training samples. The performance of our incremental learning framework was validated in two experiments. A curve fitting example shows that the L2-KRLS based AKL model is suitable for incremental learning. The second experiment is based on robot learning tasks. The results show that our framework can incrementally learn behaviors in varying environments. Local -greedy policy-based Q-learning is faster than existing Q-learning algorithms.