Visual cognitive ability is important for intelligent robots in unstructured and dynamic environments. The high reliance on large amounts of data prevents prior methods to handle this task. Therefore, we propose a model called knowledge-experience fusion graph (KEFG) network for novel inference. It exploits information from both knowledge and experience. With the employment of graph convolutional network (GCN), KEFG generates the predictive classifiers of the novel classes with few labeled samples. Experiments show that KEFG can decrease the training time by the fusion of the source information and also increase the classification accuracy in cross-task learning.