The tumor microenvironment (TME) is the internal environment in which tumors develop and consists of tumor cells, various immune cells, and interstitial cells. Understanding TME can help predict clinical response of immunotherapy and offer guidance for therapeutic optimization. Current pathological practice utilizes multiplexed immunohistochemistry (mIHC) to make assessment of different types of cell distribution in TME. However, these staining methods are not only costly but  can also reduce the quality of the sample tissues and staining results often require professional pathologists to interpret, which can be possibly influenced by subjectivity. In this work, we propose a computational prediction method for cell distribution in TME using a modified U-Net structure, which can learn useful features from the hematoxylin and eosin (H&E) images and predict PanCK positive colon cells and tumor infiltrating lymphocytes (TILs) at cellular-level. Our created datasets contain H&E images and cellular-level segmentation labels annotated by board-certified pathologists according to the corresponding registered mIHC images. We combined U-Net and Inception block structure and created a modified U-net network which can predict PanCK positive colon cells and TILs distribution in TME,with the accuracy of 84.6% on test set. Hence, this method shows the potential to make assessment of different types of cell distribution in TME more objectively and efficiently.