This paper proposes a novel convolutional neural network architecture for semantic segmentation of bridges with various scales in optical remote sensing images. In the context of R-SI analysis on objects with irregular shapes, it is necessary to get dense, pixelwise classification maps. To address the issue, a new network architecture for producing refined shapes is required instead of image categorization labels. In our end-to-end framework, a ResNet is used as a backbone model to extract semantic features, then a cascaded top-down path is added to fuse these features as different scales. Joint features are obtained by stacking different layers of feature maps. Experiments show our proposed architecture has the ability to combine rich multi-scale contextual information to produce semantic segmentation maps with high accuracy.