Deep reinforcement learning (DRL) has achieved superhuman performance on Atari games using only raw pixels. However, when applied to complex 3D first-person shooter (FPS) environments, it often faces compound challenges of inefficient exploration, partial observability, and sparse rewards. To address this, we propose the Depth-Detection Augmented Hierarchical Proximal Policy Optimization (D2AH-PPO) method. Specifically, our framework utilizes a two-level hierarchy where the higher-level controller handles option control learning, while the lower-level workers focus on mastering sub-tasks. To boost the learning of sub-tasks, D2AH-PPO involves a combination technique, which includes 1) object-aware representation learning that extracts high-dimensional information representation of crucial components, and 2) a rule-based action mask for safer and more purposeful exploration. We assessed the efficacy of our framework in the 3D FPS game 'ViZDoom'. Extensive experiments indicate that D2AH-PPO significantly enhances exploration and outperforms several baselines.