Sunspot structures can be significantly affected by major flares. In this study, we reported a large penumbral area experiencing two kinds of transformations during the flare SOL2013-11-03T05:22 (M5.0). One penumbral segment decayed and transformed into a small pore when swept by the flare ribbon. At the same time, an adjacent penumbral segment expanded, permeating the granular area along the flaring magnetic polarity inversion line. EUV and X-ray observations indicated that the penumbral enhancement area was close to the flare center, while the penumbral decay area was on the relatively outer side. By tracking the magnetic motions and local magnetic field changes, we found that the magnetic transformations within two regions were totally different during the flare. The central penumbral enhancement area was accompanied by the field collapsing down, whereas the outer penumbral decay area was associated with the field lifting up toward the upper flare center. Particularly, following the uplift motion of the magnetic fields in the outer region, the magnetic flux in the decaying penumbra decreased and that in the forming pore subsequently increased. These results implied that the rearrangement of the magnetic field during the flare would be the action that resulted in the variations of sunspot structures.