To overcome the disadvantages of traditional artificial fish swarm algorithm (AF SA),an artificial fish swarm algorithm based on survival behavior (SBAFSA) is proposed. A 12×40Gbit/s wavelength divis ion multiplexing (WDM) system is established by the coupled nonlinear Schrodinger equation.Choosing degree of polarization (DOP) as the feed back signal,SBAFSA is introduced into PMD compensation in WDM system using the strategy of compensating the worst channel. The simulation results show that the DOP value of every compensated channel is above 0.96after compensation ,eye diagram is obviously improved,and SBAFSA can compensate the small change of PMD,where the tracking co mpensation time is less than 0.452ms.