In coherent optical orthogonal frequency division multiplexing (CO-OF DM) communication system,backward propagation (BP) compensation algorithm is put forward to compensate the nonlinear damage.However,in engineering application,the BP comp ensation algorithm has high computational complexity.In this paper, an improved algorithm is proposed to reduce the complexity of BP compensation algorithm.An appropriate threshold power point is found according to the relationship between the nonlinearity and the optical power.When the light power transmitted in fiber drops to threshold powe r point,the effect of nonlinearity in fiber can be ignored and the unnecessary BP computation can be o mitted,thus the computational complexity can be largely reduced.We apply the improved meth od to a 100Gbit/s CO-OFDM system based on VPI simulation platform.In the simulation,the zero disp ersion fiber with 0.2dB/km attenuation factor is used,the optical signal noise ratio (OSNR) of system is 16dB and the transmission distance is 3200km.The simulation r esults indicate that the number of BP compensation steps is decreased from 61to 13,and the Q-factor of the system enhanced by 0.204dB.