In the high-speed optical polarization division multiplexed 16quadrature ampli tude modulation (PDM-16-QAM) communication system with coherent detection,the polarization de-multiplexing algorithm is an essential technique that takes car e of separating the information in polarization domain and compensating the Stoc hastic distortion.The widely used method in this area is the classic constant mo dulus algorithm (CMA),but it suffers from the singularity problem that the outpu t converges to the same source.Independent component analysis (ICA) is a new way to solve this problem without singularity problem,but it performs not well in 16-QAM modulation system.Thus we propose the modified ICA (MICA) method to impro ve the convergence performance of ICA.Taking the distance between the compensate d signal and the ideal constellation point as the convergence measurement,the al gorithm adaptively changes the learning rate to optimize the convergence.The per formance of the algorithm is numerically tested in a 20G baud/s PDM-16-QAM si mulation system with coherent detection.The simulation results show that the pro posed algorithm can eliminate the singularity problem with or without polarizati on-dependent loss.Besides,it offers 1dB optical signal-to-noise ratio (OSNR) improvement at the forward error correction (FEC) bit error rate (B ER) threshol d.The convergence measurements show that the proposed algorithm has advantages i n both convergence speed and precision over the other two methods.