In the automatic optical demultiplexing system,the feedback signal is the key factor to determine the demultiplexing effect.So the selection and correct modeling of feedback signal is very importa nt.In this paper,we analyze the influence of coherence of the two optical carriers in different states of polarization (SOP) on the two common used feedback signals (optical power difference and radio frequency signal power) with automatic optical polari zation demultiplexing scheme in polarization division multiplexed (PDM) system and derive the mathematical models of two type s of feedback signals in detail and complete the analysis of related model.Then we establish a simulation platform of 100Gbi t/ s PDM-DQPSK transmission system using Matlab and verify the effect of the demul tiplexing scheme using the two types of feedback signals.The four feedback signa l maps show that the mathematical models of the two types of feedback signals are co rrect,and no matter the two optical carriers are coherent or not,the automatic demultiplexing scheme using optical power dif ference as feedback signal is effective. However.the automatic demultiplexing scheme using radio frequency signal power as feedback signal can be used only when the two optical carriers are nearly coherent.The eye diagram and the BER curves show that when the two types of feedback signals are effective,their effects on polarization demultiplexing are nearly th e same,and the optical power penalty of the system using the two schemes is near ly 0.1dB.