A nonlinear equalization algorithm has been proposed through combining fuzzy c-means (FCM) algorithm and K-nearest neighbor (KNN) in coherent optical s ystem with high-order quadrature amplitude modulation (QAM).After performing p h ase recovery at the receiver,a certain number of the training data are tailored by FCM algorithm,and the test data are classified to make decision,which redu ce the computational complexity greatly in subsequent KNN algorithm.The initial centroid and membership grade are firstly calculated for the original training data.After several iterations,the ultimate centroid and membership grade are o btained.The data point is adopted as the tailored training data when its member ship grade is larger than the threshold,and the test point is temporarily class ified according to the distance between it and the centroids.Subsequently,the KNN algorithm is performed to obtain the final classification results.For a 28G baud single carrier single polarization circular 16quadrature amplitude modulat ion (16-QAM) coherent transmission system with 1040km trans mission distance,the numerical simulations have been performed in order to investigate the nonlinear equalization performance of the proposed FCM-KNN algorithm.It is proved by th e simulation results that the FCM-KNN scheme shares the same performance with th e KNN algorithm,and moreover its computational complexity is twenty times less t han that of KNN algorithm,which benefits greatly to the application of coherent optical QAM transmission technology in long distance optical fiber transmission .