In view of the high computational comp lexity of high efficiency video coding (HEVC) encoding,a fast algorithm based on machine learning is proposed in this paper.According to the smoothness of image content,we divide prediction units (PUs) into th ree classes.The smooth PU has no need to test all the intra prediction modes.Thus,the computational complexity of the algorithm ca n be reduced effectively.First,we calculate the variance of the reference pixels on the left side,the above side of each PU,and the variance of all the reference pixels,as well as the optimal intra prediction mode for each PU.The variances reflect the smoothness of the reference pixels.Then,the machine learning software of Weka is used to classify the obtained data previously,and a decision tree is generated.Finally,according to the decision tree, the intra prediction modes for each PU to be tested are determined,then these i ntra modes are tested for each PU to choose the optimal mode,reducing unnecessary process,thus reducing the encoding complexity. Experimental results show that compared with the standard HEVC 15.0coding algorithm,in the case of high bitrate,the encoding time is reduced by about 16.18% on average with negligible increase of Bjontegaard delta rate (BD-rate) (about 0.25%) and decrease of Bjontegaard delta peak signal-to-noise rate (BD-PS) NR (about 0.02dB).In the case of low bitrate,the encoding time is reduced by about 20.75% on average with negligible increase of BD-rate (about 0.04%) and decrease of BD-PSNR (about 0.00dB).