This paper proposes a novel image restoration algorithm introducing two types of constrains under the framework of fractal affine transformation.The fractal affine transformation consists of geometrical transformation,isomorphic transformation and luminance transformation.Firstly,some domain blocks are selected and then transformed by the first two fractal affine transformations.Using the resulting patches,the codebook is constructed and served as the new searching scope.Secondly,the detail of the damaged image is extracted using the improved bilateral filter,and then the weight map is obtained according to the extracted image detail.Thirdly,during the luminance transformation,its two parameters are derived through minimizing a constrained energy function between the target patch and each codebook patch.In the constrained energy function,we introduce two types of constraints:one is the weighted consistency constraint between the codebook patch and the target patch over the already known pixels,where the weight patch is obtained from the weight map,and the other is the neighborhood similarity constraint between the codebook patch and the weighted mean of the neighboring patches over the missing pixels.Lastly,the target patch is filled with the estimated patch containing the minimum constrained energy.The experiment results show that compared with the existing congeneric algorithms,the proposed one preserves better continuity of the broken structure and forces the newly filled area to be more consistent with the source area.Therefore,the restored results are improved both subjectively and objectively.