Fractional-order PDEs based modeling method is emerging as a powerful and effective tool in many areas, including image processing, multi-physics, finance, etc. Since medical image quality is usually affected by different kinds of inevitable noises introduced in imaging procedure, which makes it difficult to carry out the further image segmentation and analysis tasks. To deal with this problem, we develop a new algorithm based on fractional operators of variable-order in order to enhance the poor image quality. A mask optimization method for selecting the fractional order adaptively is applied to construct a variable-order fractional differential mask, and the coefficients of the mask are generated from three different popular high-order discrete formulas. We carry out experiments on OCT thoracic aorta images and some nature images with low contrast and noise, demonstrating that the high-order discrete method leads to significantly better performance in enhancing the edge information nonlinearly compared to the standard first-order discrete method.