In this paper, a new approach is proposed to enhance the handwriting image by using learning-based windowing contrast enhancement and Gaussian Mixture Model (GMM). A fixed size window moves over the handwriting image and two quantitative methods which are discrete entropy (DE) and edge-based contrast measure (EBCM) are used to estimate the quality of each patch. The obtained results are used in the unsupervised learning method by using k-means clustering to assign the quality of handwriting as bad (if it is low contrast) or good (if it is high contrast). After that, if the corresponding patch is estimated as low contrast, a contrast enhancement method is applied to the window to enhance the handwriting. GMM is used as a final step to smoothly exchange information between original and enhanced images to discard the artifacts to represent the final image. The proposed method has been compared with the other contrast enhancement methods for different datasets which are Swedish historical documents, DIBCO2010, DIBCO2012 and DIBCO2013. Results illustrate that proposed method performs well to enhance the handwriting comparing to the existing contrast enhancement methods.