Enhanced OCR Techniques for Recognizing Mathematical Expressions in Scanned Documents

Yahya Baher  Al-Askary; Saad  Al-Momen

doi:10.30526/38.4.3640

Authors

Yahya Baher Al-Askary Department of Mathematics, College of Sciences, University of Baghdad, Baghdad, Iraq https://orcid.org/0009-0002-6783-9102
Saad Al-Momen Department of Mathematics, College of Science, University of Baghdad, Baghdad, Iraq https://orcid.org/0000-0002-9715-4529

DOI:

https://doi.org/10.30526/38.4.3640

Keywords:

Classification, Mathematical Expression, Neural Network, Character Recognition, Image Processing

Abstract

When OCR systems are utilized to recognize mathematical expressions in scanned documents, they encounter numerous challenges. These challenges arise because the mathematical expressions use an extensive array of symbols, variables, numbers, and operations, each with its distinct writing style. Moreover, mathematical expressions have a hierarchical relationship and adhere to logical rules governing the grouping of their components. This paper presents a three-step approach segmentation, symbol recognition, and interpretation to overcome these issues. The segmentation process aims to identify and separate each symbol based on its spatial location. Then, various features are extracted to describe the horizontal and vertical projections of the symbol, enabling effective recognition. Two neural network-based classification methods were proposed for the recognition of the symbols. The first one achieved 96.6% recognition rates, while the second achieved 97.7%. Lastly, the paper introduces three guidelines for interpreting the mathematical meaning of the expression and accurately converting it into textual form. The study demonstrates the potential of these methods in enhancing the capabilities of OCR systems for recognizing mathematical expressions.

Author Biographies

Yahya Baher Al-Askary, Department of Mathematics, College of Sciences, University of Baghdad, Baghdad, Iraq

Department of Mathematics, College of Sciences, University of Baghdad, Baghdad, Iraq
Saad Al-Momen, Department of Mathematics, College of Science, University of Baghdad, Baghdad, Iraq

math

References

1. Memon J, Sami M, Khan RA. Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR). IEEE Access. 2020;8:142642-68. https://doi.org/10.1109/ACCESS.2020.3012542

2. Sakila A, Vijayarani S. Mathematical Symbol Extraction From Document Images A Comprehensive Review. Int J Sci Technol Res. 2020;9(3):648-51. https://doi.org/10.31142/ijstr.v9.i03.2020.25

3. Wang H, Shan G. Recognizing handwritten mathematical expressions as LaTex sequences using a multiscale robust neural network. arXiv:2003.00817. 2020. https://doi.org/10.48550/arXiv.2003.00817

4. Awal A-M, Mouchère H, Viard-Gaudin C. The problem of handwritten mathematical expression recognition evaluation. In: 2010 12th International Conference on Frontiers in Handwriting Recognition; 2010 Sep 19-22; Kolkata, India. IEEE; 2010: 646-51. https://doi.org/10.1109/ICFHR.2010.106

5. Awal AM, Mouchère H, Viard-Gaudin C. Towards Handwritten Mathematical Expression Recognition. In: International Conference on Document Analysis and Recognition; 2009 Jul 26-29; Barcelona, Spain. IEEE; 2009. https://doi.org/10.1109/ICDAR.2009.285

6. Baldominos A, Saez Y, Isasi P. A Survey of Handwritten Character Recognition with MNIST and EMNIST. Appl Sci. 2019;9(15):3169. https://doi.org/10.3390/app9153169

7. Aggarwal R, Pandey S, Tiwari A, Harit G. Survey of mathematical expression recognition for printed and handwritten documents. IETE Tech Rev. 2022;39(6):1245-53. https://doi.org/10.1080/02564602.2021.2008277

8. Huang J, Tan J, Bi N. Overview of Mathematical Expression Recognition. In: Pattern Recognition and Artificial Intelligence, International Conference, ICPRAI; 2020 Oct 17-21; Zhongshan, China. IEEE; 2020. https://doi.org/10.1109/ICPRAI51034.2020.9307062

9. Ogwok D, Ehlers EM. Detecting, Contextualizing and Computing Basic Mathematical Equations from Noisy Images using Machine Learning. In: 2020 The 3rd International Conference on Computational Intelligence and Intelligent Systems; 2020 Nov 20-22; Online. ACM; 2020. p. 8-14. https://doi.org/10.1145/3440787.3440798

10. Hossain MA, Ali MM. Recognition of handwritten digit using convolutional neural network (CNN). Glob J Comput Sci Technol. 2019;19(2). https://doi.org/10.34257/GJCSTA.190204

11. Alvaro F, Sanchez J-A, Bened J-M. An integrated grammar-based approach for mathematical expression recognition. Pattern Recognit. 2016;51:135-47. https://doi.org/10.1016/j.patcog.2015.09.006

12. Gunawan TS, Noor AF, Kartiwi M. Development of English Handwritten Recognition Using Deep Neural Network. Indones J Electr Eng Comput Sci. 2018;10(2):562-8. https://doi.org/10.11591/ijeecs.v10.i2.pp562-568

13. Veres O, Rishnyak I, Rishniak H. Application of Methods of Machine Learning for the Recognition of Mathematical Expressions. In: COLINS; 2019 Apr 24-25; Lviv, Ukraine. Cham: Springer; 2019. p. 378-89. https://doi.org/10.1007/978-3-030-33671-1_31

14. Nazemi A, Tavakolian N, Fitzpatrick D, Fernando C, Suen CY. Offline handwritten mathematical symbol recognition utilising deep learning. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); 2019 Jun 16-17; Long Beach, CA, USA. IEEE; 2019. https://doi.org/10.48550/arXiv.1910.07395

15. Chan C. Stroke extraction for offline handwritten mathematical expression recognition. IEEE Access. 2020;8:61565-75. https://doi.org/10.1109/ACCESS.2020.2984180

16. Yan Z, Zhang X, Gao L, Ke Y, Tang Z. ConvMath: a convolutional sequence network for mathematical expression recognition. In: 2020 25th International Conference on Pattern Recognition (ICPR); 2021 Jan 10-15; Milan, Italy. IEEE; 2021:4566-72. https://doi.org/10.1109/ICPR48806.2021.9413203

17. Truong T-N, Nguyen HT, Nguyen CT, Nakagawa M. Learning symbol relation tree for online mathematical expression recognition [Preprint]. arXiv:2105.06084. 2021 [cited 2025 Jun 19]. https://doi.org/10.48550/arXiv.2105.06084

18. MacLean S, Labahn G. A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. Int J Doc Anal Recognit (IJDAR). 2013;16:139-63. https://doi.org/10.1007/s10032-012-0183-7

19. Al-Askary YB, Al-Momen S. Tackling Skewness, Noise, and Broken Characters in Mathematical Expression Segmentation. Iraqi J Sci. 2023;64(6):3998-4013. https://doi.org/10.24996/ijs.2023.64.6.38

20. Siddharth KS, Dhir R, Rani R. Handwritten Gurmukhi Numeral Recognitionusing Different Feature Sets. Int J Comput Appl. 2011;28(2):0975-8887. https://doi.org/10.5120/3565-4927

21. Devore JL, Berk KN. Modern Mathematical Statistics with Applications. New York: Springer; 2012. https://doi.org/10.1007/978-1-4614-0391-3

22. Aaref AM. English character recognition algorithm by improving the weights of MLP neural network with dragonfly algorithm. Period Eng Nat Sci. 2021;9(2):616-28. https://doi.org/10.1016/j.jesit.2015.06.003

23. Abdulameer AT. An Improvement of MRI Brain Images Classification Using Dragonfly Algorithm as Trainer of Artificial Neural Network. Ibn Al-Haitham J Pure Appl Sci (IHJPAS). 2018;31(1). https://doi.org/10.30526/31.1.1895

24. Feng Y. Convolutional Neural Network for English Character Recognition. In: CIBDA 2022; 3rd International Conference on Computer Information and Big Data Applications; 2022 Dec 9-11; Wuhan, China. IEEE; 2022. https://doi.org/10.1109/CIBDA56586.2022.00030

25. Chen W, Wang Z, Zhou J. Large-scale L-BFGS using MapReduce. In: Advances in Neural Information Processing Systems 27 (NIPS 2014); 2014 Dec 8-13; Montreal, Canada. Curran Associates Inc.; 2014. https://doi.org/10.5591/978-1-57753-616-9_73

26. Liu DC, Nocedal J. On the limited memory bfgs method for large scale optimization. Math Program. 1989;45:503-28. https://doi.org/10.1007/BF01589116

27. Soujanya B, Sitamahalakshmi T. Optimization with ADAM and RMSprop in Convolution neural Network (CNN): A Case study for Telugu Handwritten Characters. Int J Emerg Trends Eng Res. 2020;8(9). https://doi.org/10.30534/ijeter/2020/38892020

28. De S, Mukherjee A, Ullah E. Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration. In: International Conference on Learning Representations (ICLR); 2018 Apr 30-May 3; Vancouver, BC, Canada. ICLR; 2018. https://doi.org/10.48550/arXiv.1807.06766

29. Dunlop Gr. A Rapid Computational Method for Improvements to Nearest Neighbour Interpolation. Comp & Maths with Appls. 1980;6:349-53. https://doi.org/10.1016/0898-1221(80)90042-5

30. Samadiani N, Hassanpour H. A neural network-based approach for recognizing multi-font printed English characters. J Electr Syst Inf Technol. 2015;2(2):207-18. https://doi.org/10.1016/j.jesit.2015.06.003

31. Mousavi SM, Omid SS, Abu-Bakar SR. Car Plate Segmentation Based on Morphological and Labeling Approach. In: Advances in Computing Control and Telecommunication Technologies; 2011.

32. Gharde SS, Baviskar PV, Adhiya KP. Identification of Handwritten Simple Mathematical Equation Based on SVM and Projection Histogram. Int JSoft Comp Eng (IJSCE), 2013; 3(2).