Automatic Text Normalization in Uzbek: Problems, Tools, And Solutions

Аннотация

In recent years, research in the field of Natural Language Processing (NLP) has increased the demand for automated text analysis across multiple languages, including Uzbek. The multi-form, morphologically complex, and stylistically diverse nature of texts written in Uzbek poses certain challenges for automatic analysis. The central focus of this article is the automatic normalization of Uzbek texts—that is, the process of text normalization. It is dedicated to studying the linguistic and technological issues that arise during automatic text normalization in the Uzbek language. Complex morphological structures, polyform words, dialectal variants, Cyrillic-Latin script differences, and non-standard expressions complicate this process. The results of this research contribute to the deeper digital processing of the Uzbek language and to improving the quality of systems for machine translation, speech-to-text conversion, and text analysis. Keywords: Uzbek language, text normalization, natural language processing, artificial intelligence, neural networks, rule-based approach, morphological analysis, BERT, writing systems, linguistic issues. tagging across Uzbek, Turkish, and Uyghur languages, Introduction: Text normalization is the process of discussing the challenges and solutions of converting various writing styles, spelling errors, implementing stemming in agglutinative languages. dialectal expressions, abbreviations, incorrectly written They also showed that a hybrid approach—combining words, and non-standard phrases commonly found on rule-based and statistical methods—improves social media into a standardized, dictionary-compliant effectiveness [2]. form. This process is crucial for the accuracy of subsequent tasks such as text analysis, classification, The UzMorphAnalyser model and software, developed translation, or speech synthesis. In Uzbek, the by Ulug‘bek Salaev (2024), analyzes all possible forms differences between Cyrillic and Latin scripts, phonetic of words in Uzbek. The study compiled a list of all writing practices, dialectal variations, and the existence grammatical affixes in Uzbek and developed analysis of multiple morphological forms of a single word add rules for each. When tested, the model achieved 91% further complexity. accuracy [3], which is a high result for the Uzbek language. This indicates the strong potential of In recent years, several scientific studies have focused morphological normalization tools. on the normalization of Uzbek texts, particularly in terms of orthography, syntax, and lexicon. While there are few dedicated tools for syntactic normalization in Uzbek, the Uzbek-UT treebank M. Sharipov and O. Sobirov (2022), in their article, developed within the Universal Dependencies presented an algorithm for affix separation and lemma framework (Kurbanova N. et al., 2025) serves as a identification in the Uzbek language using a finite-state resource for consistent syntactic annotation. This work automaton. They demonstrated the difference highlights challenges in annotating specific syntactic between stemming and lemmatization with a concrete features of Uzbek—for example, compound verb example, emphasizing the importance of identifying constructions formed with auxiliary verbs [4]. Such the correct root form of a word [1]. treebanks provide a foundational basis for syntactic B. Elov et al. (2023) compared stemming and POS normalization research. International Journal Of Literature And Languages 114 https://theusajournals.com/index.php/ijll International Journal Of Literature And Languages (ISSN: 2771-2834) Additionally, the GREW (graph rewriting) tool proposed Additionally, G‘. Matlatipov et al. (2022) created a by Bruno Guillaume (2021) helps consistently preserve labeled corpus for sentiment analysis in Uzbek. and transform syntactic annotations in corpora, which Although these studies are not directly about indirectly supports syntactic normalization (e.g., normalization, they note that during corpus harmonizing parse trees across languages). preparation, normalization (e.g., text cleaning, case folding, removing unnecessary characters) was a key The article by E. Kuriyozov et al. (2021), within the step. UzWordNet (Uzbek WordNet) project, introduces a lexical-semantic network for the Uzbek language [5]. These referenced studies demonstrate a growing The resource includes synonymy, antonymy, and scientific interest in computational processing of the hyponymy relations between words. This database Uzbek language. In particular, significant progress has provides a scientific basis for lexical normalization tasks been made in morphology and lexicography. Future such as grouping synonyms and consolidating research is expected to delve deeper into syntactic and redundant variants. semantic analysis. The results of these studies form the foundation for software and practical NLP systems, Kh. Madatov et al. (2022) described the process of enabling more efficient normalization an