Cognitive Models of Polycode Texts: A Comprehensive Analysis

Аннотация

Polycode texts, which combine verbal and non-verbal elements, are increasingly prevalent in modern communication. This study investigates the cognitive models underlying the comprehension of polycode texts, aiming to elucidate the mental processes involved in integrating multiple semiotic systems. Using a mixed- methods approach, we conducted experiments with 120 participants (Mage = 28.5, SD = 4.2) to assess their comprehension of various polycode texts. Eye-tracking data and think-aloud protocols were collected and analyzed using both quantitative and qualitative methods. Results indicate that successful polycode text comprehension involves a complex interplay of visual attention, verbal processing, and cognitive integration. A novel "Integrated Polycode Comprehension Model" (IPCM) is proposed, synthesizing elements from dual coding theory and cognitive load theory. The IPCM suggests that comprehension is optimized when verbal and non-verbal elements are semantically congruent and spatially proximate. Furthermore, individual differences in cognitive styles significantly influenced comprehension patterns. These findings have important implications for the design of educational materials, user interfaces, and multimodal communication strategies. Future research directions and practical applications are discussed. Keywords: Polycode texts, cognitive models, multimodal comprehension, eye-tracking, think-aloud protocols, integrated polycode comprehension model. Introduction: In an increasingly digital and visually The present study aims to address this research gap by oriented world, polycode texts—those that combine investigating the cognitive models that govern verbal and non-verbal elements—have become polycode text comprehension. Specifically, we seek to: ubiquitous in communication across various domains, Identify the key cognitive processes involved in including education, advertising, and scientific integrating verbal and non-verbal elements in polycode discourse (Bateman, 2014). These multimodal texts texts. Examine how individual differences in cognitive present unique challenges and opportunities for styles influence polycode text comprehension. comprehension, necessitating a deeper understanding Develop a comprehensive model that explains the of the cognitive processes involved in their mental representations formed during polycode text interpretation. processing. While extensive research has been conducted on text Understanding these cognitive mechanisms is crucial comprehension (e.g., Kintsch, 1998) and visual for several reasons. First, it can inform the design of perception (e.g., Ware, 2008) separately, the cognitive more effective educational materials, potentially mechanisms underlying the integration of multiple enhancing learning outcomes across various disciplines semiotic systems in polycode texts remain (Schnotz & Bannert, 2003). Second, it can guide the underexplored. This gap in knowledge is particularly development of more intuitive user interfaces and data significant given the growing prevalence of visualization techniques, improving human-computer infographics, multimedia presentations, and interaction (Tufte, 2001). Finally, insights into polycode interactive digital content in both professional and text comprehension can contribute to broader theories educational settings (Mayer, 2009). American Journal Of Philological Sciences 69 https://theusajournals.com/index.php/ajps American Journal Of Philological Sciences (ISSN – 2771-2273) of multimodal cognition and communication. Cognitive Style Assessment: The Verbal-Visual Learning Style Rating (VVLSR) questionnaire (Mayer & Massa, This paper is structured as follows: We begin with a 2003) was administered to assess participants' review of relevant literature, synthesizing findings from cognitive preferences. cognitive psychology, semiotics, and multimedia learning. Next, we describe our mixed-methods Comprehension Tests: For each polycode text, a approach, which combines eye-tracking technology comprehension test consisting of 10 multiple-choice with think-aloud protocols to capture both quantitative questions was developed to assess participants' and qualitative aspects of polycode text processing. We understanding of both verbal and visual information. then present our results, introducing the Integrated Procedure Polycode Comprehension Model (IPCM) as a novel 1. Participants completed the VVLSR questionnaire to framework for understanding multimodal cognition. determine their cognitive style preferences. Finally, we discuss the implications of our findings for theory and practice, acknowledging limitations and 2. After calibration of the eye-tracking equipment, suggesting directions for future research. participants were presented with the polycode texts in a randomized order on a 24-inch monitor. By elucidating the cognitive models underlying polycode text comprehension, this study aims to 3. For each text, participants were ins