THE INFLUENCE OF RATING EXPERTISE AND LANGUAGE BACKGROUND ON JUDGMENTS OF SYNTACTIC COMPLEXITY IN L2 WRITING IN ITALIAN
DOI:
https://doi.org/10.54103/2037-3597/27753Abstract
The purpose of the study is to examine, by means of a quantitative and a qualitative analysis, the influence of rating expertise and language background on judgments of syntactic complexity in L2 writing in Italian. An additional aim is to explore the extent to which raters’ reflections are related to current views in SLA (Second Language Acquisition) on the development of syntactic complexity in L2. By means of a quantitative and a qualitative analysis, the paper compares the complexity ratings and score motivations of three groups of raters: 16 language teachers with Italian as their first language (L1); 18 (high)intermediate L2 university students of Italian, with Finnish and Hungarian as their L1); 20 Italian university students, native speakers of Italian. All raters were asked to evaluate the syntactic complexity of six argumentative texts written by (low) intermediate learners of Italian. No impact of rating expertise or L1 vs. L2 background could be established, as high correlations were found between the three groups. The qualitative analysis of the score motivations revealed, however, some interesting differences between expert en non-expert raters. In a number of cases, raters’ scoring motivations and feedback were found to be influenced by syntactic characteristics of their L1. Raters’reflections on syntactic complexity appeared to be only partly aligned with existing theoretical views on syntactic complexity.
L’IMPATTO DELL’EXPERTISE E DELLA (NON) NATIVESSNESS NELLA VALUTAZIONE DELLA COMPLESSITÀ SINTATTICA NELLE PRODUZIONI SCRITTE IN ITALIANO L2
L’obiettivo di questo contributo è l’esame dei giudizi e il tipo di feedback di un gruppo di valutatori esperti (insegnanti) e ínesperti (studenti), della complessità sintattica nelle produzioni scritte in italiano L2, attraverso un’ analisi quantitativa e qualitativa. Una seconda variabile su cui lo studio si focalizza è l’impatto della (non)nativeness (parlante nativo vs. parlante non nativo) del valutatore. Lo studio si propone inoltre di esplorare la misura in cui i giudizi dei valutatori sono allineati con le ipotesi sulla crescità della complessità sintattica nella letteratura SLA (Second Language Acquisition). Lo studio compare i giudizi di tre gruppi di valutatori: 16 insegnanti di lingua, parlanti nativi dell’italiano; 18 apprendenti di italiano (studenti universitari, di lingua materna ungherese e finlandese); 20 parlanti L1 (studenti universitari, di madrelingua italiana). Agli informanti è stato chiesto di giudicare la complessità sintattica di un piccolo campione di testi argomentativi (n=6), scritti da apprendenti di livello basso-intermedio. L’analisi quantitativa dei dati ha dimostrato che i giudizi di insegnanti vs. studenti, e di parlanti nativi vs. parlanti non nativi, concordano largamente e risultano altrettanto affidabili. L’analisi qualitativa del feedback e delle motivazioni dei giudizi assegnati, ha invece evidenziato delle differenze interessanti tra valutatori ‘esperti’ e ‘non-esperti’. Anche le preferenze sintattiche della lingua materna (L1 vs. L2) sembrano influenzare i loro giudizi nella lingua target. Lo studio ha infine fatto vedere che le motivationi su cui si basano i giudizi degli insegnanti rispecchiano solo in parte i risultati emersi dalla ricerca sullo sviluppo della complessità in L2.
Downloads
Riferimenti bibliografici
Ågren M., Granfeldt J., Schlyten S. (2012), “The growth of complexity and accuracy in L2 French. Past observations and recent applications of developmental stages”, in Housen A., Kuiken F., Vedder I. (eds.), Dimensions of L2 performance and proficiency: Complexity, accuracy and fluency in SLA, John Benjamins, Amsterdam, pp. 95-119: https://doi.org/10.1075/lllt.32. DOI: https://doi.org/10.1075/lllt.32.05agr
Ait Eljoudi Q. (2018), “Algerian teachers’ and learners’ beliefs about learner autonomy”, in Mackay J., Birello M., Xerri D. (eds.), ELT Research in action. Bridging the gap between research and classroom practice, IATEFL, Kent, pp. 65-70.
Bardovi-Harlig K. (1992), “A second look at T-unit analysis: Reconsidering the sentence”, in TESOL Quarterly, 26, 2, pp. 390-395: https://doi.org/10.2307/3587016. DOI: https://doi.org/10.2307/3587016
Biber D., Gray B. (2011), “Grammatical change in the noun phrase: The influence of written language use”, in English Language and Linguistics, 15, 2, pp. 223-250: https://doi.org/10.1017/S1360674311000025. DOI: https://doi.org/10.1017/S1360674311000025
Biber D., Gray B., Poonpon K. (2011), “Should we use characteristics of conversation to measure grammatical complexity in L2 writing development?”, in TESOL Quarterly, 45, 1, pp. 5-35: https://doi.org/10.5054/tq.2011.244483. DOI: https://doi.org/10.5054/tq.2011.244483
Biber D., Gray B., Staples S. (2016), “Predicting patterns of grammatical complexity across language-exam task types and proficiency levels”, in Applied Linguistics, 37, 5, pp. 639-668: https://doi/10.1093/applin/amu05. DOI: https://doi.org/10.1093/applin/amu059
Bogorevich V. (2018), Native and non-native raters of L2-speaking performance: Accent familiarity and cognitive processes, Doctoral dissertation, Northern Arizona University.
Bonk W. J., Ockey, G. J. (2003), “A many-facet Rasch analysis of the second language group oral discussion task”, in Language Testing, 20, 1:
https://doi.org/10.1191/0265532203lt245oa. DOI: https://doi.org/10.1191/0265532203lt245oa
Brezina V., Pallotti G. (2019), “Morphological complexity in written L2 texts”, in Language Research, 35, 1, pp. 99-120: https://doi.org/10.1177/0267658316643125. DOI: https://doi.org/10.1177/0267658316643125
Brown A., Iwashita N., McNamara T. (2005), An examination of rater orientations and testtaker performance on English‐for‐academic‐purposes speaking tasks, ETS Research Report
Series, John Wiley & Sons Inc., Hoboken (N. J.): https://doi.org/10.1002/j.2333-8504.2005.tb01982.x. DOI: https://doi.org/10.1002/j.2333-8504.2005.tb01982.x
Bulté B., Housen A. (2012), “Defining and operationalising L2 complexity”, in Housen A., Kuiken F., Vedder I. (eds.), Dimensions of L2 performance and proficiency: Complexity, accuracy and fluency in SLA, John Benjamins, Amsterdam, pp. 23-46: https://doi.org/10.1075/lllt.32. DOI: https://doi.org/10.1075/lllt.32.02bul
Bulté B., Housen A., Pallotti G. (2024), “Complexity and difficulty in second language acquisition: A theoretical and methodological overview”, in Language Learning, 20, pp. 1-42: https://doi.org/10.1111/lang.12669. DOI: https://doi.org/10.1111/lang.12669
Council of Europe (2001), Common European Framework of Reference for Languages. Learning, Teaching, Assessment, Council of Europe, Strasbourg.
Cucinotta G. (2018), “Teachers’ perceptions of motivational strategies in the language classroom. An empirical study of Italian Fl and Italian L2 teachers”, in EL.LE, 7, 3, pp. 446-472: https://doi.org/10.30687/ELLE/2280-6792/2018/03/006. DOI: https://doi.org/10.30687/ELLE/2280-6792/2018/21/006
Duym K., Schoonen R., Hulstijn J. H. (2018), “Professional and non-professional raters’ responsiveness to accuracy in L2 speech: An experimental approach”, in Language Testing, 35, 4, pp. 501-527: https://doi.org/10.1177/0265532217712553. DOI: https://doi.org/10.1177/0265532217712553
Granfeldt J., Nugues P., Ågren M., Thulin J., Persson E., Schlyter S. (2006), “CEFLE and Direkt Profil: A new computer learner corpus in French L2 and a system for grammatical profiles”, in Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), European Language Resources Association (ELRA), pp. 565-570: http://www.lrec-conf.org/proceedings/lrec2006/pdf/246_pdf.pdf.
Housen A., Kuiken F., Vedder I. (eds.) (2012). Dimensions of L2 performance and proficiency. Complexity, accuracy and fluency in SLA. John Benjamins, Amsterdam: https://doi.org/10.1075/lllt.32. DOI: https://doi.org/10.1075/lllt.32
Kim H. J. (2015), A qualitative analysis of rater behavior on an L2 speaking assessment, in Language Assessment Quarterly, 12, 3, pp. 239-261: https://doi.org/01.1080/1543303.2015.1049353. DOI: https://doi.org/10.1080/15434303.2015.1049353
Knoch U. (2009), “Diagnostic assessment of writing: A comparison of two rating scales”, in Language Testing, 26, 2, pp. 275-304: https://doi.org/10.1177/0265532208101008. DOI: https://doi.org/10.1177/0265532208101008
Kuiken F., Vedder I. (2014), “Raters’ decisions, rating procedures and rating scales, in Language Testing, 31, 3, pp. 279-284: https://doi.org/10.1177/0265532214526179. DOI: https://doi.org/10.1177/0265532214526179
Kuiken F., Vedder I. (2019a), “Syntactic complexity across proficiency and languages: L2 and L1 writing in Dutch, Italian and Spanish”, in International Journal of Applied Linguistics, 29, 2, pp. 192-210: https://doi.org/10.1111/ijal.12256. DOI: https://doi.org/10.1111/ijal.12256
Kuiken F., Vedder I. (2019b), “Investigating teachers’ perceptions of syntactic complexity in L2 academic writing”, in Instructed Second Language Acquisition, 3, 2, pp. 228-248: https://doi.org/10.1558/isla.37977. DOI: https://doi.org/10.1558/isla.37977
Kuiken F., Vedder I., Gilabert R. (2010), “Communicative adequacy and linguistic complexity in L2 writing”, in Bartning I., Martin M., Vedder I. (eds.), Communicative proficiency and linguistic development: Intersections between SLA and language testing research, Eurosla Monograph Studies, 1, pp. 1-56.
Kuiken F., Vedder I., Michel M. (2019), “Linguistic complexity in second language acquisition: Introduction”, in Instructed Second Language Acquisition, 3, 2, pp. 119-123: https://doi.org/10.1558/isla.39602. DOI: https://doi.org/10.1558/isla.39602
Kyle K., Crossly S. A. (2018), “Measuring syntactic complexity in L2 writing. Using finegrained clausal and phrasal indices”, in The Modern Language Journal, 102, 2, pp. 333-349. DOI: https://doi.org/10.1111/modl.12468
Kyle K., Crossly S. A., Verspoor M. (2021), “Measuring longitudinal writing development using indices of syntactic complexity and sophistication”, in Studies in Second Language Acquisition, 43, 4, pp. 781-812: https://doi.org/10.1111/modl.12468. DOI: https://doi.org/10.1017/S0272263120000546
Lambert G., Kormos J. (2014), “Complexity, accuracy and fluency in Task-based L2 research: Toward more developmentally based measures of second language acquisition”, in Applied Linguistics, 35, 5, pp. 607-614: https://doi.org/10.1093/applin/amu047. DOI: https://doi.org/10.1093/applin/amu047
Lu X. (2010), “Automatic analysis of syntactic complexity in second language writing”, in International Journal of Corpus Linguistics, 15, 4, pp. 474-496: https://doi.org/10.1075/ijcl.15.4.02lu. DOI: https://doi.org/10.1075/ijcl.15.4.02lu
Lu X. (2011), “A corpus-based evaluation of syntactic complexity measures as indics of college-level ESL writers’ language development”, in TESOL Quarterly, 45, 1, pp. 36-62: https://doi.org/10.5054/tq.2011.240859. DOI: https://doi.org/10.5054/tq.2011.240859
Lumley T. (2005), Assessing second language writing: The rater’s perspective, Peter Lang, Lausanne.
Norris J. M., Ortega L. (2009), “Towards an organic approach to investigating CAF in instructed SLA: The case of complexity”, in Applied Linguistics, 30, 4, pp. 555-578: https://doi.org/10.1093/applin/amp044. DOI: https://doi.org/10.1093/applin/amp044
O’Leary J. A., Steinkraus R. (2022), “Syntactic and lexical complexity in L2 academic writing: Development and competition”, in Ampersand, 9, 100096, pp. 1-10: https://doi.org/10.1016/j.amper.2022.100096. DOI: https://doi.org/10.1016/j.amper.2022.100096
Ortega L. (2003), “Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing”, in Applied Linguistics, 24, 4, pp. 492-518: https://doi.org/10.1093/applin/24.4.492. DOI: https://doi.org/10.1093/applin/24.4.492
Palermo M. (2012), Linguistica testuale dell’italiano, il Mulino, Bologna.
Pallotti G. (2009), “CAF: Defining, refining and differentiating constructs”, in Applied Linguistics, 30, 4, pp. 590-601: https://doi.org/10.1093/applin/amp045. DOI: https://doi.org/10.1093/applin/amp045
Pallotti G. (2017), “Applying the interlanguage approach to language teaching”, in International Review of Applied Linguistics in Language Teaching (IRAL), 55, 4, pp. 393-412: https://doi.org/10.1515/iral-2017-0145. DOI: https://doi.org/10.1515/iral-2017-0145
Pallotti G. (2021), “Measuring complexity, accuracy, and fluency (CAF)”, in Winke P., Brunfaut T. (eds.), The Routledge handbook of second language acquisition and language testing, Routledge, London-New York, pp. 201-210. DOI: https://doi.org/10.4324/9781351034784-23
Paquot M. (2019), “The phraseological dimension in interlanguage complexity research”, in Second Language Research 35, 1, pp. 121-145:
https://doi.org/10.1177/0267658317694221. DOI: https://doi.org/10.1177/0267658317694221
Pill J., Smart C. (2021), “Raters: Behavior and training”, in Winke P., Brunfaut T. (eds.), The Routledge handbook of second language acquisition and language testing, Routledge, London-New York, pp. 135-144. DOI: https://doi.org/10.4324/9781351034784-15
Révész A., Gurzynski-Weiss L. (2016), “Teachers’ perspectives on second language task difficulty: Insights from think-alouds and eye-tracking”, in Annual Review of Applied Linguistics, 36, pp. 182-204: https://doi.org/10.1017/S0267190515000124. DOI: https://doi.org/10.1017/S0267190515000124
Vasylets O. R. G., Manchón R. M. (2019), “Differential contribution of oral and written modes to lexical and propositional complexity in L2 performance in instructed contexts”, in Instructed Second Language Acquisition, 3, 2, pp. 206-227: https://doi.org/10.1558/isla.38289. DOI: https://doi.org/10.1558/isla.38289
Vyatkina N. (2012), “The development of second language writing complexity in groups and individuals: A longitudinal learner corpus study”, in Modern Language Journal, 96, 4, pp. 572-594: https://doi.org/10.1111/j.1540-4781.2012.01401.x. DOI: https://doi.org/10.1111/j.1540-4781.2012.01401.x
Winke P., Gass S., Myford C. (2012), “L2 background as a potential source of bias in rating performance”, in Language Testing, 30, 2, pp. 231-252: https://doi.org/10.1177/0265532212456968. DOI: https://doi.org/10.1177/0265532212456968
Wolfe-Quintero K., Inagaki S., Kim H-Y. (1998), Second language development in writing: Measures of fluency, accuracy, and complexity, University of Hawai‛i Press, Honolulu.
Yoon H.-J. (2017), “Linguistic complexity in L2 writing revisited: Issues of topic, proficiency, and construct multidimensionality”, in System, 66, pp. 130-141: https://doi.org/10.1016/j.system.2017.03.007. DOI: https://doi.org/10.1016/j.system.2017.03.007
Zhang Y., Elder C. (2011), “Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?”, in Language Testing, 28, 1, pp. 31-50: https://doi.org/10.1177/0265532209360671. DOI: https://doi.org/10.1177/0265532209360671
Dowloads
Pubblicato
Come citare
Fascicolo
Sezione
Licenza
Questo lavoro è fornito con la licenza Creative Commons Attribuzione - Condividi allo stesso modo 4.0.
