Abstract

This study focuses on the rubrics used for text evaluation, with the aim of introducing the Reader-Friendly Informative Text Rubric (ODBIMDEPA). This rubric, developed and validated through a reliability study, evaluates the reader-friendliness of informative texts used both in educational settings and in exams administered by institutions such as the Ministry of National Education (MoNE), the Assessment, Selection and Placement Center (ÖSYM), and the Yunus Emre Institute (YEE). The study also seeks to share insights gained from the analysis of evaluated texts. To achieve these goals, the document review method was employed, resulting in the creation of two distinct datasets. The first dataset, analyzed during the initial stage of the study, comprised 50 informative texts taken from the 5th, 6th, 7th, and 8th-grade textbooks used in MoNE schools in 2018. The second dataset, analyzed in the subsequent stage, included 37 informative texts selected from the YKS (University Entrance) exams administered by ÖSYM in 2023 and 2024. In the first stage, two independent academic raters scored the texts, while in the second stage, 11 expert raters for 4 subjects, 13 raters for 1 subject assessed them across eight dimensions defined by ODBIMDEPA. In the initial analysis, the weighted kappa coefficient was used to evaluate inter-rater reliability, indicating that the weighted average was acceptable and demonstrating that ODBIMDEPA exhibited significant consistency. In the second stage, the dataset was analyzed using the many-facet Rasch model, revealing a rater reliability of 0.95. This high reliability underscores the robustness of the developed rubric. Based on these findings, ODBIMDEPA provides a reliable framework for quantifying the reader-friendliness of informative texts used in assessments administered by institutions such as MoNE and ÖSYM, as well as those in educational textbooks. The rubric facilitates a quantitative assessment of whether these texts are suitable for use in exams or educational settings, offering insights into text selection and evaluation.

Keywords: Educational assessment, Informative text, Rubric, Reader-friendly text, ODBIMDEPA

References

  1. Aidinlou, N., Khodamard, N., & Azami, J. (2012). The effect of textual cohesive reference instruction on the reading comprehension of Iranian EFL students. International Journal English Linguistics, 2(5), 18-24. doi:10.5539/ijel.v2n5p18
  2. Aktaş, M., & Alıcı, D. (2018). Yazılan hikâyeyi değerlendirmeye yönelik analitik rubrik geliştirme: Geçerlik ve güvenirlik çalışması. Mersin Üniversitesi Eğitim Fakültesi Dergisi, 14(2), 597-610. doi:10.17860/mersinefd.424198
  3. Beck, I. L., McKeown, M. G., Sinatra, G. M., & Loxterman, J. A. (1991). Revising social studies text from a text-processing perspective: Evidence of improved comprehensibility. Reading Research Quarterly, 26, 251-276.
  4. Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3th ed.). London: Routledge/Taylor & Francis Group. doi:10.4324/9781315814698
  5. Brewer, W. F. (1980). Literary theory, rhetoric, and stylistics: Implications for psychology. In R. J. Shapiro, B. C. Bruce, & W. F. Brewer (Eds.), Theoretical issues in reading comprehension (pp. 221-239). Hillsdale, NJ: Erlbaum.
  6. Britton, B. K., & Gulgoz, S. (1991). Using Kintsch's computational model to improve instructional text: Effects of repairing inference calls on recall and cognitive structures. Journal of Educational Psychology, 83, 329-404.
  7. Çelik, T., Demirgüneş, S., & Baştuğ, M. (2014). Metin oluşturma ve değerlendirme çalışmalarına yönelik “okur dostu” metin değerlendirme rubriği. International Journal of Eurasia Social Sciences, 5(14), 65-83.
  8. Çelik, T., Demirgüneş, S., & Fidan, D. (2015). Okur dostu metin olma özelliği ile okuduğunu anlama başarısı arasındaki ilişkinin incelenmesi. Başkent Üniversitesi Eğitim Fakültesi Dergisi, 2(1), 119-127.
  9. Çıralı Sarıca, H., & Koçak Usluel, Y. (2016). Eğitsel bağlamda dijital hikâye anlatımı: Bir rubrik geliştirme çalışması. Eğitim Teknolojisi: Kuram ve Uygulama, 6(2), 65-84.
  10. Demirgüneş, F. (2015). Türkçe ders kitaplarındaki metinlerin okur dostu metin bağlamında değerlendirilmesi (Unpublished master's thesis). Niğde University, Niğde.
  11. Dreher, M. J., & Singer, H. (1989). Friendly texts and friendly teachers. Theory into practice. Learning Through Text, 28(2), 98-104.
  12. Duke, N., & Bennett Armistead, V. (2003). Reading and writing informational text in the primary grades: Research-based practices. New York: Scholastic.
  13. Eser, M. T., & Aksu, G. (2022). R programlama dili ile puanlayıcılar arası uyum ve güvenirlik (N. Doğan, Ed.). Ankara: Pegem.
  14. Field, A. (2013). Discovering statistics using IBM SPSS statistics (4th ed.). Thousand Oaks, CA: SAGE Publications.
  15. Fisher, D., Frey, N., & Lapp, D. (2008). Shared reading: Modeling comprehension, vocabulary, text structures, and text features for older students. The Reading Teacher, 6(7), 584-556. doi:10.1598/RT.61.7.4
  16. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382.
  17. Gray, W. S., & Leary, B. E. (1935). What makes a book readable. Chicago: University of Chicago Press.
  18. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer.
  19. İşeri, K., Çelik, T., & Demirgüneş, S. (2015). Türkçe ders kitaplarında okur dostu metinlerin incelenmesi. E-Dil Dergisi, 1(5), 227-242.
  20. Kintsch, W. (1982). Aspects of text comprehension. Advances in Psychology, 9, 301-312.
  21. Kintsch, W. (1988). The role of knowledge in discourse processing: A construction-integration model. Psychological Review, 95, 163-182.
  22. Klink, S., Dengel, A., & Kieninger, T. (2000). Document structure analysis based on layout and textual features. Proceedings of International Workshop on Document Analysis Systems, DAS2000.
  23. Linacre, M. (1993). Generalizability theory and many-facet Rasch measurement. Paper presented at the Annual Meeting of the American Educational Research Association, Atlanta, GA.
  24. Lorch, R., Lemarié, J., & Grant, R. (2011). Signaling hierarchical and sequential organization in expository text. Scientific Studies of Reading, 15(3), 267-284. doi:10.1080/10888431003747535
  25. Mann, W., & Thompson, S. (1988). Rethorical structure theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse, 8(3), 243-281. doi:10.1515/text.1.1988.8.3.243.
  26. McNamara, T. (2001). Language assessment as social practice: challenges for research. Language Testing, 18, 333-349. doi:10.1177/026553220101800402.
  27. McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14(1), 1-43. doi:10.1207/s1532690xci1401_1.
  28. McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010). Coh-Metrix: Capturing Linguistic Features of Cohesion. Discourse Processes, 47(4), 292–330. doi:10.1080/01638530902959943
  29. Ministry of National Education. (2024). Ortaokul Türkçe dersi öğretim programı (5, 6, 7 ve 8. sınıflar). Retrieved from https://mufredat.meb.gov.tr
  30. Moskal, B. M., & Leydens, J. A. (2000). Scoring rubric development: Validity and reliability. Practical Assessment, Research & Evaluation, 7(10). Retrieved from https://www.researchgate.net/publication/242637290_Scoring_Rubric_Development_Validity_and_Reliability
  31. Müldür, M., & Şimşek, N. D. (2020). Türkçe ders kitaplarındaki bilgilendirici metinlerin ve yapılarının incelenmesi üzerine bir araştırma. Ana Dili Eğitimi Dergisi, 8(3), 843-867.
  32. Nagabhand, S., Nation, I., & Franken, M. (1993). Can text be too friendly?. Reading in a Foreign Language, 9(2), 895-907.
  33. Rasch, G. (1960). Probabilistic model for some intelligence and achievement tests. Copenhagen: Danish Institute for Educational Research.
  34. Smith, F. (1982). Writing and the writer. London: Heinemann.
  35. Soyuçok, M., & Kartal, H. (2018). İlkokuma-yazma ders kitaplarının metin oluşturma basamağında oluşturulan metinlerin değerlendirilmesi. Mehmet Akif Ersoy Üniversite Eğitim Fakültesi Dergisi, 47, 435-460.
  36. Şencan, H. (2005). Sosyal ve davranışsal ölçmelerde güvenirlik ve geçerlilik. Ankara: Seçkin Yayınları.
  37. Tuncel, G. (2011). Sosyal bilgiler dersinde rubriklerin etkili kullanımı. Marmara Coğrafya Dergisi, 23, 213-233.
  38. Yıldırım, H., & Ocak, G. (2016). İlkokul sosyal bilgiler dersi “okur dostu” metinlere yönelik rubrik hazırlama çalışması. In XV. Uluslarararası Sınıf Öğretmenliği Sempozyumu bildiri özet kitabı (pp. 422-423). Ankara: Pegem Akademi.
  39. Yeung, L. (2007). In search of commonalities: Some linguistic and rhetorical features of business reports as a genre. English for Specific Purposes, 26(2), 156-179.
  40. Yıldırım, A., & Şimşek, H. (2008). Sosyal bilimlerde nitel araştırma yöntemleri (6. ed.). Ankara: Seçkin Yayıncılık.

Copyright and license

Copyright © 2025 The Author(s). This is an open access article distributed under the Creative Commons Attribution License (CC BY), which permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is properly cited.

How to cite

Fidan, D., Çelik Korat, T., Demirgüneş, S., Alaca, S., & Kırömeroğlu, N. (2025). A Tool for Selecting and Writing Texts for Assessing Reading: Reader-Friendly Informative Text Rubric (ODBIMDEPA). Education and Science, 50, 43-68. https://doi.org/10.15390/EB.2025.14166