The Algorithmic Limits of Historical Evidence and Responsibility for Justification in Generative Artificial Intelligence - AI-JISR PDF / Print
AI-Driven Journal of Interdisciplinary Social Research (AI-JISR)
e-ISSN: 3150-0951 • DOI: 10.5281/zenodo.24174687
Volume 1, Issue 2 (2026)
Pages: 101-125
OPEN ACCESS • CC-BY 4.0

The Algorithmic Limits of Historical Evidence and Responsibility for Justification in Generative Artificial Intelligence

1 Milli Eğitim Bakanlığı, Türkiye
Abstract (English)

The use of generative artificial intelligence in historical writing exposes a distinction between citing sources and justifying historical claims. This article examines the conditions under which support judgments used in language model evaluation can inform historical source criticism. It combines a comparative conceptual analysis of ten purposively selected scholarly and technical publications with a bounded examination of the historical text of Turkey’s Law on the Unification of Education. The publications are compared in terms of their objects of evaluation, evidential grounds, inferential scope, and unresolved questions. The analysis shows that identifying sources, tracing production processes, establishing support for propositions, and justifying historical explanations are distinct operations. It develops algorithmic source criticism as a framework connecting source provenance, textual transformation, and claim justification. The framework separates the mode of a claim’s relationship to its source from its evidential status and records the conditions for revising an assessment. The documentary examination demonstrates that a legal text can directly support claims about its provisions while requiring additional evidence for implementation, public reception, and causal effects. The example does not measure model performance. The article contributes a method for examining the relationship between the scope of a historical claim and its available grounds without reducing credibility to a single accuracy score. It offers implications for historical scholarship, education, and editorial assessment while preserving interpretive plurality and specifying responsibility for correction.

Keywords: Historiography, Generative Artificial Intelligence, Source Criticism, Historical Evidence, Digital History, Authorship Responsibility

References

  1. American Historical Association. (2025). Guiding principles for artificial intelligence in history education. https://www.historians.org/resource/guiding-principles-for-artificial-intelligence-in-history-education/
  2. Bender, E. M., & Koller, A. (2020). Climbing towards NLU: On meaning, form, and understanding in the age of data. In D. Jurafsky, J. Chai, N. Schluter, & J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 5185–5198). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.463
  3. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
  4. Elangovan, A., Liu, L., Xu, L., Bodapati, S. B., & Roth, D. (2024). ConSiDERS-The-Human evaluation framework: Rethinking human evaluation for generative large language models. In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1137–1160). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.acl-long.63
  5. Fickers, A. (2020). Update für die Hermeneutik: Geschichtswissenschaft auf dem Weg zur digitalen Forensik? [An update for hermeneutics: Historical scholarship on the way to digital forensics?] Zeithistorische Forschungen/Studies in Contemporary History, 17(1), 157–168. https://doi.org/10.14765/zzf.dok-1765
  6. Gao, T., Yen, H., Yu, J., & Chen, D. (2023). Enabling large language models to generate text with citations. In H. Bouamor, J. Pino, & K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 6465–6488). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.398
  7. Hughes-Warrington, M. (2025). Ethics for artificial historians. History and Theory, 64(2), 159–177. https://doi.org/10.1111/hith.12377
  8. Hughes-Warrington, M. (2026). History and artificial intelligence. Cambridge University Press. https://doi.org/10.1017/9781009572187
  9. Jaakkola, E. (2020). Designing conceptual articles: Four approaches. AMS Review, 10(1–2), 18–26. https://doi.org/10.1007/s13162-020-00161-0
  10. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), Article 248. https://doi.org/10.1145/3571730
  11. Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103(1), 54–69. https://doi.org/10.1037/a0028347
  12. Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000
  13. Lebo, T., Sahoo, S., & McGuinness, D. (Eds.). (2013). PROV-O: The PROV ontology. World Wide Web Consortium. https://www.w3.org/TR/prov-o/
  14. Leme Lopes, A. P. (2023). Artificial history? Inquiring ChatGPT on historiography. Rethinking History, 27(4), 709–749. https://doi.org/10.1080/13642529.2023.2234227
  15. Lévesque, S. (2008). Thinking historically: Educating students for the twenty-first century. University of Toronto Press.
  16. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.), Advances in Neural Information Processing Systems 33 (pp. 9459–9474). Curran Associates. https://papers.nips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
  17. Milligan, I. (2019). History in the age of abundance? How the web is transforming historical research. McGill-Queen’s University Press.
  18. Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023). FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In H. Bouamor, J. Pino, & K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 12076–12100). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.741
  19. Moreau, L., & Missier, P. (Eds.). (2013). PROV-DM: The PROV data model. World Wide Web Consortium. https://www.w3.org/TR/prov-dm/
  20. Putnam, L. (2016). The transnational and the text-searchable: Digitized sources and the shadows they cast. The American Historical Review, 121(2), 377–402. https://doi.org/10.1093/ahr/121.2.377
  21. Ramsay, S. (2011). Reading machines: Toward an algorithmic criticism. University of Illinois Press. https://doi.org/10.5406/illinois/9780252036415.001.0001
  22. Reber, R., & Schwarz, N. (1999). Effects of perceptual fluency on judgments of truth. Consciousness and Cognition, 8(3), 338–342. https://doi.org/10.1006/ccog.1999.0386
  23. Trouillot, M.-R. (1995). Silencing the past: Power and the production of history. Beacon Press.
  24. Türkiye Büyük Millet Meclisi. (1924). Tevhidi tedrisat kanunu [Law on the unification of education] (Law No. 430, March 3, 1340 [1924]). Kanunlar Dergisi, 2, 242. https://www5.tbmm.gov.tr/tutanaklar/KANUNLAR_KARARLAR/kanuntbmmc002/kanuntbmmc002/kanuntbmmc00200430.pdf
  25. van Drie, J., & van Boxtel, C. (2008). Historical reasoning: Towards a framework for analyzing students’ reasoning about the past. Educational Psychology Review, 20(2), 87–110. https://doi.org/10.1007/s10648-007-9056-1
  26. Walters, W. H., & Wilder, E. I. (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports, 13(1), Article 14045. https://doi.org/10.1038/s41598-023-41032-5
  27. Westfall, J., Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143(5), 2020–2045. https://doi.org/10.1037/xge0000014
  28. Wineburg, S. (2001). Historical thinking and other unnatural acts: Charting the future of teaching the past. Temple University Press.
  29. Zhao, J. X., Liu, J. Z. J., Hooi, B., & Ng, S.-K. (2025). How does response length affect long-form factuality. In W. Che, J. Nabende, E. Shutova, & M. T. Pilehvar (Eds.), Findings of the Association for Computational Linguistics: ACL 2025 (pp. 3102–3125). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-acl.161
  30. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems 36 (pp. 46595–46623). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract.html