← Research & Papers

1 Introduction

People increasingly hand parts of their thinking to external tools, a practice discussed as cognitive offloading (Shanmugasundaram and Tamilarasu 2023). A review of how digital technology affects cognition discusses over-reliance on AI tools in decision-making, but does not compare AI tools with other digital sources as means of looking up or cross-checking information. A second line of work asks whether ChatGPT functions as an epistemic authority: a source whose answer is taken as a reason to believe (Hauswald 2025; Richards 2025). Students report treating ChatGPT as a trusted learning source (Rastogi and Lawati 2024), and educators have raised concerns about what this does to justification practices (Cooper et al. 2024; Jose et al. 2025). The study was also prompted by an everyday observation: students verify even simple sums with a calculator, although they were sure of the answer.

We asked whether meeting ChatGPT as the first available checking tool changes checking behaviour, compared with meeting Wikipedia, which is similarly familiar and encyclopaedic. We predicted:

• H1 Participants who can check with ChatGPT first check more of their answers in the first round than participants who can check with Wikipedia first.

• H2 Participants who check with ChatGPT first check fewer answers with the other source in the second round.

H2 follows the idea that an answer from a tool treated as an authority is less often cross-checked.

2 Method

2.1 Participants

The class ran the study together, recruiting a convenience sample. Participants had to read English, be familiar with digital search tools, and not be enrolled in advanced courses in psychology or AI. Of 49 participants, two lacked a recorded group assignment and were excluded, leaving 47: 29 in the Wikipedia-first group and 18 in the ChatGPT-first group. The groups are unequal in size; the pilot had revealed a problem with the online group assignment, which was fixed before the main study (Section 2.4), and the record does not say why the main groups differ.

2.2 Materials

Ten general-knowledge questions each offered two answers (for example, What is the coloured portion of the eye called? Iris / Pupil; the full list is in Appendix A). For each answer, participants rated their certainty from 1 (not certain) to 4 (highly certain). The checking tools were presented as static screenshots of Wikipedia’s or ChatGPT’s answer to the question, so that every participant saw the same content. After the task, participants rated each tool from 1 (is never correct) to 4 (is always correct), and their usual double-checking from 1 (I never double-check) to 4 (I always double-check).

2.3 Design and procedure

The independent variable was the first available checking tool, manipulated between subjects: group 0 could check with Wikipedia first, group 1 with ChatGPT first (Figure 1). After giving consent online, participants read a task description that did not mention the checking opportunities. In round 1, they answered each question, rated their certainty, and could check the answer with their group’s tool. After an overview of their answers and of which ones they had checked, round 2 offered, for each question, a check with the other tool. The questionnaire followed, asking about the usual double-checking only after the task so that it would not reveal the study’s interest. Participants were then debriefed about the cover story and the purpose of the study. The study was anonymous, and no identifying information was collected. It was implemented in PsychoPy and run online through Pavlovia.

Measures. The two dependent variables are the number of questions checked in round 1 (0–10) and the number checked with the other tool in round 2 (0–10). Round 2 was offered for every question, so its count can include questions that were not checked in round 1; it measures checking with a second source rather than strictly double-checking the same answer.

2.4 Pilot and revisions

Eight people took part in a pilot; seven consented. Although the counterbalancing was set up in PsychoPy and on Pavlovia, six of the eight were assigned to the ChatGPT-first group, and the assignment was then fixed. Participants answered almost all questions correctly and with high certainty, and two did not check any answer, which suggested that the questions were too easy to prompt checking. We therefore replaced two questions with harder ones. One participant noted that a question about the Spanish siesta favoured people with a European background and said they had guessed it; it was among the two replaced. Finally, because one participant rated ChatGPT as almost always correct but checked only with Wikipedia, we added an authority rating for Wikipedia alongside the one for ChatGPT.

3 Results

Table 1 and Figures 2 and 3 summarise the two measures. In round 1, the ChatGPT-first group checked slightly fewer answers than the Wikipedia-first group, and its scores were more spread out. In round 2, the ChatGPT-first group checked more answers with the other tool; half of its members checked nine or more of the ten. Round-2 scores clustered at the extremes: many participants checked nearly all answers with the second tool, and in the Wikipedia-first group many checked almost none.

The course analysis was descriptive. For this write-up, we compared the groups with Welch’s $t$-test, computed from the summary statistics. Neither difference was reliable. In round 1 the difference was $-0.46$ questions (ChatGPT first minus Wikipedia first), $t(29.4) = -0.46$, $p = .65$, Cohen’s $d = -0.15$. In round 2 it was $+1.62$ questions, $t(38.1) = 1.44$, $p = .16$, $d = 0.43$. The scores are bounded counts with skewed distributions, so these tests are a rough guide rather than an exact analysis.

4 Discussion

Neither prediction was supported. Descriptively, both differences ran the other way: participants who met ChatGPT first checked slightly fewer answers at first, and checked more answers with Wikipedia afterwards. The second pattern is the larger one, and if it held in a larger sample it would suggest the opposite of the authority hypothesis: that an answer from ChatGPT prompts people to look elsewhere, rather than settling the question. With 18 and 29 participants and effect sizes of this magnitude, however, the study could not tell these patterns apart from chance, and we draw no conclusion about the direction.

Limitations. The sample was small, unbalanced between groups, and recruited by convenience, largely among students. Checking cost one click and showed a static screenshot, which is much easier than using the real tools; the high rate of checking may reflect this. The questions were easy for many participants even after revision. Round 2 counted checks with the second tool for any question, not only for answers already checked. The certainty and authority ratings were collected but are not analysed here, so we cannot say whether checking tracked uncertainty or trust in a tool. The study was a course exercise without pre-registration or a power analysis.

What a follow-up should change. A larger, balanced sample; harder questions calibrated so that participants are genuinely uncertain; a checking step with a real cost, such as time; a second round restricted to answers already checked, so that it measures double-checking; and an analysis that relates each check to the rated certainty of the answer and to the rated authority of each tool.

Contributions and ethics

I designed the study with two classmates; the class adopted our protocol by vote as the study it would run, and the data were collected by the class. I analysed the data during the course (descriptive statistics in JASP). This write-up, its figures, and the Welch tests computed from summary statistics were prepared by me after the course. Participants gave informed consent online, took part anonymously, and were debriefed about the cover story.

References

Appendix A: Questions