Next time you’re about to ask an AI chatbot to help you solve a hard problem, you might want to slow your roll.
People who waited to consult an AI chatbot until they had partially worked through a problem on their own performed better on a critical thinking task than those who used the chatbot from the start, researchers reported April 14 at the 2026 CHI conference on Human Factors in Computing Systems in Barcelona. Under tight deadlines, though, using AI early in the process did provide a boost, highlighting a trade-off between speed and independent reasoning, and raising questions about how and when we should use chatbots.
In the study, computer scientist Mina Lee of the University of Chicago and colleagues randomly assigned 393 people to one of eight categories. First, participants were divided into two large groups: those given sufficient time (30 minutes) or insufficient time (10 minutes). Then, they were divided into smaller groups based on when, or if, they could use the OpenAI’s GPT-4o chatbot: early, continuous, late or no access. Each group had roughly 40 to 50 participants.
Next, participants were instructed to play the role of a city council member and decide, using a set of seven documents, whether to accept or reject a company’s proposal to mitigate a water contamination problem. Each participant had to write an essay explaining their decision.
The researchers scored the essays based, in part, on how many valid arguments and textual references they contained and found that participants given 30 minutes performed better across the board than those given only 10 minutes. The most successful in terms of essay scores were participants who had enough time to complete the task and had access to the chatbot later in the process.
When the researchers looked at how well participants remembered information in the provided documents, the most successful group was the one that had sufficient time and never had access to the chatbot. The researchers also scored myside bias, measuring how many perspectives participants incorporated in their arguments. They found that the group with sufficient time and late chatbot access did best.
The results align with research on two kinds of learning: one based on slow, effortful reasoning and another based on fast, automatic thinking, says Barbara Oakley, a systems engineer and education expert at Oakland University in Rochester Hills, Mich. Slow learning involves carefully building an understanding of the problem and weighing options, while fast learning relies on habits and quick judgments with little reflection. Participants who had time to reason through the material on their own before using AI did best because they had already engaged in that slower, more deliberate learning, she says.
Of course, in the real world, people often have to complete critical thinking tasks under time pressure. In the four groups in the “insufficient time” category, the group with access to the chatbot early on scored the highest on their essays. That doesn’t mean we should rush to use AI, Lee says. “When you are under time pressure and use AI to boost your performance, then you are basically risking [just taking and using the] AI’s framing, and that reduces the kinds of arguments that you make and your engagement with the documents or different pieces of information,” she says. You have to “at least be aware of what you’re signing up [for].”
That awareness is probably what everyone should aim for right now. People will need strong AI literacy and knowledge of their own thinking patterns to weigh the risks and benefits of using chatbots in different scenarios and at different points in problem-solving, Lee says. “I think our work kind of targets time constraints as the first step towards [that] understanding.”
Read the full article here
