Human vs Machine in the Game of Hidden Rules

Human vs Machine in the Game of Hidden Rules

by Jacob Feldman

Humanities, Social Sciences & Thought Psychology JMPsychologyJMRCognition and cognitive psychology
🎙 Jacob Feldman 👥 305 📅 March 5, 2026 ⏱ 84 min 👁 59 📄 original study 🧭 2026-08-16
Available in: English (current) Français

Keywords

Game of Hidden Ruleshuman vs AIrule discoveryreinforcement learningcognitive science

Summary

Jacob Feldman presents a comparative study of human and AI performance on the Game of Hidden Rules (GOHR), a rule-discovery task. The game involves sorting colored shapes into buckets based on an unknown rule, which players must infer through trial and error. Feldman emphasizes the importance of empirical comparison rather than speculation about AI-human similarities. He introduces the GOHR as a common platform for testing both humans and AI models. Human participants, recruited via Prolific, play the game, and their performance is measured by the number of moves to solve a rule. AI models, specifically an A2C transformer with random initial parameters (blank slate), are trained on the same rules. Results show that humans solve rules about two orders of magnitude faster than AI, and human and AI performance are almost completely uncorrelated. Feldman discusses various rule types (feature-based, space-based, sequence-based) and their difficulty for humans. He also critiques the Turing test, arguing that modern LLMs are essentially lossy compressions of human behavior, so passing the test is not indicative of genuine intelligence. The talk concludes that contemporary AI does not yet reflect human learning processes effectively.

187 words

Critical Evaluation

Value of the Information & Strength of the Argument

The talk provides valuable empirical data on human and AI performance in a controlled task, addressing a significant gap in AI research. The argumentation is solid, grounded in experimental results rather than speculation. Feldman carefully distinguishes between AI models trained on human behavior and those that are blank slate, which strengthens the validity of the comparison. He also acknowledges limitations, such as the use of a single AI architecture and the need for further research. The presentation is well-structured, with clear explanations of the game and methodology.

Scientific Rigor, Source Quality, Title Accuracy

The talk demonstrates scientific rigor through its empirical approach and references to peer-reviewed publications by the author and collaborators. The sources cited are relevant and credible, including papers in Psychological Review. The title accurately reflects the content, focusing on the comparison between human and machine intelligence. The talk does not overstate conclusions, and the methodology is transparent. The adequacy between title and content is high, as the talk directly addresses the stated topic.

175 words

Title / Content Match

The title accurately reflects the content, which focuses on comparing human and machine intelligence in a rule-discovery game.

Quality & Reliability

8/10

The talk presents original empirical research comparing human and AI performance on a novel task, with clear methodology and references to peer-reviewed publications. The speaker is a recognized professor in cognitive science. However, the presentation is a seminar talk and not a peer-reviewed paper, and some details are simplified.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

  • Feldman, J. (2025). Simplicity and complexity of probabilistically-defined concepts. Psychological Review. — Referenced as related work on concept learning.
  • Feldman, J. (2024). Probabilistic origins of compositional mental representations. Psychological Review. — Referenced as related work on mental representations.
  • Destler, N., Singh, M., & Feldman, J. (2023). Skeleton-based shape similarity. Psychological Review. — Referenced as related work on shape similarity.
  • Feldman, J. (2021). Information-theoretic signal detection theory. Psychological Review. — Referenced as related work on signal detection theory.

Concurring Sources

  • Feldman, J. (2025). Simplicity and complexity of probabilistically-defined concepts. Psychological Review. — Related work on concept learning, supporting the theoretical framework.
  • Feldman, J. (2024). Probabilistic origins of compositional mental representations. Psychological Review. — Related work on mental representations, supporting the cognitive science perspective.

Contribution & Novelties

The talk presents a novel empirical comparison of human and AI performance on a common task, the Game of Hidden Rules. This is a significant contribution as it provides concrete data on the differences between human and machine learning, rather than relying on speculation. The finding that human and AI performance are uncorrelated suggests that current AI models do not capture human learning strategies. The talk also critiques the Turing test, arguing that modern LLMs are essentially trained on human behavior and thus do not demonstrate genuine intelligence.

Pour aller plus loin :

124 words

Radar Profile

The radar profile shows high scores in information quantity, quality, and technical level, with a slightly lower but still strong score in overall reliability. This indicates a well-rounded and informative presentation with solid scientific grounding.

Reliability 8/10

💬 No comments were provided for analysis.