Evaluating language models for mathematics through interactions

Abstract

There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of evaluating LLMs relies on static pairs of inputs and outputs; this is insufficient for making an informed decision about which LLMs are best to use in an interactive setting, and how that varies by setting. Static assessment therefore limits how we understand language model capabilities. We introduce CheckMate, an adaptable prototype platform for humans to interact with and evaluate LLMs. We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics, with a mixed cohort of participants from undergraduate students to professors of mathematics. We release the resulting interaction and rating dataset, MathConverse. By analyzing MathConverse, we derive a taxonomy of human query behaviors and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness in LLM generations, among other findings. Further, we garner a more granular understanding of GPT-4 mathematical problem-solving through a series of case studies, contributed by experienced mathematicians. We conclude with actionable takeaways for ML practitioners and mathematicians: models that communicate uncertainty, respond well to user corrections, and can provide a concise rationale for their recommendations, may constitute better assistants. Humans should inspect LLM output carefully given their current shortcomings and potential for surprising fallibility.

Keywords: AI; human–computer interaction; language models; theorem proving.

Fig. 1.

(A) Contrasting typical static evaluation (Top) with interactive evaluation (Bottom), wherein a human iteratively queries a model and rates the quality of responses. (B) Example subset of the chat interface from CheckMate where users interact with an LLM. The participant can type their query (Lower Left), which is compiled in LaTeX (Lower Right). When ready, the participant can press “Interact” and have their query routed to the model. The user can continue for multiple interactions. The entire chat history is presented for the user to refer to. When the user is done with their interaction, they can press “Done with interaction” to be proceed to rating the interaction. The participant also sees the problem text above the chat window; we include full screenshots in SI Appendix, Figs. S2 and S3.

Fig. 2.

(A) Counts of model preference ratings postinteractions; participants ranked models based on preference as a mathematical assistant (lower rank is better). Ties were allowed and are included: Participants were permitted to assign the same rank to multiple models (SI Appendix, Additional Survey Observations). (B) Mathematical correctness and perceived helpfulness scores (all scores are an integer $\in \{0,1,...,6\}$; higher is better) received for each model. Full details about the text associated with the scales of each score are included in SI Appendix, Additional Survey Details. (C) Comparing participants’ scores of the mathematical correctness against perceived helpfulness of each models’ generations. Each dot is a score for a single human–model interaction. We add slight jitter for visual ease given that points overlap. Interestingly, we observe cases where the perceived helpfulness and correctness of a generation diverge; that is, particular instances can be deemed incorrect yet somewhat helpful, or correct, but somewhat unhelpful. (D) The relationship between correctness and helpfulness scores and whether the step is terminal (i.e., the step after which the participant stopped interacting for a particular problem). The size of the bubbles indicates the number of that particular score pair (correctness, helpfulness). For a fixed score pair, the opacity indicates the ratio of stopping steps, that is, the number of terminal steps divided by the number of total steps.

Fig. 3.

(A) Query profiles as a function of the interaction step. Users prefer to ask for definitions or general mathematics questions, and to paste in the full text, in the first interaction, compared to correcting the model’s output, asking why, etc. in the second interaction. Interaction step 0 is the initial interaction; step 1 is the query made after receiving the first AI response to the query made in step 0. (B) Query profiles for the first interaction step (i.e., step 0) as a function of the amount of experience the user has with AI systems prior to participating.