We introduce SimulBench, a benchmark designed to evaluate large language models (LLMs) across a diverse collection of simulation scenarios, such as acting as a Linux terminal or playing text games with users. While these simulation tasks serve as effective measures of an LLM's general intelligence, they are seldom incorporated into existing benchmarks. A major challenge is to develop a shared evaluation environment for testing different LLMs in multi-turn interactions between users and AI. To tackle this issue, we suggest using a fixed LLM as a user agent to engage with multiple target LLMs under evaluation. To facilitate automatic assessment on SimulBench, GPT-4 is employed as the evaluator, tasked with reviewing the quality of the multi-turn dialogues between the user agent and the target LLMs. Our comprehensive experiments indicate that these simulation tasks continue to pose a significant challenge for even the most advanced open LLMs. For example, GPT-4-turbo outperforms Llama-2-70b-chat by a margin of 37.95%.

The ability of large language models (LLMs) to simulate complex tasks is pivotal in driving the evolution of AI towards achieving general intelligence. These models exhibit remarkable versatility by adeptly assuming a wide range of roles—from acting as a Linux terminal to serving as an investment manager—highlighting their adaptability across various domains. Such flexibility underscores their potential for broad implementation. Consequently, the development of a benchmark dataset for simulation tasks is imperative in nurturing LLMs' progression towards becoming true generalists.

Nonetheless, existing benchmarks do not fully evaluate this potential. Current evaluations mainly focus on single-turn, static interactions between users and LLMs. While MT-bench attempts to consider multi-turn interactions with 80 examples, its reliance on predefined second queries fails to effectively examine the dynamic responses of different LLMs when engaging with users in complex, long-horizon simulation tasks.

In addition, these benchmarks primarily concentrate on tasks related to general information retrieval and creative writing, with less emphasis on complex simulation abilities. The existing role-playing benchmarks tend to focus on replicating the language styles of famous characters, an aspect that barely begins to explore the full range of simulation capabilities. This highlights the need for more comprehensive benchmarks that can thoroughly assess the wide-ranging simulation potential of LLMs.

We have gathered 168 distinct simulation tasks that require ChatGPT to perform in a variety of roles. These roles include acting as a Linux terminal, an SQL executor, a player for text-based games such as tic-tac-toe, an interviewer for specific roles such as a software developer, a personal investment manager, a generator for passwords with particular constraints, an ASCII art creator, a predictor of chemical reactions, and more. Each task specification comes with a role description, some output requirements and an initial user request.

MT-bench is designed to test Language Models (LLMs) in a two-turn conversation, where the second turn is predefined. However, our SimulBench necessitates multiple turns between users and LLMs. Depending on the task types and context window limit, some tasks may involve conversations exceeding 5 turns, with the majority spanning over 2 turns. To replicate realistic usage scenarios of LLMs, we employ OpenAI's GPT-3.5 to simulate a user interacting continuously with the LLMs under evaluation. After gathering the dialogues between the user agent and each target LLM, we follow the methodology of previous studies, using GPT-4 to assess and rate the quality of these conversations.

Our study involved an analysis of four widely used open Language Learning Models (LLMs), specifically Llama-2-chat (7B/13B/70B) and Mixtral (8x7B). These models are often ranked highly on several existing leaderboards, such as the Chatbot Arena. Our findings revealed a substantial performance gap between these open-source LLMs and GPT-4-turbo. Even the largest open LLM, Llama-2-70b-chat, was surpassed by GPT-4 turbo by a margin of 37.95\% on the hard subset of SimulBench.

We also observed that LLMs, including GPT-4-turbo, tend to exhibit poorer performance in simulation tasks that necessitate objective outputs. For instance, GPT-4-turbo's performance on simulation tasks of simple but useful objective tools, such as generating a random password with requirements for characters, scored only 8.20, conspicuously dropped from 9.23 of the subjective ones. In addition, we noticed a significant decrease in LLMs' performance when simulation tasks demand long-horizon memory, complex reasoning, and compositional instructions, especially for open LLMs. Mixtral achieved 9.07 on general role-playing tasks, but only 7.43 in simulation tasks that required a self-contained system (such as a Linux terminal).