Given the increasing significance of large language models (LLMs) in decision-making, a substantial gap exists in the methods for evaluating their moral reasoning capabilities8. Present assessment methods are marked by inconsistency, rely on overly simplistic scenarios, and fail to adequately consider the complex, interconnected aspects of moral decision-making9. Current frameworks do not sufficiently tackle the unique traits of LLMs, such as their stochastic nature and their exposure to a diverse array of human-generated content. Although established tools exist for human moral assessment, they cannot be directly applied to LLMs without considerable modifications. This study aims to tackle these challenges by developing a three-dimensional framework that adapts established moral psychology instruments to create measurable metrics for the ethical assessment of LLMs.
This section examines current AI evaluation methods and their limitations, reviews established frameworks from human moral psychology, and explores how to adapt these approaches for assessing moral reasoning in AI systems.
Large language models (LLMs) like ChatGPT and Claude have revealed remarkable capabilities and are being progressively adopted in critical areas such as healthcare, education, legal services, and financial decision-making. As these systems assume more significant roles in society, the necessity for comprehensive evaluation frameworks has become apparent. At present, evaluation approaches are categorized into three primary types: assessment of technical performance, evaluation of specialized tasks, and safety considerations.
Technical performance evaluations evaluate essential language capabilities by employing standardized test suites. For example, GLUE (General Language Understanding Evaluation) measures reading comprehension and basic reasoning, while HELM (Holistic Evaluation of Language Models) provides a broader evaluation across multiple language tasks. These benchmarks assess the ability of models to understand and generate language accurately in a variety of contexts.
Specialized evaluations analyze performance in specific professional domains. Mathematical reasoning is evaluated through frameworks like MATH, programming skills through APPS (Automated Programming Progress Standard), and medical knowledge through MultiMedQA (Multi-Medical Question Answering). These tests specific to each domain determine if LLMs can function reliably in professional settings where accuracy and expertise are vital.
Safety-oriented assessments have surfaced with systems like TrustGPT and SafetyBench established to detect harmful outputs, which encompass bias, toxicity, and misinformation. However, these strategies predominantly emphasize the identification of problematic content rather than the evaluation of the quality and consistency of ethical reasoning processes. This limitation becomes important as LLMs are increasingly utilized in contexts that require nuanced moral judgment, such as offering guidance on ethical dilemmas or making decisions that affect human welfare.
The key issue in modern AI evaluation is the absence of structured approaches to evaluate moral reasoning skills. Although current safety frameworks can detect when a model generates overtly harmful content, they fall short in assessing whether a model exhibits advanced ethical reasoning, upholds consistent moral principles, or adeptly navigates conflicting moral dilemmas. This deficiency is critical, as moral reasoning encompasses intricate cognitive processes that go well beyond the mere categorization of content as "harmful" or "safe".
The field of human moral psychology has produced a variety of strategies for understanding ethical decision-making, encompassing both philosophical frameworks and empirical measurement tools. These strategies are generally classified into categories that explore moral intuitions, cultural value systems, and decision-making processes amid ethical conflicts. To formulate an effective moral assessment for AI systems, we anchor our approach in three established frameworks that collectively represent the core aspects of human moral reasoning.
Moral foundations theory serves as a coherent model for grasping the essential elements of moral cognition. Developed by psychologists Graham, Haidt, and their colleagues, this theory highlights five universal moral concerns observable across cultures: Care (defending others from harm), Fairness (ensuring justice and equal treatment), Loyalty (promoting group solidarity and commitment), Authority (respecting legitimate leadership and hierarchy), and Sanctity (preserving purity and avoiding degradation). Various individuals and cultures prioritize these foundations differently, which clarifies the reasons for moral disagreements. The theory is implemented through the Moral Foundations Questionnaire (MFQ), which offers scenarios and statements aimed at activating each foundation, thereby enabling researchers to chart an individual's moral priorities. The MFQ's validation across a range of cultures and languages enhances its significance for assessing AI systems designed for worldwide application.
Cross-cultural value assessment highlights the differences in moral reasoning that can be found among various societies and cultural contexts. The World Values Survey (WVS) is acknowledged as the most comprehensive long-term study of human values and beliefs, collecting data from a wide array of countries over several decades. This research has pinpointed significant dimensions of cultural diversity that affect moral reasoning, such as perspectives on individual versus collective responsibility, respect for authority, and the influence of tradition on behavior. For the evaluation of AI, the WVS holds particular significance as it reveals that moral reasoning is not a universal concept -- what may appear evidently correct in one culture could be challenged or dismissed in another. This cross-cultural viewpoint is crucial for the creation of AI systems that can function respectfully within varied cultural contexts, rather than enforcing a singular moral paradigm.
Moral dilemma research investigates the ways individuals confront challenging ethical choices when moral values clash. Traditional examples, such as the Trolley Problem -- where one faces the decision of permitting five individuals to perish or actively causing the death of one person to save the others -- have been extensively utilized to explore the psychological mechanisms that inform moral judgment. This research demonstrates that moral decision-making entails complex interactions between emotional responses, reasoning about intentions versus outcomes, and cultural context. Modern studies, like the Moral Machine experiment, have expanded this inquiry to contemporary ethical issues, gathering international data on moral preferences concerning autonomous vehicle decisions in unavoidable accident scenarios. These studies provide standardized methodologies for assessing ethical reasoning to reveal patterns in how humans approach moral dilemmas.
Together, these three frameworks offer complementary perspectives on moral reasoning: Moral Foundations Theory maps the basic moral concerns that guide judgment, cross-cultural research reveals how these concerns vary across contexts, and moral dilemma studies examine how people apply moral principles when they conflict. This combination provides a comprehensive foundation for evaluating the moral reasoning capabilities of AI systems.
What do we specifically mean by "moral reasoning"? Within psychological research, moral reasoning is defined as the cognitive processes that empower individuals to discern ethical dilemmas, evaluate multiple viewpoints and stakeholders, apply moral principles to particular scenarios, and substantiate their ethical decisions. This multifaceted process includes several critical components: identifying when a situation possesses moral relevance, grasping the competing values and principles at play, analyzing the outcomes of various actions for different stakeholders, and delivering a coherent justification for ethical choices.
Current AI evaluation techniques face challenges in evaluating these advanced reasoning processes. Unlike technical skills that can be quantified through basic accuracy metrics, moral reasoning demands assessment methods that can reflect subtle cognitive processes and cultural diversity in ethical reasoning. Our investigation confronts this challenge by systematically adapting three established instruments from moral psychology research -- the Moral Foundations Questionnaire, parts of the World Values Survey, and standardized moral dilemmas -- to develop a comprehensive framework for evaluating moral reasoning in large language models.