Paper link: https://arxiv.org/abs/2502.04695

Abstract

Artificial Intelligence (AI) is becoming increasingly significant in sectors such as healthcare, finance, and law enforcement, influencing crucial decision-making processes.Transparency and explainability in AI systems are vital for their ethical use. Explainable AI (XAI) aims to connect complex machine learning models with end-users.

However, there is a significant challenge: the lack of standardized metrics to evaluate AI-generated explanations effectively. This paper discusses the issues of subjective and biased evaluation methods in XAI and highlights the need for reliable, scalable, and resistant evaluation metrics for regulatory compliance. By addressing these challenges, we can build trust in AI systems and ensure their responsible use across industries.

1. Introduction 

As Artificial Intelligence (AI) becomes increasingly integrated into sectors like healthcare, finance, and law enforcement, the decisions made by AI systems gain significant importance. Ensuring the transparency and explainability of these systems is crucial, which is where Explainable AI (XAI) comes in to connect complex machine learning models with users.

Despite advancements in XAI, a significant issue remains: the lack of standardized, reliable metrics to evaluate AI explanations. Madsen et al., 2024 point out that fragmented and subjective evaluation methods undermine the practical value and trustworthiness of XAI. The absence of universally accepted benchmarks complicates the assessment of various XAI approaches, hindering the adoption of transparent AI models in real-world applications.

2. Background and Context 

Explainable AI (XAI) is a field dedicated to making the decision-making processes of AI systems more interpretable and understandable. As AI technology becomes more prevalent in high-risk areas, ensuring transparency is essential for ethical, legal, and regulatory compliance. Without strong evaluation frameworks, stakeholders—such as regulators, businesses, and end-users—find it challenging to assess the reliability of AI explanations. This section highlights key concepts in XAI and their implications.

2.1 Interpretability, Explainability, and Feature Attribution

Interpretability refers to the ease with which a human can understand a model's decision-making process. Simpler models, like decision trees and linear regression, are inherently interpretable. In contrast, more complex deep learning models often require additional tools to provide explanations for their decisions. Explainability is focused on clarifying why a particular model made a specific decision, often employing techniques such as LIME (Local Interpretable Model-Agnostic Explanations) and SHAP (SHapley Additive exPlanations). Feature attribution methods, which identify the input features that contributed most to an AI model's output, are a crucial part of explainability frameworks.

2.2 Explainability Methods

Explainability methods can be broadly categorized into two main types:

1. Intrinsic Explainability: This refers to models that are inherently interpretable, such as linear regression and decision trees. These models allow users to easily follow the decision-making pathways.

2. Post hoc Explainability: This involves techniques applied after a model has been trained to generate explanations for its predictions. Examples of post hoc methods include SHAP, LIME, and Grad-CAM, which provide insights into how the model makes its decisions.

Among post-hoc explanation methods, only a few, such as Layerwise Relevance Propagation (Achtibat et al., 2024), and DLBacktrace (Sankarapu et al., 2024), extend to multi-modal settings, leaving a significant evaluation gap.

While post hoc methods offer flexibility, they also have limitations. These include reliance on approximations, potential inconsistency, and vulnerability to adversarial manipulations. The researchers note that "their reliance on approximations can lead to explanations that may not fully capture the complexity of a model's decision-making process."

2.3. The Role of Evaluation Metrics in XAI 

Evaluation metrics are essential for assessing the effectiveness and reliability of explainability methods. Without standardized criteria for evaluation, comparing different XAI techniques becomes challenging, and potential biases may emerge. To tackle these issues, researchers have identified six fundamental pillars for evaluating XAI.

The Six Pillars of XAI Metrics

Recent developments in the quantitative analysis of XAI explanations have provided researchers with a wide array of evaluation metrics to utilize. Most of these XAI metrics can be grouped into one of six categories, creating a structured framework for evaluating explanations:

1. Faithfulness: This metric measures how accurately an explanation reflects the model's true decision-making process. The aim is to ensure that the explanation faithfully represents the internal workings of the model.

2. Robustness: This assesses the stability and consistency of explanations when exposed to varying inputs and adversarial attacks. The focus is on maintaining the integrity of the explanation across different test conditions.

3. Localization: This metric evaluates the explanation's ability to highlight relevant regions or features in the input data that significantly influence the model's decision. For example, in image data, localization metrics indicate how effectively the explanation identifies the specific areas of the image that contribute to the model’s prediction.

4. Complexity: This measures the simplicity and comprehensibility of an explanation. Effective explanations should be straightforward enough for end-users to understand and utilize, avoiding unnecessary complexity that could impede decision-making.

5. Randomization (Sensitivity): This examines how sensitive explanations are to changes in input data or model parameters. These metrics ensure that the explanation does not become overly reliant on trivial input variations and remains stable when subjected to minor perturbations.

6. Axiomatic Metrics: This evaluates the fundamental properties of the explanation method itself, such as consistency, completeness, and the preservation of critical properties across different model architectures or tasks. These metrics are grounded in the theoretical foundations of XAI.

3. Challenges in XAI Metrics 

Despite advances in the field, several challenges continue to hinder the development of reliable metrics for Explainable Artificial Intelligence (XAI).

3.1 Neglect of Modern AI Models: Traditional XAI metrics were designed for simpler models and do not effectively capture the complexities of large language models (LLMs) and multi-modal AI systems. Many emerging AI models, such as autoregressive transformers, rely on deep layers of abstraction, rendering current XAI assessment tools inadequate for evaluating their decision-making processes.

3.2 Fragmentation and Inconsistency: The lack of standardization has led to a variety of evaluation methodologies that are often incompatible. Different industries and research groups prioritize various aspects of explainability, resulting in a fragmented landscape where findings cannot be easily generalized.

3.3 Manipulation Vulnerabilities: Evaluation outcomes can be affected by hyperparameters, which makes it easy to manipulate results. Developers might optimize explanations to meet specific benchmarks rather than genuinely enhancing the interpretability of their models. This practice undermines trust in XAI methods and complicates regulatory enforcement.

4. Key Requirements for Reliable Metrics 

Beyond the six pillars, truly reliable metrics must have several key attributes:

1. Transparency & Robustness 

Metrics must provide clear, consistent insights into how explanations align with model decisions. Transparency builds trust among stakeholders, while robustness ensures stability across different conditions, including adversarial inputs and data variations.

2. Adaptability & Tamper-Resistance

Metrics should be flexible across domains like healthcare and finance while resisting manipulation. Safeguards such as adversarial testing ensure explanations remain unbiased and reliable.

3. Scalability 

Evaluation methods need to effectively scale to handle the complexities of modern AI models, including large language models (LLMs), ensuring that explanations remain accessible and comprehensible as models increase in size and sophistication.

4. Regulatory & Legal Compliance

Metrics must align with evolving regulations (e.g., EU AI Act) to ensure AI decisions are transparent, auditable, and justifiable, particularly in high-risk domains like finance and healthcare.

5. Alternative Views 

While the standardization of metrics in Explainable AI (XAI) is widely accepted, some alternative perspectives highlight important considerations:

• Universal Metrics Are Unrealistic: Given the diversity of AI applications, it may not be feasible to adopt a single, universal metric across all domains. Different industries emphasize different aspects of explainability—financial AI models might prioritize auditability, while medical AI requires interpretability to assist human decision-making. This variation complicates the creation of one-size-fits-all evaluation criteria.

• The Role of Human Judgment: Some experts argue that qualitative assessments from domain specialists should complement automated metrics to ensure practical applicability. Automated evaluation frameworks may lack contextual understanding, whereas human judgment can provide critical insights that ensure explanations are useful and meaningful in real-world situations.
• Hybrid Approaches: Striking a balance between standardized core benchmarks and domain-specific evaluation frameworks may result in a more effective system for evaluating XAI. Instead of relying solely on universal metrics or subjective assessments, a combination of standardized core metrics and industry-specific adjustments can offer a more robust evaluation framework.

6. Call to Action 

To address the current gaps in the evaluation of Explainable Artificial Intelligence (XAI) and to ensure the development of reliable, transparent, and compliant AI systems, stakeholders must take the following actions:

1. Develop Core Benchmarks: Establish standardized metrics for assessing explainability to ensure consistency and comparability across different AI models.

2. Create Scalable, Tamper-Resistant Evaluation Methods: Develop computationally efficient techniques that prevent manipulation and ensure reliability across diverse applications.

3. Align XAI Metrics with Regulatory Requirements: Collaborate with policymakers to design evaluation frameworks that comply with legal and ethical mandates for AI.

4. Foster Collaboration: Encourage partnerships among academia, industry, and regulatory bodies to refine evaluation methodologies and build consensus on best practices.

7. Broader Implications 

Developing reliable metrics for explainable artificial intelligence (XAI) has significant implications beyond just technical performance:

• Regulatory Compliance: Clear and reliable metrics help organizations comply with regulations such as the EU AI Act. This prevents legal disputes and ensures responsible AI deployment.
• Auditability and Traceability: Organizations can demonstrate the fairness and reliability of their AI systems, which is essential for accountability in critical decision-making processes.
• Building Trust: Standardized evaluation of AI fosters confidence among users, regulators, and industry stakeholders, promoting broader acceptance of AI technologies.
• Advancing Ethical AI: Transparent and unbiased evaluation supports responsible AI adoption, helping to reduce the risk of bias and discrimination in AI-driven decisions.
• Fostering Innovation: A unified approach to assessing XAI encourages collaboration and progress in the field, accelerating the development of more explainable and user-friendly AI systems.

8. Conclusion

In conclusion, developing and adopting reliable metrics for explainable AI (XAI) is crucial for realizing its full potential. By establishing metrics that are resistant to manipulation, sensitive to context, and rooted in human judgment, we can ensure that AI explanations are meaningful, trustworthy, and compliant with evolving regulations.

This approach will not only advance the technical aspects of XAI but also encourage the responsible use of AI across various applications. Ultimately, this will foster a future where AI is both intelligent and accountable. The impact of these developments will be significant, influencing how AI is designed, deployed, governed, and shaping the overall relationship between humans and machines.