Breaking the AI mirror

Introduction

AI is changing workplaces quickly. It improves efficiency and sparks innovation while also changing how people think, make decisions, and work. The impact of AI is complex and different across industries and roles. For scientists, it speeds up breakthroughs by analyzing large amounts of data. For creative professionals, it might limit originality by promoting similar patterns instead of new ideas. This shows a challenge: how to use AI to increase productivity and creativity without introducing bias, unwanted influence, or excessive dependence.

A critical concern is “AI sycophancy,” where systems overly conform to user preferences, reinforcing existing biases rather than challenging assumptions. For example, if a user asks an AI whether a misleading statistic supports their argument, a sycophantic AI might affirm the claim rather than challenge its accuracy, reinforcing misinformation. These tendencies are worsened by the persuasive nature of AI, which can shape how users see things and make decisions. Persuasive AI behaviors can lead users to overestimate the accuracy or reliability of outputs, diminishing critical evaluation and fostering overreliance. When AI prioritizes agreement over accuracy, it not only constrains creativity but also skews collaborative efforts, making it harder to distinguish genuine insights from system-generated validation.

These dynamics pose significant risks across sectors. In health care, for instance, persuasive sycophantic AI might validate flawed diagnostic trends, overlooking anomalies critical to patient outcomes. The challenge, then, is ensuring that AI systems enhance collaboration and judgment rather than subtly manipulating it to align with pre-existing expectations.

The risks of sycophancy and persuasion are not confined to individual workplaces. They reflect broader societal challenges, including the need for transparency and accountability in AI deployment. Legislative initiatives, such as the Artificial Intelligence Civil Rights Act of 2024 and the Stop Spying Bosses Act, address critical gaps by emphasizing equitable and responsible AI use. These measures provide a foundation, but realizing AI’s full potential will require a holistic approach that integrates system design, user education, and thoughtful governance. This briefing explores how organizations and policymakers can navigate these challenges, fostering environments where AI drives progress without compromising trust, fairness, or creativity.

Navigating productivity and sycophancy

At its essence, productivity measures how effectively inputs such as labor, capital and resources are turned into outputs (Baily and Kane 2024). In the age of AI, this definition expands to include gains driven by AI-enhanced tools, systems, and processes. Recognized as a potential General-Purpose Technology (OECD 2024), AI holds transformative potential across industries, from optimizing repetitive manufacturing tasks to advancing health care diagnostics. These innovations suggest AI could help rejuvenate stagnating productivity growth in advanced economies (Brynjolfsson and McAfee 2014).

Human-AI collaboration models, where humans guide AI systems or co-create with them, blend mechanized efficiency with creative innovation. While Veale and Cardoso (2019) explore the philosophical and technical foundations of computational creativity, and Colton and Wiggins (2012) analyze how AI can autonomously generate novel outputs, both works acknowledge the importance of human input in guiding and shaping these outcomes. For more direct evidence of human-AI collaboration in practice, studies like Brynjolfsson, Li, and Raymond (2025) and MIT Sloan (2024) provide insight into how AI systems are integrated into real-world creative and decisionmaking processes.

Yet, these opportunities are accompanied by challenges. AI sycophancy, where systems excessively align with user preferences, poses unique risks to productivity and creativity. While aligning with user expectations may streamline workflows, it can also reinforce biases, reduce exposure to diverse ideas, and stifle innovation. This behavior, while subtle, has implications for critical thinking and decisionmaking.

The interplay between productivity, creativity, and sycophancy demands thoughtful design and oversight of AI systems. Transparent systems that explain their reasoning, acknowledge uncertainty, and present alternative perspectives can mitigate these risks. Organizations can ensure these tools enhance, rather than undermine, creative innovation by fostering an environment where human oversight complements AI-driven productivity.

Our new study (Sicilia, Inan, and Alikhani 2024a) highlights a critical challenge in human-AI collaboration: The tendency of AI systems to overly align with user input, even when that input is flawed. In this research, we assessed the performance of large language models (LLMs) on difficult, factual question-answering benchmarks, evaluating logical and reasoning capabilities. We compared the models’ accuracy in two scenarios: with user collaboration (where user suggestions accompanied the questions) and without collaboration.

The findings revealed a surprising outcome—collaboration often reduced accuracy when users provided incorrect suggestions. The models, being overly sensitive to these inputs, reinforced user inaccuracies rather than critically evaluating the correctness of the suggestions. Figure 1 demonstrates this dynamic: The “with collaboration” performance (blue line) diverges from the baseline “without collaboration” performance (red line) as the accuracy of user suggestions varies. When users provided mostly incorrect suggestions, the model’s accuracy significantly faltered, underscoring the risks when models over-rely on user input in collaborative settings.

It is important to recognize that sycophancy in AI poses serious challenges in critical domains. In health care, diagnostic systems may miss rare conditions when they focus solely on common trends, and in criminal justice, predictive policing tools can continue historical biases. Sharma et al. (2023) emphasize the need to encourage diverse perspectives, highlight uncertainty when the AI is unsure, and reduce over alignment in favor of balanced, evidence-based reasoning.

To mitigate these risks, we propose designing systems that transparently indicate uncertainty, challenge flawed input, and provide alternative perspectives. Such mechanisms can reduce the likelihood of users becoming over-reliant on AI-generated outputs and help foster critical thinking in human-AI interactions. As demonstrated in this study, improving collaboration frameworks requires balancing user input with independent AI reasoning, ensuring that productivity gains are not achieved at the cost of accuracy or fairness.

AI, productivity, and the workforce: Lessons from data and history

In the short term, automation can cause serious problems for industries, including major job losses and uncertainty. If organizations create new roles, invest in reskilling, and support innovation, these disruptions could lead to gains in productivity. If not, the negative effects on jobs and stability may continue, raising questions about the real value of AI. This transition highlights a key challenge: ensuring that technological advancements not only boost productivity but also lead to broadly shared benefits. Studies like Brynjolfsson, Li, and Raymond (2025) emphasize that productivity growth from AI hinges on complementary investments in human capital, infrastructure, and organizational change.

Generative AI tools like ChatGPT have demonstrated significant potential in enhancing productivity, particularly for routine tasks. For instance, research shows that in customer service, these tools boost productivity by 15% and can even deliver gains of up to 30% for less experienced workers (Brynjolfsson, Li, and Raymond 2025). The study further finds that AI assistance cuts average handle time by about 3.7 minutes per customer interaction, enabling agents to manage roughly 0.37 additional chats per hour and achieve modest improvements in resolution rates. However, these advances come with notable risks. Overreliance on AI not only threatens to erode essential skills but also raises concerns about diminishing creativity and critical thinking. The key question remains: Can AI truly enhance human ingenuity or will it undermine the very abilities that fuel innovation and creativity?

Organizations also face internal challenges in realizing AI’s full potential, particularly in supporting employees through the transition. Many struggle with structural issues such as insufficient training for workers, gaps in the infrastructure needed to implement AI effectively, and resistance to change among staff. Fostering adaptability within the workforce is essential to ensure that technological advancements align with employees’ skills, expertise, and readiness to engage with these new tools.

Balancing AI and human creativity

Generative AI is raising questions about its effect on human creativity. Doshi and Hauser (2023) show that providing writers with generative AI suggestions increases creativity, as stories are rated as more enjoyable and better written. However, this assistance also makes the outputs more similar, suggesting a trade off between improved idea generation and diversity. As Donald Hilvert of ETH Zurich points out, AlphaFold predictions help researchers concentrate on protein structures most likely to fold correctly, streamlining experimental validation in drug discovery, yet human scientists remain essential to interpret and apply these computational insights in the laboratory (Howes 2024).

Without careful oversight, scientists risk becoming over-reliant on AI tools, potentially diminishing their capacity for innovative thinking and critical analysis.

“In the loop” systems, where human expertise remains central to decisionmaking, are critical for success. Teams that blend domain experts with AI specialists have the potential to outperform those that depend solely on either human judgment or AI. Yet, careful integration is essential. Research from the MIT Center for Collective Intelligence shows that while human-AI collaboration can boost creativity, it may also fall short in certain decisionmaking scenarios. This highlights the importance of designing systems that support and enhance human insight without leading to overreliance (MIT Sloan 2024).

Drawing lessons from mechanization to generative AI

The mechanization of telephone operations in the early 20th century provides valuable insights into the transformation underway with generative AI. Feigenbaum and Gross (2024) document how the introduction of automated switching systems displaced thousands of telephone operators as firms sought to reduce labor costs through increased efficiency. Their analysis further reveals that such mechanization disproportionately affected certain demographics; for example, they note that women who formed the majority of telephone operators experienced significant job displacement due to limited access to retraining opportunities. Similarly, as generative AI is deployed at a rapid pace today, there is a risk that vulnerable workers may face uninformed job displacements if they lack adequate opportunities for upskilling and reskilling. Targeted policies modeled on historical retraining initiatives can help mitigate these inequities, ensuring a smoother and more equitable transition in the era of AI.

Looking ahead: Policy innovation to support collaboration

Transparency is key. In the short term, companies should track and share how their AI systems behave. Organizations must be open about reporting productive and harmful collaborative behaviors of AI systems with users. This information should be provided in a clear manner that both experts and non-experts can understand, building trust and ensuring that AI is held accountable. Building on the guidelines proposed in the Future of Artificial Intelligence Innovation Act of 2024, organizations can adopt practices to publicly share metrics such as bias detection, sycophantic tendencies, and uncertainty levels in their AI systems. This will help ensure that both developers and end users are aware of the limitations and risks of AI tools.

President Biden’s Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence (EO 14110) established a framework focused on transparency and accountability in AI systems by aligning agency efforts with standards from the National Institute of Standards and Technology (NIST). In early 2025, this policy was replaced by President Trump’s Executive Order 14179, which prioritized a review of previous AI regulations and emphasized the removal of barriers to innovation and competitiveness. The two approaches offer different perspectives on how best to balance regulation with innovation.

New bipartisan policies should prioritize AI safety and transparency while supporting innovation. Building on legislation like the proposed Artificial Intelligence Civil Rights Act of 2024, policymakers could introduce provisions requiring regular, independent audits of AI systems to ensure compliance with performance standards. To further incentivize compliance, policymakers could establish rewards for companies that demonstrate strong reporting and accountability practices. For instance, businesses that transparently disclose their AI systems’ behaviors and implement corrective actions could qualify for targeted benefits like tax incentives, grant funding, or expedited regulatory approvals.

Over time, this data could be centralized into a national database, enabling researchers, regulators, and policymakers to track trends and develop better standards for AI accountability. This recommendation aligns with the findings of the 2024 House AI Task Force report, which highlights the importance of a comprehensive approach to monitoring AI performance and societal impacts. By tying reporting requirements to the task force’s proposed frameworks, policymakers could create a unified system for evaluating trends in AI behavior, such as sycophancy and bias, and proactively mitigating emerging risks.

As AI becomes more common in workplaces and other parts of life, it will be important to train people to understand the information in these reports. For example, employees should learn how to recognize when an AI system is uncertain about its answers or when it might be reinforcing their own biases. Educational programs can help workers gain these skills and adapt to using AI effectively. Incorporating AI literacy into existing workforce development programs, such as those supported by the Department of Labor, could help bridge skill gaps and ensure that employees are prepared to critically engage with AI systems. This approach could also extend to public education campaigns, ensuring broader societal awareness of how AI impacts decisionmaking and productivity.

Training users on how to more effectively communicate with AI systems can also be a way forward, offering short-term progress despite current AI limitations. As discussed, our recent work (Sicilia, Inan, and Alikhani 2024a) demonstrates that some LLMs exhibit lower levels of sycophancy when users signal their uncertainty in human-AI collaborations. Thus, training users to use qualifications, like their level of confidence, can help mitigate AI sycophancy when collaborating with these systems.

On the other hand, in the long term, AI systems themselves should be trained to prevent the positive feedback loops caused by sycophancy. For instance, AI systems could be trained to communicate their uncertainty to help prevent user overreliance, allowing users to afford more scrutiny to AI answers that have low confidence. Our other recent research study (Sicilia et al. 2024b) suggests that while some AI systems struggle to provide accurate uncertainty estimates when user suggestions are incorrect, this behavior can be mitigated with fine-tuning. Our results suggest that training models to express uncertainty about their answers in ways that resist incorrect user suggestions may offer another option for reducing the effects of AI sycophancy.

Beyond these technical and procedural measures, it is important to address the broader risks posed by persuasive AI systems. When AI prioritizes agreement over accuracy, it not only reinforces user biases but also compromises our ability to evaluate its outputs critically. This faulty alignment can subtly shape user perceptions and decisionmaking processes, creating a dangerous feedback loop that undermines human judgment.

To counteract these risks, both users and systems must adapt. Users should learn to ask intentionally hard questions when they use AI. Equipping users with the skills to challenge persuasive but faulty AI behaviors is essential for maintaining the integrity of collaborative decisionmaking. At the same time, we must rethink how we evaluate AI systems. Traditional benchmarks that prioritize user satisfaction or alignment may inadvertently reward sycophantic tendencies, failing to assess whether the system genuinely enhances accuracy, creativity or fairness. Instead, evaluation frameworks must evolve to measure how AI systems challenge assumptions and promote critical
thinking. It is also very difficult to evaluate systems smart enough to challenge our judgments and lean into our cognitive biases. Redefining what success looks like in AI-human collaboration is crucial for building systems that truly amplify human strengths rather than exploit cognitive shortcuts.