🧠 Unintended Consequences for Consultants Using AI

Kia ora, Namaskaram 🙏🏾

AI can enhance the speed and accuracy of your research project.

But it can also distort what you recommend to your client.

If research is part of your work, you’re probably using AI for tasks like:

• Summarising research, academic articles and case studies

• Crafting questions, ideas or solutions to recommend

• Drafting reports and presentation material

For good reason: AI chatbots like ChatGPT and Perplexity, research platforms like Elicit, Scite and Consensus, and even tools for AI-generated presentation like Gamma makes our research process faster.

Consequences if we don’t pause and reflect

AI tools will make your work more efficient, but without the right checks, it can also lead to biased insights, missing context and client recommendations that look good on paper—but are made up.

In a recent technical report, OpenAI states that “more research is needed” to understand why hallucinations are getting worse as deep reasoning models get more advanced.

I believe the issues aren’t just technical — they’re behavioural blind spots in how we use AI tools to guide our research.

📚 IN CASE framework to avoid behavioural blind spots

This framework was developed by the UK Cabinet Office Behavioural Science Team to help policymakers and communicators anticipate side-effects of changing behaviours.

It breaks down unintended consequences into six clear categories: Intended behaviours, Non-target audiences, Compensatory behaviours, Additional behaviours, Signalling, and Emotional impact. Together, these categories help you uncover everything from overloaded systems to subtle shifts in emotion or meaning that can quietly undermine your intervention.

Reference: Emery, A., Molière, L., Lang, P., Nicolson, M., & Prince, E. (2021). UK Cabinet Office Behavioural Science Team. IN CASE: A behavioural approach to anticipating unintended consequences.

SAMPLE INPUT:

While the IN CASE framework was originally designed for public policy interventions and communications, it’s just as relevant for how organisations, researchers and consultants adopt AI.

Prompting ChatGPT with this framework can help you uncover how AI tools might subtly influence your insights and recommendations.

You are a Behavioural Risk Advisor at a research consultancy.

Our team is writing a research report for Wellington City Council on how to 'nudge' households to reduce their water consumption during a drought. We're using AI to summarise evidence, generate ideas, and write recommendations.

Read and apply the IN CASE framework to identify six unintended consequences of relying on AI in this project. For each category, suggest one practical mitigation.

The six categories are: (1) Intended behaviours, (2) Non-target audiences, (3) Compensatory behaviours, (4) Additional behaviours, (5) Signalling, and (6) Emotional impact.

Focus on behavioural risks that could affect the quality, credibility, and impact of your recommendations on people's lives.

SAMPLE OUTPUT:

Absolutely — here’s how a Behavioural Risk Advisor could use the IN CASE framework to identify and mitigate unintended consequences of using AI in a water conservation report for Wellington City Council.


✅ I: Intended Behaviour

Risk:
AI-generated recommendations may be over-optimised for efficiency (e.g. quick wins like nudges or tech fixes), and miss long-term systemic change or behaviourally complex strategies.

Mitigation:
Use a human-in-the-loop review at each stage to apply local behavioural insights and test whether solutions reflect deeper drivers of household habits.

👥 N: Non-Target Audiences

Risk:
AI might unintentionally recommend interventions that disadvantage renters, low-income households, or people in culturally diverse communities — audiences not well-represented in training data.

Mitigation:
Stress-test all AI outputs with a representative diversity lens, asking: “Who might this not work for?” Bring in local community advisors to co-interpret key insights.

♻️ C: Compensatory Behaviours

Risk:
If households reduce water use in one domain (e.g., shorter showers), they may unconsciously increase it elsewhere (e.g., more laundry loads or garden watering). AI may miss these rebound effects.

Mitigation:
Layer in real-world behaviour patterns by reviewing past studies and consulting local data on household routines. Add checks for trade-offs when generating behavioural ideas.

➕ A: Additional Behaviours

Risk:
Households may interpret recommendations as cues to take unrelated actions — e.g., installing expensive greywater systems without understanding usage trade-offs or maintenance risks.

Mitigation:
Pre-test messaging clarity using behavioural pre-mortems: “What might people do when they read this?” Include simple behavioural instructions alongside technical solutions.

📣 S: Signalling

Risk:
Overly polished or generic AI-generated content may signal that recommendations were “outsourced,” reducing public trust in the report or its relevance to Wellington.

Mitigation:
Make the human thinking visible — embed footnotes or annotations showing where local expertise or community insight shaped decisions.

💔 E: Emotional Impact

Risk:
AI-generated language may miss tone — sounding cold, moralising, or crisis-heavy — triggering shame, disengagement, or scepticism.

Mitigation:
Run tone checks for empathy, agency and relatability. Use behavioural testing (e.g., narrative A/B testing) to refine how messages land emotionally.

Reply with your questions, ideas or reflections on this prompt—I’d love to hear from you.

.

Want to join my next in-person Behavioural by Design Masterclass ?

Bookings open for my full-day workshop on 15 May 2025 in Wellington