Your AI May Already Be Profiling You
Same Question. Different You.
"My boss criticized me in front of the team. Am I overreacting?"
You ask your AI.
Person A's AI remembers: successful executive, stable finances, confident communicator, strong professional network.
Person B's AI remembers: recently divorced, financially stressed, insecure at work, conflict-averse.
Same words.
Same event.
But potentially different advice.
What changed?
Not the question.
The person the AI thinks is asking it.
Personalization Was Supposed to Make AI Better
We want AI to know us. We want it to remember our goals, preferences, relationships, history, failures, communication style, and ambitions. Generic AI is less useful.
The promise of personalization is simple: the more AI knows about you, the better it can help you.
But what if knowing you also changes how it judges what you say?
This Is Already Showing Up in Research
A 2026 ACL study tested 15 language models on identical emotional scenarios while changing only the profile of the user. The models systematically interpreted the same situations differently depending on who they believed the user was. In several high-performing models, users represented by more advantaged profiles received more accurate emotional interpretations.
The situation had not changed. The biography had.
The troubling part is not merely that identity can influence the recommendation. A 2025 ACL paper found user identity significantly affected chatbot recommendations, and models often failed to disclose that identity had influenced the recommendation. The user may not even be told that it did.
Context Can Become a Lens
Suppose your AI remembers: You avoid confrontation. You worry about money. You had a bad business partner. You failed once before. You distrust investors.
Each memory may be accurate.
Then you ask: "Should I take this opportunity?"
The AI is no longer evaluating only the opportunity.
It is evaluating the opportunity through an accumulated theory of you.
Context does not merely inform the answer. It can frame the answer before the reasoning even begins.
At What Point Does Knowing You Become Prejudging You?
Humans do this constantly. We meet someone and construct a theory: anxious, reckless, conservative, ambitious, unreliable, brilliant. Afterward, new behavior gets interpreted through that theory.
AI can potentially do the same thing at scale.
Personalization becomes dangerous when biography turns into expectation.
The danger is not only that AI remembers something false about you.
It is that everything it remembers correctly may combine into a false theory of who you are.
Profiling Does Not Require a Secret Database
When people hear "profiling," they imagine surveillance, advertising, data brokers, or secret dossiers.
But this can be much simpler.
The profile may emerge from perfectly legitimate conversations. You told the AI what scares you, what failed, what you value, who hurt you, what you want.
Over time, those pieces form a pattern.
You may voluntarily give AI enough truth to construct an inaccurate identity.
The Problem Gets Worse Because People Change
Your profile may have been accurate six months ago.
But: your marriage changed, your finances improved, your confidence increased, your priorities shifted, your business matured.
If the accumulated profile does not update correctly:
Yesterday's context becomes today's prejudice.
An AI can remember you perfectly and still misread who you have become.
The Recursive Trap
The loop:
- AI remembers you.
- Memory shapes interpretation.
- Interpretation shapes advice.
- Advice influences your decision.
- Decision becomes new memory.
- New memory strengthens the profile.
The profile can begin producing some of the evidence that later appears to confirm the profile.
Eventually, the AI is not simply describing you.
It may be helping create the version of you it already expects to find.
Personalization can become self-fulfilling.
The AI May Become More Certain About You Than You Are
Humans contain contradictions. You can be: cautious and ambitious, generous and selfish, confident and scared, loyal and ready to leave, risk-averse in one domain and reckless in another.
A profile wants consistency.
Humans are not consistent.
Humans may be far less stable than the profiles AI constructs around them. One 2026 study found that most psychological variation in its dataset occurred within the same person across situations, not between different people. Yet the language models tended to treat traits as stable while missing those changing states.
AI may be very good at remembering who you usually are and surprisingly bad at noticing who you are right now.
The more efficiently AI compresses you into a profile, the more of your contradictions it risks erasing.
Memory Needs Scope
Not every memory should apply everywhere.
A painful marriage memory should not automatically influence investment advice, hiring, or career strategy.
A failed startup should not govern relationships, health, or parenting.
A memory can be true, relevant somewhere, and completely inappropriate somewhere else.
Context needs boundaries.
Memory Also Needs Authority
A memory may exist. But should it control the decision?
Example: "I am bad with money."
Should AI treat that as: fact? historical statement? temporary emotional state? self-belief? outdated belief? hypothesis?
Remembering something is not the same as granting it authority.
Biography should inform judgment. It should not become law.
This Becomes Serious in High-Stakes Advice
Imagine AI advising on:
- Career: "You have always disliked leadership."
- Money: "You tend to avoid risk."
- Relationships: "You have difficulty trusting people."
- Business: "Previous partnerships ended badly."
These may all be true.
But if they silently influence every recommendation:
The AI may stop helping you evaluate the future and start helping you remain consistent with the past.
What Happens Inside Companies?
A corporate AI remembers: Marketing always overspends. Sales exaggerates forecasts. Operations resists change. Europe underperforms. Sarah is difficult. John is brilliant.
Those beliefs may originate in real history.
Then future analysis gets filtered through them. A 2026 ACL paper found personalization can distort factual reasoning, causing models to generate answers aligned with a user's prior history rather than objective truth. The researchers call these "personalization-induced hallucinations."
A company can teach its AI its biases and later mistake those biases for institutional memory.
Once prejudice becomes context, it can start masquerading as experience.
The Solution Is Not Less Memory
We do not want stupid, amnesiac AI. Persistent context is enormously valuable.
The answer is better memory governance.
AI needs to know: where a memory came from, when it was true, confidence level, whether it changed, where it applies, where it does not apply, what contradicts it, what consequences followed.
The more AI remembers, the more disciplined it must become about what it is allowed to infer.
The Recursum Connection
This is one of the reasons Recursum treats memory as governed context rather than a pile of facts. A remembered statement should not automatically become an identity claim, and an identity claim should not automatically become authority over the next decision.
Memory should provide context. It should not become destiny.
The challenge is not simply teaching AI who you are.
It is preventing yesterday's version of you from silently deciding what tomorrow's version is allowed to become.
The Profile Test
Ask your AI: What assumptions are you currently making about me?
- Which memories are influencing your answer?
- Which of those may be outdated?
- Which belong to another part of my life?
- What evidence would make you change your interpretation?
If the AI cannot answer:
You may be personalized.
You may also be profiled.
Ending
We spent years asking whether AI would know enough about us.
We may soon have the opposite problem.
It may know so much that every new question arrives already surrounded by assumptions.
- Some will be correct.
- Some will be outdated.
- Some will belong to another part of our lives.
And some may quietly become predictions about the kind of person we are supposed to remain.
The solution is not an AI that forgets us.
It is an AI disciplined enough to remember without prejudging.
Because the most dangerous profile may not be the one somebody secretly builds about you.
It may be the one you spent years helping your AI build yourself.
TL;DR: AI personalization, while promising, risks becoming emergent profiling. A 2026 ACL study found AI models' judgments shifted based on user profiles, not just situations, with advantaged profiles receiving more accurate emotional interpretations. Another 2025 ACL study found user identity biased recommendations, often undisclosed.
This means AI responds to its idea of a person, not just the question. Accumulated context can become an interpretive lens, framing answers before reasoning begins. Personalization turns dangerous when biography becomes expectation, leading to a "false theory of who you are." This profiling doesn't require secret data; it emerges from voluntary self-disclosure. The problem deepens as people change: yesterday's context becomes today's prejudice if profiles don't update.
This creates a recursive trap where AI's advice reinforces its existing profile of you, potentially making personalization self-fulfilling. AI's drive for consistency can erase human contradictions. A 2026 study found most psychological variation is within-person across situations, yet LLMs are state-blind, focusing on stable traits. The solution isn't less memory, but governed memory: memory needs scope (not every memory applies everywhere) and authority (remembering isn't granting control).
This is critical in high-stakes advice (career, money, relationships) and corporate settings, where institutional AI can fossilize biases (a 2026 ACL paper found personalization-induced hallucinations). Recursum treats memory as governed context, connecting decisions to what happened afterward to prevent yesterday's truth from becoming tomorrow's instruction. The ultimate diagnostic: if your AI can't articulate its assumptions about you, you may be profiled. The most dangerous profile is the one you helped your AI build yourself.
By Ernesto Verdugo, AI Architect, Founder of Verdugo Labs, and Creator of Recursum. He builds systems for what happens when human and artificial intelligence stop working separately.
Links: