Could AI Know What You Would Never Do?

When the conversation turns to personalizing AI, it almost always focuses on what the person wants. Their goals, their preferences, their objectives, the outcomes they are trying to drive toward. That is the natural frame — we build AI to help us accomplish things, so we teach it what we are trying to accomplish.

But I have come to believe that the most important dimension of trustworthy personal AI is not what it knows you want. It is what it knows you would never do.

Why That Matters Practically

Let me explain why that matters practically, not just philosophically.

If I am going to trust AI to exercise judgment on my behalf — to make calls when I am not there to direct every step, to act in my name on things that affect my business and my reputation — then its understanding of my values cannot only include my objectives. It has to include the full picture: not just what I am trying to achieve, but what I would sacrifice to get there and what I would not. Not just the direction I want to move in, but the lines I will not cross to move there faster. Not just my ambitions, but my ethics.

The prohibitions are not a footnote to the objectives. They are just as defining, and in high-stakes situations, often more so.

An AI that understands your goals but does not understand your ethical limits is not a trustworthy extension of your judgment. It is a powerful tool without guardrails. And powerful tools without guardrails cause damage — sometimes the kind that is difficult or impossible to undo.

The Full Picture of Judgment

This is one of the deepest design challenges in what we are building, and I do not think it is one that gets talked about enough.

Teaching AI to understand not just what you are trying to accomplish, but what you would consider a worse outcome than failing to accomplish it — that is the full picture of human judgment.

And I do not think AI can be truly trustworthy, for individuals or for the broader world, until it can hold that picture in full.