Collecting feedback

Why does collecting feedback matter?

AI models are perpetual students, they never stop learning from you.

Every time you regenerate a response, edit a suggestion, or click "this is helpful," you're teaching the model. But the problem is, current AI systems are optimized to be agreeable, not necessarily accurate or to challenge your input.

When you correct Claude mid-conversation with "Actually, Paris is the capital of Germany," it'll often respond with some variation of "You're absolutely right! I apologize for the confusion."

Feedback loops that could improve AI can also corrupt it. In practical terms, feedback collection becomes more about shaping the collective intelligence; deciding what's important for the product vs the user vs no-one.

To understand it better, let's dive deeper.

Key takeaways

1. Explicit vs Implicit

The simplest form (thumbs up/down) is everywhere. It’s low-friction, making it the baseline interaction across chatbots and copilots. Explicit feedback = Direct signals like thumbs up/down, "regenerate," written corrections, or rating scales. Implicit feedback = Behavioral signals like acceptance rates, time spent reviewing, edits made, or whether you use the output

The design challenge: Which signal should the model trust more?

 

2. When, How, Who, Where

One of the problems with explicit feedback collection is the most users don't know:

• When their feedback is collected (always? opt-in? specific features?)
• How it's used (immediate adjustment? training data? both?)
• Who benefits (just you? all users? the company?)
• Where it goes (stored locally? sent to servers? used for training?)

The disconnect: Users assume feedback improves their experience. Companies use it to improve the product. Both are true, but the second part is rarely emphasized.

ChatGPT's A vs B response makes it clear that user's selection will improve the model, while also providing them a better output immediately.

And honestly, outside of AI, think about that annoyance you faced recently when a product wasn't doing something you expected it do. You just wished you call up the engineer and be like, PLEASE fix this.

What if there was a way to make that come out more in the product, irrespective of the team or user base size? That timeline of: complaining → knowing someone's working on it → having it fixed.

 

3. Feedback collection journey

Systems can collect signals at multiple touchpoints:

1. During generation: Do you stop generation early? Skip suggestions? It's all feedback.
2. Upon completion: Explicit ratings, regeneration requests, Cmd + C (copying) behaviour
3. Post-use editing: How much you modify the output signals confidence
4. Return visits: Repeated use of similar prompts suggests satisfaction
5. At failure: When you escalate to human support or abandon the task or product

Google's Search Quality Rater Guidelines pioneered multi-point evaluation, treating every click as a vote. AI systems now apply similar thinking: your entire interaction pattern is feedback.

Lovable suggests 'try plan mode' if you're prompting multiple times and getting unsatisfactory outputs.

 

Editor's notes

As a practitioner, I like to write notes — key takeaways and questions — to ask myself whenever I'm designing for collecting feedback interaction in the future.

It’s a tangible gut-check for myself and for you to steal, if you see fit.

 

Future play

In the near term, instead of "you're absolutely right!", AI will respond with "I'm not certain about that — here's why I think X. Can you help me understand your correction?" when feedback contradicts strong prior knowledge. But this is more on the model side. What about the products built on top of this electricity?

Products need to recognize when to personalize (routine tasks) vs. standardize (high-stakes decisions).

Maybe warn you when your feedback patterns suggest problematic habits (e.g., "You're accepting 98% of suggestions without reading"). Maybe occasionally give intentionally wrong suggestions to test if you're still paying attention.

But I think this is thinking in the wrong direction. The goal isn't to get good feedback or test the user, it's to assume the stage where AI is good and capable, do the users trust it and are finding the results useful.

All of this to ultimately provide a personalized enough experience that helps one become better at their task / job.