The second-report test: which fix a wrong box in Ember Console needs

Ember Console is a session-report builder for coaches, in closed beta at $8 a month with 5 free sessions a month. When a box comes out wrong there are three fixes and they do different jobs. An edit changes this report. Changing the prompt changes what the box is asked for, and changing the eval changes what that box has to pass before you read it.

Shorten a long section in the editor, send the report, and next week the same section runs long again.

Which fix does a wrong box need?

Before you touch anything, ask whether next week’s report would come out the same way. That question is the second-report test, and the answer picks the fix for you.

1. Read the box against the standard you set.
2. Ask whether next week’s report repeats the fault.
3. A one-off: edit it and send the report.
4. A repeat, and the box was asked wrongly: change the prompt.
5. A repeat, and your minimum was loose: change the eval.

Search for a fix to AI-generated notes and the help pages stop at step 3. Notta’s support pages offer edit, regenerate and delete. None of the three reaches what next week’s section gets asked for.

One r/ChatGPT poster asked for a sentence to come out of a draft and got nothing: ChatGPT “either ignores it or forgets what I wrote 2-3 messages ago” (r/ChatGPT). One person, one chat window, not a rate.

Should you change the prompt or the eval?

Look at what the box wrote about. A box that answered a different question was asked the wrong question, so the prompt is where you go. If the subject was right and the writing came in under your minimum, nobody had told the box the minimum, and that is the eval.

Scoring one model’s output against written criteria has a name outside coaching software: Jeffrey Ip’s DeepEval guide calls it LLM-as-a-judge.

One prompt and one eval sit in every box, the box system I wrote about this week. What Ember’s eval inspects is still unpublished, and a check only reports on what it measures, like a symptom score that missed a better week.

What is not stated yet?

The edit is the fix I can say least about. Ember’s landing page says: “Refine the report directly in Ember’s editor, and your next report will be created with those past edits and preferences already in mind.”

Two questions on my own fact sheet are still blank.

1. What an edit changes for the next report: the prompt, the eval, or an example.
2. Whether that learning is per template, per client, or per coach.

I could invent a mechanism that sounds right and you could not check it. An edit fixes the report in front of you; the rest is a debt of mine.

Triage your first report box by box. Edit what will not happen again, point at the prompt or the eval for what happens twice, and ask me for the third answer once the sheet is filled.

Questions people ask next?

Why does AI keep making the same mistake?

A correction lands on the output. The instruction that produced it stays where it was, so the next run is asked the same thing and returns the same fault. One r/ChatGPT poster asked for a sentence to be cut and got nothing.

How do you fix AI generated notes without rewriting them?

Take one section at a time and ask whether next week’s notes come out the same way. A one-off gets an edit. A repeat gets a change to the instruction, or to the minimum that section has to clear before you read it.