PPT/docs/superpowers/specs/2026-04-25-english-speaking-report-pipeline-design.md

English Speaking Report Pipeline Design

Scope

This design covers the runtime report pipeline for src/views/Editor/EnglishSpeaking/preview/DetailedReport.vue and OverallReport.vue, plus the backend report contract in /Users/buoy/Development/gitrepo/cococlass-english-speaking-api.

The goal is to separate per-round sentence evaluation from whole-session overall evaluation while keeping /report as the single frontend entry point for the final report page.

Round Model

A practice round is:

AI prompt/question for round N + student answer for round N

The AI message generated after the student answers round N is the prompt/question for round N+1, unless the practice is complete and the AI is only closing the conversation.

This means report grouping should not treat AI prompt N + student answer N + AI follow-up as one round. The follow-up belongs to the next round when it asks the next question.

For persistence:

• The opening greeting/question is stored as an AI message for round = 1.
• The student's first answer is stored as a student message for round = 1.
• The AI reply after that answer is stored as an AI message for round = 2 when another round remains.
• On the final student answer, the AI closing response may be stored with the final round or a separate non-report role/metadata in a later design; it should not create another expected student evaluation.

Problem

The current backend stores Azure pronunciation scores in pronunciation_evaluation and has a content_feedback JSON field, but the existing feedback shape uses highlights / corrections / suggestions. That shape is not correct for DetailedReport because those fields describe whole-session conclusions, not one student turn.

DetailedReport needs one feedback object per completed student answer:

{
  "comment": "一句话点评",
  "betterExpression": "进阶表达"
}

OverallReport needs whole-session fields only after all expected student answers have completed sentence evaluation.

Architecture

The report flow has three layers:

1. Sentence evaluation: one record per completed student answer.
2. Overall evaluation: one cached summary per completed session.
3. Report API: one /report response that returns both detailed and overall report data with a readiness status.

/speak-stream remains optimized for conversation responsiveness. It should not wait for report generation before sending done.

Sentence Evaluation

Each student answer is evaluated asynchronously after /speak-stream writes the student message. The sentence feedback context includes the AI prompt for that same round and earlier dialogue text.

The sentence evaluation job runs in this order:

1. Call Azure Speech API for pronunciation scoring.
2. If Azure succeeds, call the sentence feedback LLM evaluator with the current turn's pronunciation scores.
3. If Azure fails after retries are exhausted, still call the sentence feedback LLM evaluator in text-only mode.
4. Store any available Azure scores and sentence feedback in pronunciation_evaluation.
5. If Azure fails, mark the pronunciation evaluation failed; contentFeedback may still be present from text-only feedback.

The sentence feedback evaluator input is:

{
  "conversationHistory": [
    { "role": "ai", "content": "..." },
    { "role": "student", "content": "..." }
  ],
  "latestStudentTurn": {
    "round": 1,
    "content": "...",
    "pronunciation": {
      "accuracy": 82,
      "fluency": 76,
      "completeness": 88,
      "prosody": 70
    }
  },
  "grade": "五年级",
  "vocabulary": ["..."],
  "sentences": ["..."]
}

The conversation history for round N includes only dialogue text: prior AI/student turns and the latest student turn being evaluated. It should not include any previous rounds' Azure scores, previous sentence feedback, or the AI reply after the latest student turn. Previous scores and feedback are not needed for DetailedReport; they would bias the per-turn comment away from the student's current utterance.

The only pronunciation scores passed to the sentence feedback evaluator are the Azure scores for the latest student turn. If Azure failed after retries, latestStudentTurn.pronunciation should be null and the evaluator must generate text-only expression feedback.

In text-only mode, the evaluator must not mention pronunciation, fluency, intonation, stress, prosody, or any speech-score-dependent observation. It should focus on grammar, vocabulary, sentence completeness, communication intent, grade fit, target vocabulary, and target sentence patterns.

The evaluator output is:

{
  "comment": "表达清楚，because 用得很好。",
  "betterExpression": "My favorite animal is the panda because it is cute."
}

Both keys are required. Values may be empty strings only if the evaluator fails validation and the backend chooses to store a normalized fallback; the preferred failure behavior is to store null for contentFeedback.

Overall Evaluation

Overall evaluation is generated only when the conversation has ended and every expected student answer has a completed pronunciation evaluation.

The report API status should be one of:

evaluating | ready | failed | incomplete

If the conversation has not ended or any evaluation is still pending, /report returns status: "evaluating" and should not generate the overall report. A failed pronunciation evaluation can still include text-only contentFeedback, but it blocks Full OverallReport.

When all sentence evaluations are completed, the backend generates and caches the overall report. Overall output owns:

• aiComment
• highlights
• improvements
• aggregate dimensions
• overallScore
• statistics such as average score, highest score, grammar error count, and excellent expression count

These fields must be based on all expected student answers, all Azure scores, all available sentence feedback objects, and the full dialogue transcript. If a sentence feedback LLM call failed but Azure succeeded, that answer still has completed pronunciation scores and contentFeedback = null; overall generation should tolerate that missing feedback instead of blocking forever.

Full OverallReport still requires all expected student answers to have completed Azure pronunciation scores. Text-only sentence feedback is useful for DetailedReport, but it is not enough to compute complete aggregate pronunciation scores.

OverallReport Evaluator Contract

The OverallReport evaluator receives:

{
  "conversationHistory": [
    {
      "round": 1,
      "role": "ai",
      "content": "...",
      "timestamp": "..."
    },
    {
      "round": 1,
      "role": "student",
      "content": "...",
      "timestamp": "...",
      "pronunciation": {
        "accuracy": 82,
        "fluency": 76,
        "completeness": 88,
        "prosody": 70
      },
      "contentFeedback": {
        "comment": "表达清楚，because 用得很好。",
        "betterExpression": "My favorite animal is the panda because it is cute."
      }
    }
  ],
  "grade": "五年级",
  "vocabulary": ["favorite", "adorable", "bamboo"],
  "sentences": ["My favorite animal is ... because ..."]
}

pronunciation and contentFeedback are included for student turns when available. Missing sentence feedback should not block OverallReport generation if Azure pronunciation scores are complete.

The prompt should ask the evaluator to:

• analyze the full multi-turn AI/student dialogue
• use every student's Speech API scores
• consider grade, target vocabulary, and target sentence patterns
• produce specific, verifiable highlights and actionable suggestions
• output valid JSON only

The raw LLM output should be:

{
  "overall_evaluation": {
    "evaluation": "你的聊天特别棒！你能说出自己上学、下雨、和朋友玩这些真实故事，很不错呀！"
  },
  "highlights": [
    "积极主动参与对话，回应及时",
    "能够灵活使用多种句型结构进行交流",
    "发音清晰易懂，语调自然流畅"
  ],
  "suggestions": [
    "尝试使用更多形容词和副词丰富表达细节，如将 good 替换为 fantastic",
    "注意第三人称单数动词变化，确保主谓一致，如 He goes 而非 He go",
    "增加连接词使用使句子更连贯，如 because、however、in addition"
  ]
}

For robustness, the parser may also accept overall_evaluation.chinese as an alias for overall_evaluation.evaluation, because sample prompts sometimes use chinese. The normalized backend object should use one stable shape.

Normalized OverallReport fields:

{
  "aiComment": "你的聊天特别棒！你能说出自己上学、下雨、和朋友玩这些真实故事，很不错呀！",
  "highlights": ["..."],
  "improvements": ["..."]
}

Mapping rules:

• overall_evaluation.evaluation maps to frontend aiComment
• overall_evaluation.chinese maps to frontend aiComment only if evaluation is absent
• highlights maps to frontend highlights
• suggestions maps to frontend improvements

Overall Generation Gate

/report should decide whether Full OverallReport can be generated with an explicit gate:

1. The conversation has ended.
2. At least one valid student answer exists.
3. Every saved student answer has a PronunciationEvaluation row.
4. Every PronunciationEvaluation is terminal.
5. Every PronunciationEvaluation has status = completed.
6. OverallReport has not already been generated and cached.

The first implementation should use saved student messages as the expected answer set, not total_rounds. total_rounds is the configured target, but it may not equal the final number of valid answers because time mode can expire, the user can end practice manually, ASR can fail before a student message is created, or the final AI closing message should not create another expected student answer.

Recommended status decisions:

evaluating:
  - conversation has not ended; or
  - a saved student answer has no evaluation row yet; or
  - at least one evaluation is still pending/running/retrying

ready:
  - conversation has ended; and
  - at least one valid student answer exists; and
  - every saved student answer has completed Azure pronunciation scores; and
  - overall has been generated or can be generated now

failed:
  - conversation has ended; and
  - at least one valid student answer exists; and
  - any evaluation is failed after retries

incomplete:
  - conversation has ended; and
  - no valid student answers exist; or
  - the product later requires a minimum valid answer count and the session did not reach it

If the product later requires exactly three valid answers for a completed practice, that requirement should be added as a product-level gate. It should not be inferred from total_rounds inside the report generator without an explicit requirement.

Bounded Retries

Automatic retries must be bounded and owned by the worker performing that stage.

/report must never create or retry sentence-evaluation work. It only reads sentence evaluation state and, when the Overall Generation Gate passes, generates or returns cached OverallReport data. This prevents frontend polling from starting duplicate sentence jobs.

Each external call should have an explicit maximum attempt count:

Azure pronunciation scoring: max_attempts = 3
Sentence feedback LLM: max_attempts = 1 or 2
OverallReport LLM: max_attempts = 1 or 2

Attempt exhaustion must always produce a terminal state:

Azure exhausted:
  evaluation.status = failed
  optional text-only contentFeedback may still be attempted

Sentence feedback exhausted after Azure success:
  evaluation.status = completed
  contentFeedback = null

Sentence feedback exhausted after Azure failure:
  evaluation.status = failed
  contentFeedback = null

OverallReport exhausted:
  report status = failed
  no automatic retry on the next /report poll

Once a stage reaches a terminal state, normal /report polling must not rerun it:

evaluation.status = completed
evaluation.status = failed
overall report exists
overall generation failed

Future explicit retry can be added as a separate admin or user action, but it must not be implicit in report polling.

If the first implementation does not add persistent retry counters or job rows, retries should still be bounded inside the single background task:

for attempt in range(1, max_attempts + 1):
    try:
        run_stage()
        mark_success()
        return
    except Exception as exc:
        log_attempt_failure(attempt, exc)
        if attempt == max_attempts:
            mark_terminal_failure()
            return
        await sleep(backoff_seconds(attempt))

OverallReport lazy generation needs the same protection. If the backend attempts OverallReport generation inside /report, a failed generation must set a terminal failed state for that report response path or use an in-process guard so frontend polling does not repeatedly invoke the OverallReport LLM.

API Contract

GET /report?sessionId=... remains the frontend entry point.

The backend response should include:

{
  "sessionId": "...",
  "topic": "...",
  "status": "evaluating",
  "rounds": [
    {
      "round": 1,
      "role": "student",
      "content": "...",
      "audioUrl": "...",
      "evaluation": {
        "status": "completed",
        "accuracyScore": 82,
        "fluencyScore": 76,
        "completenessScore": 88,
        "prosodyScore": 70,
        "wordAnalysis": null,
        "contentFeedback": {
          "comment": "表达清楚，because 用得很好。",
          "betterExpression": "My favorite animal is the panda because it is cute."
        }
      }
    }
  ],
  "overall": null
}

When the report is ready:

{
  "status": "ready",
  "rounds": [
    {
      "round": 1,
      "role": "student",
      "content": "...",
      "audioUrl": "...",
      "evaluation": {
        "status": "completed",
        "accuracyScore": 82,
        "fluencyScore": 76,
        "completenessScore": 88,
        "prosodyScore": 70,
        "wordAnalysis": null,
        "contentFeedback": {
          "comment": "表达清楚，because 用得很好。",
          "betterExpression": "My favorite animal is the panda because it is cute."
        }
      }
    }
  ],
  "overall": {
    "overallScore": 85,
    "scoreLevel": "good",
    "percentile": 0,
    "dimensions": {
      "fluency": 82,
      "interaction": 80,
      "vocabulary": 78,
      "grammar": 84
    },
    "aiComment": "...",
    "highlights": ["..."],
    "improvements": ["..."],
    "statistics": {}
  }
}

The frontend adapter may keep the existing DialogueReport shape, but it should map this response into a clear internal model:

• evaluation.sentenceEvaluations for DetailedReport
• evaluation.aiComment, highlights, improvements, dimensions, score, and statistics for OverallReport

Frontend DetailedReport Mapping

SentenceEvaluation.feedback should change from the old whole-session-like shape to:

feedback?: {
  comment: string
  betterExpression: string
}

DetailedReport.vue should display, for completed student answers:

• score badge derived from the four Azure scores
• four pronunciation rows
• 一句话点评: feedback.comment
• 进阶表达: feedback.betterExpression

If contentFeedback is missing, the expanded card should still show available pronunciation scores and omit the feedback blocks.

If pronunciation scores are missing but contentFeedback is present, the expanded card should show the feedback blocks and omit the pronunciation score rows.

Frontend OverallReport Mapping

OverallReport.vue should not consume per-sentence comment or betterExpression directly. It should consume only the overall report fields generated after all sentence evaluations are complete.

The existing highlights and improvements arrays belong here.

Polling Behavior

When the dialogue completes and the frontend enters report mode, it should call /report.

If status !== "ready", the frontend should show a report-generating state and poll /report until either:

• status === "ready"
• status === "failed"
• status === "incomplete"
• a retry limit is reached
• the request fails with an unrecoverable error

The detailed report may be shown partially only if the UI explicitly distinguishes partial data from a completed report. The first implementation should prefer a single loading state until ready to avoid showing misleading aggregate results.

Observability and Stall Diagnosis

The first implementation should use structured logs to diagnose where report generation is stuck. It should not add database job tables or persistent stage columns yet.

Every report-related log entry should include these fields when available:

trace_id
session_id
round
message_id
evaluation_id
stage
event
duration_ms
status
attempt
error_code
error_type
error_message

trace_id should default to the session UUID so one command can retrieve the whole conversation/report chain.

event should use this fixed set:

start | success | failed | timeout | skipped

stage should use fixed names rather than free-form strings:

speak_stream.start_received
speak_stream.session_loaded
speak_stream.asr_started
speak_stream.asr_completed
speak_stream.student_message_saved
speak_stream.ai_reply_started
speak_stream.ai_reply_completed
speak_stream.conversation_committed
speak_stream.background_eval_started

sentence_eval.started
sentence_eval.azure_started
sentence_eval.azure_completed
sentence_eval.feedback_started
sentence_eval.feedback_completed
sentence_eval.committed
sentence_eval.failed

report.requested
report.evaluations_checked
report.waiting_sentence_evals
report.overall_started
report.overall_completed
report.returned
report.failed

Each external provider call should log a start event before the call and a success, failed, or timeout event after the call, with duration_ms.

Examples:

{
  "trace_id": "session-uuid",
  "session_id": "session-uuid",
  "round": 2,
  "evaluation_id": 456,
  "stage": "sentence_eval.feedback_started",
  "event": "start",
  "attempt": 1
}

{
  "trace_id": "session-uuid",
  "session_id": "session-uuid",
  "round": 2,
  "evaluation_id": 456,
  "stage": "sentence_eval.feedback_completed",
  "event": "success",
  "duration_ms": 842,
  "status": "completed",
  "attempt": 1
}

{
  "trace_id": "session-uuid",
  "session_id": "session-uuid",
  "round": 2,
  "evaluation_id": 456,
  "stage": "sentence_eval.azure_completed",
  "event": "timeout",
  "duration_ms": 10000,
  "error_code": "AZURE_TIMEOUT",
  "error_type": "TimeoutError",
  "error_message": "Azure pronunciation assessment timed out",
  "attempt": 1
}

With this logging shape, debugging should be possible with simple filters:

rg 'session_id=session-uuid' logs/app.log
rg 'session_id=session-uuid.*event=(failed|timeout)' logs/app.log
rg 'session_id=session-uuid.*stage=report.waiting_sentence_evals' logs/app.log

If structured logs are not JSON-formatted in the current runtime, the same fields should still be emitted as stable key=value pairs.

If these logs are not enough to diagnose production stalls, a later implementation can add persistent stage, stage_updated_at, attempt, and last_error fields or a dedicated report job table.

Error Handling

Sentence feedback has two modes:

• Scored mode: Azure succeeded, so the evaluator receives current-turn pronunciation scores plus dialogue text.
• Text-only mode: Azure failed after retries, so the evaluator receives dialogue text without pronunciation scores.

If Azure fails after retries, keep the pronunciation evaluation status = "failed" but still attempt text-only sentence feedback. If text-only feedback succeeds, store it in contentFeedback.

If Azure succeeds but the sentence feedback LLM fails or returns invalid JSON, keep the Azure scores and store contentFeedback = null. This keeps DetailedReport useful without inventing feedback.

If overall generation fails, keep the sentence-level data intact and return a non-ready status or a soft error field. The frontend should keep the report loading/error state separate from the completed conversation state.

Testing

Backend tests should cover:

• sentence feedback evaluator parses { comment, betterExpression }
• invalid evaluator JSON returns None
• Azure success followed by feedback success writes both scores and content_feedback
• Azure success followed by feedback failure writes scores and leaves content_feedback null
• Azure failure after retries attempts text-only feedback
• Azure failure after text-only feedback success stores status = "failed" and non-null content_feedback
• Azure failure after text-only feedback failure stores status = "failed" and null content_feedback
• /report does not generate overall data while any evaluation is pending
• /report generates or returns cached overall data once all sentence evaluations are complete
• report pipeline logs include session_id, stage, event, and duration_ms for provider calls

Frontend tests or focused manual checks should cover:

• adaptReport maps contentFeedback.comment and betterExpression
• DetailedReport.vue renders pronunciation scores without feedback
• DetailedReport.vue renders both feedback blocks when present
• report mode polls until ready before showing OverallReport

Out of Scope

This design does not define the final overall-report prompt in detail. That prompt should be designed separately after the sentence-level report contract is implemented.

This design also does not change the conversation behavior of /speak-stream; the endpoint should continue returning transcript, AI tokens, and done without waiting for report jobs.