# English Speaking Report Pipeline Design

## Scope

This design covers the runtime report pipeline for `src/views/Editor/EnglishSpeaking/preview/DetailedReport.vue` and `OverallReport.vue`, plus the backend report contract in `/Users/buoy/Development/gitrepo/cococlass-english-speaking-api`.

The goal is to separate per-round sentence evaluation from whole-session overall evaluation while keeping `/report` as the single frontend entry point for the final report page.

## Round Model

A practice round is:

```text
AI prompt/question for round N + student answer for round N
```

The AI message generated after the student answers round N is the prompt/question for round N+1, unless the practice is complete and the AI is only closing the conversation.

This means report grouping should not treat `AI prompt N + student answer N + AI follow-up` as one round. The follow-up belongs to the next round when it asks the next question.

For persistence:

- The opening greeting/question is stored as an AI message for `round = 1`.
- The student's first answer is stored as a student message for `round = 1`.
- The AI reply after that answer is stored as an AI message for `round = 2` when another round remains.
- On the final student answer, the AI closing response may be stored with the final round or a separate non-report role/metadata in a later design; it should not create another expected student evaluation.

## Problem

The current backend stores Azure pronunciation scores in `pronunciation_evaluation` and has a `content_feedback` JSON field, but the existing feedback shape uses `highlights / corrections / suggestions`. That shape is not correct for `DetailedReport` because those fields describe whole-session conclusions, not one student turn.

`DetailedReport` needs one feedback object per completed student answer:

```json
{
  "comment": "一句话点评",
  "betterExpression": "进阶表达"
}
```

`OverallReport` needs whole-session fields only after all expected student answers have completed sentence evaluation.

## Architecture

The report flow has three layers:

1. Sentence evaluation: one record per completed student answer.
2. Overall evaluation: one cached summary per completed session.
3. Report API: one `/report` response that returns both detailed and overall report data with a readiness status.

`/speak-stream` remains optimized for conversation responsiveness. It should not wait for report generation before sending `done`.

## Sentence Evaluation

Each student answer is evaluated asynchronously after `/speak-stream` writes the student message. The sentence feedback context includes the AI prompt for that same round and earlier dialogue text.

The sentence evaluation job runs in this order:

1. Call Azure Speech API for pronunciation scoring.
2. If Azure succeeds, call the sentence feedback LLM evaluator with the current turn's pronunciation scores.
3. If Azure fails after retries are exhausted, still call the sentence feedback LLM evaluator in text-only mode.
4. Store any available Azure scores and sentence feedback in `pronunciation_evaluation`.
5. If Azure fails, mark the pronunciation evaluation `failed`; `contentFeedback` may still be present from text-only feedback.

The sentence feedback evaluator input is:

```json
{
  "conversationHistory": [
    { "role": "ai", "content": "..." },
    { "role": "student", "content": "..." }
  ],
  "latestStudentTurn": {
    "round": 1,
    "content": "...",
    "pronunciation": {
      "accuracy": 82,
      "fluency": 76,
      "completeness": 88,
      "prosody": 70
    }
  },
  "grade": "五年级",
  "vocabulary": ["..."],
  "sentences": ["..."]
}
```

The conversation history for round N includes only dialogue text: prior AI/student turns and the latest student turn being evaluated. It should not include any previous rounds' Azure scores, previous sentence feedback, or the AI reply after the latest student turn. Previous scores and feedback are not needed for `DetailedReport`; they would bias the per-turn comment away from the student's current utterance.

The only pronunciation scores passed to the sentence feedback evaluator are the Azure scores for the latest student turn. If Azure failed after retries, `latestStudentTurn.pronunciation` should be `null` and the evaluator must generate text-only expression feedback.

In text-only mode, the evaluator must not mention pronunciation, fluency, intonation, stress, prosody, or any speech-score-dependent observation. It should focus on grammar, vocabulary, sentence completeness, communication intent, grade fit, target vocabulary, and target sentence patterns.

The evaluator output is:

```json
{
  "comment": "表达清楚，because 用得很好。",
  "betterExpression": "My favorite animal is the panda because it is cute."
}
```

Both keys are required. Values may be empty strings only if the evaluator fails validation and the backend chooses to store a normalized fallback; the preferred failure behavior is to store `null` for `contentFeedback`.

## Overall Evaluation

Overall evaluation is generated only when the conversation has ended and every expected student answer has a completed pronunciation evaluation.

The report API status should be one of:

```text
evaluating | ready | failed | incomplete
```

If the conversation has not ended or any evaluation is still `pending`, `/report` returns `status: "evaluating"` and should not generate the overall report. A failed pronunciation evaluation can still include text-only `contentFeedback`, but it blocks Full OverallReport.

When all sentence evaluations are completed, the backend generates and caches the overall report. Overall output owns:

- `aiComment`
- `highlights`
- `improvements`
- aggregate dimensions
- `overallScore`
- statistics such as average score, highest score, grammar error count, and excellent expression count

These fields must be based on all expected student answers, all Azure scores, all available sentence feedback objects, and the full dialogue transcript. If a sentence feedback LLM call failed but Azure succeeded, that answer still has completed pronunciation scores and `contentFeedback = null`; overall generation should tolerate that missing feedback instead of blocking forever.

Full OverallReport still requires all expected student answers to have completed Azure pronunciation scores. Text-only sentence feedback is useful for `DetailedReport`, but it is not enough to compute complete aggregate pronunciation scores.

## OverallReport Evaluator Contract

The OverallReport evaluator receives:

```json
{
  "conversationHistory": [
    {
      "round": 1,
      "role": "ai",
      "content": "...",
      "timestamp": "..."
    },
    {
      "round": 1,
      "role": "student",
      "content": "...",
      "timestamp": "...",
      "pronunciation": {
        "accuracy": 82,
        "fluency": 76,
        "completeness": 88,
        "prosody": 70
      },
      "contentFeedback": {
        "comment": "表达清楚，because 用得很好。",
        "betterExpression": "My favorite animal is the panda because it is cute."
      }
    }
  ],
  "grade": "五年级",
  "vocabulary": ["favorite", "adorable", "bamboo"],
  "sentences": ["My favorite animal is ... because ..."]
}
```

`pronunciation` and `contentFeedback` are included for student turns when available. Missing sentence feedback should not block OverallReport generation if Azure pronunciation scores are complete.

The prompt should ask the evaluator to:

- analyze the full multi-turn AI/student dialogue
- use every student's Speech API scores
- consider grade, target vocabulary, and target sentence patterns
- produce specific, verifiable highlights and actionable suggestions
- output valid JSON only

The raw LLM output should be:

```json
{
  "overall_evaluation": {
    "evaluation": "你的聊天特别棒！你能说出自己上学、下雨、和朋友玩这些真实故事，很不错呀！"
  },
  "highlights": [
    "积极主动参与对话，回应及时",
    "能够灵活使用多种句型结构进行交流",
    "发音清晰易懂，语调自然流畅"
  ],
  "suggestions": [
    "尝试使用更多形容词和副词丰富表达细节，如将 good 替换为 fantastic",
    "注意第三人称单数动词变化，确保主谓一致，如 He goes 而非 He go",
    "增加连接词使用使句子更连贯，如 because、however、in addition"
  ]
}
```

For robustness, the parser may also accept `overall_evaluation.chinese` as an alias for `overall_evaluation.evaluation`, because sample prompts sometimes use `chinese`. The normalized backend object should use one stable shape.

Normalized OverallReport fields:

```json
{
  "aiComment": "你的聊天特别棒！你能说出自己上学、下雨、和朋友玩这些真实故事，很不错呀！",
  "highlights": ["..."],
  "improvements": ["..."]
}
```

Mapping rules:

- `overall_evaluation.evaluation` maps to frontend `aiComment`
- `overall_evaluation.chinese` maps to frontend `aiComment` only if `evaluation` is absent
- `highlights` maps to frontend `highlights`
- `suggestions` maps to frontend `improvements`

## Overall Generation Gate

`/report` should decide whether Full OverallReport can be generated with an explicit gate:

```text
1. The conversation has ended.
2. At least one valid student answer exists.
3. Every saved student answer has a PronunciationEvaluation row.
4. Every PronunciationEvaluation is terminal.
5. Every PronunciationEvaluation has status = completed.
6. OverallReport has not already been generated and cached.
```

The first implementation should use saved student messages as the expected answer set, not `total_rounds`. `total_rounds` is the configured target, but it may not equal the final number of valid answers because time mode can expire, the user can end practice manually, ASR can fail before a student message is created, or the final AI closing message should not create another expected student answer.

Recommended status decisions:

```text
evaluating:
  - conversation has not ended; or
  - a saved student answer has no evaluation row yet; or
  - at least one evaluation is still pending/running/retrying

ready:
  - conversation has ended; and
  - at least one valid student answer exists; and
  - every saved student answer has completed Azure pronunciation scores; and
  - overall has been generated or can be generated now

failed:
  - conversation has ended; and
  - at least one valid student answer exists; and
  - any evaluation is failed after retries

incomplete:
  - conversation has ended; and
  - no valid student answers exist; or
  - the product later requires a minimum valid answer count and the session did not reach it
```

If the product later requires exactly three valid answers for a completed practice, that requirement should be added as a product-level gate. It should not be inferred from `total_rounds` inside the report generator without an explicit requirement.

## Bounded Retries

Automatic retries must be bounded and owned by the worker performing that stage.

`/report` must never create or retry sentence-evaluation work. It only reads sentence evaluation state and, when the Overall Generation Gate passes, generates or returns cached OverallReport data. This prevents frontend polling from starting duplicate sentence jobs.

Each external call should have an explicit maximum attempt count:

```text
Azure pronunciation scoring: max_attempts = 3
Sentence feedback LLM: max_attempts = 1 or 2
OverallReport LLM: max_attempts = 1 or 2
```

Attempt exhaustion must always produce a terminal state:

```text
Azure exhausted:
  evaluation.status = failed
  optional text-only contentFeedback may still be attempted

Sentence feedback exhausted after Azure success:
  evaluation.status = completed
  contentFeedback = null

Sentence feedback exhausted after Azure failure:
  evaluation.status = failed
  contentFeedback = null

OverallReport exhausted:
  report status = failed
  no automatic retry on the next /report poll
```

Once a stage reaches a terminal state, normal `/report` polling must not rerun it:

```text
evaluation.status = completed
evaluation.status = failed
overall report exists
overall generation failed
```

Future explicit retry can be added as a separate admin or user action, but it must not be implicit in report polling.

If the first implementation does not add persistent retry counters or job rows, retries should still be bounded inside the single background task:

```py
for attempt in range(1, max_attempts + 1):
    try:
        run_stage()
        mark_success()
        return
    except Exception as exc:
        log_attempt_failure(attempt, exc)
        if attempt == max_attempts:
            mark_terminal_failure()
            return
        await sleep(backoff_seconds(attempt))
```

OverallReport lazy generation needs the same protection. If the backend attempts OverallReport generation inside `/report`, a failed generation must set a terminal failed state for that report response path or use an in-process guard so frontend polling does not repeatedly invoke the OverallReport LLM.

## API Contract

`GET /report?sessionId=...` remains the frontend entry point.

The backend response should include:

```json
{
  "sessionId": "...",
  "topic": "...",
  "status": "evaluating",
  "rounds": [
    {
      "round": 1,
      "role": "student",
      "content": "...",
      "audioUrl": "...",
      "evaluation": {
        "status": "completed",
        "accuracyScore": 82,
        "fluencyScore": 76,
        "completenessScore": 88,
        "prosodyScore": 70,
        "wordAnalysis": null,
        "contentFeedback": {
          "comment": "表达清楚，because 用得很好。",
          "betterExpression": "My favorite animal is the panda because it is cute."
        }
      }
    }
  ],
  "overall": null
}
```

When the report is ready:

```json
{
  "status": "ready",
  "rounds": [
    {
      "round": 1,
      "role": "student",
      "content": "...",
      "audioUrl": "...",
      "evaluation": {
        "status": "completed",
        "accuracyScore": 82,
        "fluencyScore": 76,
        "completenessScore": 88,
        "prosodyScore": 70,
        "wordAnalysis": null,
        "contentFeedback": {
          "comment": "表达清楚，because 用得很好。",
          "betterExpression": "My favorite animal is the panda because it is cute."
        }
      }
    }
  ],
  "overall": {
    "overallScore": 85,
    "scoreLevel": "good",
    "percentile": 0,
    "dimensions": {
      "fluency": 82,
      "interaction": 80,
      "vocabulary": 78,
      "grammar": 84
    },
    "aiComment": "...",
    "highlights": ["..."],
    "improvements": ["..."],
    "statistics": {}
  }
}
```

The frontend adapter may keep the existing `DialogueReport` shape, but it should map this response into a clear internal model:

- `evaluation.sentenceEvaluations` for `DetailedReport`
- `evaluation.aiComment`, `highlights`, `improvements`, dimensions, score, and statistics for `OverallReport`

## Frontend DetailedReport Mapping

`SentenceEvaluation.feedback` should change from the old whole-session-like shape to:

```ts
feedback?: {
  comment: string
  betterExpression: string
}
```

`DetailedReport.vue` should display, for completed student answers:

- score badge derived from the four Azure scores
- four pronunciation rows
- `一句话点评`: `feedback.comment`
- `进阶表达`: `feedback.betterExpression`

If `contentFeedback` is missing, the expanded card should still show available pronunciation scores and omit the feedback blocks.

If pronunciation scores are missing but `contentFeedback` is present, the expanded card should show the feedback blocks and omit the pronunciation score rows.

## Frontend OverallReport Mapping

`OverallReport.vue` should not consume per-sentence `comment` or `betterExpression` directly. It should consume only the overall report fields generated after all sentence evaluations are complete.

The existing `highlights` and `improvements` arrays belong here.

## Polling Behavior

When the dialogue completes and the frontend enters report mode, it should call `/report`.

If `status !== "ready"`, the frontend should show a report-generating state and poll `/report` until either:

- `status === "ready"`
- `status === "failed"`
- `status === "incomplete"`
- a retry limit is reached
- the request fails with an unrecoverable error

The detailed report may be shown partially only if the UI explicitly distinguishes partial data from a completed report. The first implementation should prefer a single loading state until `ready` to avoid showing misleading aggregate results.

## Observability and Stall Diagnosis

The first implementation should use structured logs to diagnose where report generation is stuck. It should not add database job tables or persistent stage columns yet.

Every report-related log entry should include these fields when available:

```text
trace_id
session_id
round
message_id
evaluation_id
stage
event
duration_ms
status
attempt
error_code
error_type
error_message
```

`trace_id` should default to the session UUID so one command can retrieve the whole conversation/report chain.

`event` should use this fixed set:

```text
start | success | failed | timeout | skipped
```

`stage` should use fixed names rather than free-form strings:

```text
speak_stream.start_received
speak_stream.session_loaded
speak_stream.asr_started
speak_stream.asr_completed
speak_stream.student_message_saved
speak_stream.ai_reply_started
speak_stream.ai_reply_completed
speak_stream.conversation_committed
speak_stream.background_eval_started

sentence_eval.started
sentence_eval.azure_started
sentence_eval.azure_completed
sentence_eval.feedback_started
sentence_eval.feedback_completed
sentence_eval.committed
sentence_eval.failed

report.requested
report.evaluations_checked
report.waiting_sentence_evals
report.overall_started
report.overall_completed
report.returned
report.failed
```

Each external provider call should log a `start` event before the call and a `success`, `failed`, or `timeout` event after the call, with `duration_ms`.

Examples:

```json
{
  "trace_id": "session-uuid",
  "session_id": "session-uuid",
  "round": 2,
  "evaluation_id": 456,
  "stage": "sentence_eval.feedback_started",
  "event": "start",
  "attempt": 1
}
```

```json
{
  "trace_id": "session-uuid",
  "session_id": "session-uuid",
  "round": 2,
  "evaluation_id": 456,
  "stage": "sentence_eval.feedback_completed",
  "event": "success",
  "duration_ms": 842,
  "status": "completed",
  "attempt": 1
}
```

```json
{
  "trace_id": "session-uuid",
  "session_id": "session-uuid",
  "round": 2,
  "evaluation_id": 456,
  "stage": "sentence_eval.azure_completed",
  "event": "timeout",
  "duration_ms": 10000,
  "error_code": "AZURE_TIMEOUT",
  "error_type": "TimeoutError",
  "error_message": "Azure pronunciation assessment timed out",
  "attempt": 1
}
```

With this logging shape, debugging should be possible with simple filters:

```bash
rg 'session_id=session-uuid' logs/app.log
rg 'session_id=session-uuid.*event=(failed|timeout)' logs/app.log
rg 'session_id=session-uuid.*stage=report.waiting_sentence_evals' logs/app.log
```

If structured logs are not JSON-formatted in the current runtime, the same fields should still be emitted as stable `key=value` pairs.

If these logs are not enough to diagnose production stalls, a later implementation can add persistent `stage`, `stage_updated_at`, `attempt`, and `last_error` fields or a dedicated report job table.

## Error Handling

Sentence feedback has two modes:

- Scored mode: Azure succeeded, so the evaluator receives current-turn pronunciation scores plus dialogue text.
- Text-only mode: Azure failed after retries, so the evaluator receives dialogue text without pronunciation scores.

If Azure fails after retries, keep the pronunciation evaluation `status = "failed"` but still attempt text-only sentence feedback. If text-only feedback succeeds, store it in `contentFeedback`.

If Azure succeeds but the sentence feedback LLM fails or returns invalid JSON, keep the Azure scores and store `contentFeedback = null`. This keeps `DetailedReport` useful without inventing feedback.

If overall generation fails, keep the sentence-level data intact and return a non-ready status or a soft error field. The frontend should keep the report loading/error state separate from the completed conversation state.

## Testing

Backend tests should cover:

- sentence feedback evaluator parses `{ comment, betterExpression }`
- invalid evaluator JSON returns `None`
- Azure success followed by feedback success writes both scores and `content_feedback`
- Azure success followed by feedback failure writes scores and leaves `content_feedback` null
- Azure failure after retries attempts text-only feedback
- Azure failure after text-only feedback success stores `status = "failed"` and non-null `content_feedback`
- Azure failure after text-only feedback failure stores `status = "failed"` and null `content_feedback`
- `/report` does not generate overall data while any evaluation is pending
- `/report` generates or returns cached overall data once all sentence evaluations are complete
- report pipeline logs include `session_id`, `stage`, `event`, and `duration_ms` for provider calls

Frontend tests or focused manual checks should cover:

- `adaptReport` maps `contentFeedback.comment` and `betterExpression`
- `DetailedReport.vue` renders pronunciation scores without feedback
- `DetailedReport.vue` renders both feedback blocks when present
- report mode polls until `ready` before showing `OverallReport`

## Out of Scope

This design does not define the final overall-report prompt in detail. That prompt should be designed separately after the sentence-level report contract is implemented.

This design also does not change the conversation behavior of `/speak-stream`; the endpoint should continue returning transcript, AI tokens, and `done` without waiting for report jobs.
