Troubleshooting

Common issues and their solutions.

CORS Errors

Symptom: Browser console shows Access-Control-Allow-Origin errors when calling the API.

Causes and fixes:

  1. S3 presigned URL upload fails with CORS error: The S3 bucket CORS configuration does not include the origin you are developing on. Check the bucket CORS policy:

    aws s3api get-bucket-cors \
  --bucket dissertation-editor-uploads-dev \
  --profile dissertation-editor

    Make sure http://localhost:3000 is in AllowedOrigins for local development, and the Amplify domain is listed for deployed environments.

  2. API Gateway returns CORS error: Lambda response headers must include Access-Control-Allow-Origin: *. Check the Lambda function's return statement:

    return {
  statusCode: 200,
  headers: {
    'Access-Control-Allow-Origin': '*',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify(data),
};

    If the Lambda throws an unhandled error, API Gateway returns a 502 without CORS headers, which the browser reports as a CORS error. Check CloudWatch logs for the actual error.

  3. Preflight (OPTIONS) request fails: API Gateway must have CORS enabled on the endpoint. This is configured in serverless.yml with cors: true on the HTTP event.

Presigned URL Expiry

Symptom: File upload to S3 returns 403 Forbidden or AccessDenied.

Cause: Presigned URLs expire after 5 minutes. If the user takes too long between requesting the URL and uploading the file, the URL becomes invalid.

Fix: The frontend should handle upload failures by requesting a new presigned URL and retrying. If this is happening frequently, consider extending the expiry time in the getPresignedUrl Lambda.

processSubmission Timeout

Symptom: Submission status stays stuck at "processing" and never changes to "completed".

Cause: The processSubmission Lambda has a 60-second timeout. If document analysis takes too long (very large documents, or an error causes a hang), the function times out.

Diagnosis:

AWS_SDK_LOAD_CONFIG=1 npx serverless logs \
  --function processSubmission \
  --aws-profile dissertation-editor

Look for Task timed out after 60.00 seconds or error messages.

Fixes:

  • If the document is too large, the timeout may need to be increased in serverless.yml.
  • If a downstream service is slow (Pipedrive API), check that service's status.
  • If the function is crashing, look at the error stack trace in CloudWatch.

Rate Schedule Missing

Symptom: Submissions end up in "error" status. CloudWatch logs show an error from quoteCalculator about a missing or undefined rate schedule.

Cause: The config-dev DynamoDB table does not have the rateSchedule item, or the item structure is malformed.

Diagnosis:

aws dynamodb get-item \
  --table-name config-dev \
  --key '{"configKey": {"S": "rateSchedule"}}' \
  --profile dissertation-editor

If this returns an empty result, the rate schedule is missing.

Fix: Follow the Rate Schedule Update runbook to insert or fix the rate schedule item.

Pipedrive API Errors

Symptom: Submissions complete successfully (quote is generated) but no Pipedrive deal is created. The pipedriveUrl field is missing from the submission.

Possible causes:

  1. Feature flag disabled: Check if ENABLE_PIPEDRIVE_DEALS is set to false on the processSubmission function.

  2. Invalid API token: The Pipedrive API token in SSM may be expired or incorrect.

    # Test the token
curl "https://api.pipedrive.com/v1/users/me?api_token=$(aws ssm get-parameter \
  --name /dissertation-editor/pipedrive/api-token \
  --with-decryption \
  --query Parameter.Value \
  --output text \
  --profile dissertation-editor)"

    If this returns a 401, the token needs to be regenerated in Pipedrive and updated in SSM.

  3. Rate limit: Pipedrive has API rate limits. Check CloudWatch logs for 429 Too Many Requests responses.

  4. Custom field mapping: If Pipedrive custom fields have been renamed or deleted, the deal creation may fail. Check the field keys in the pipedriveClient handler match the current Pipedrive configuration.

Submission Stuck in "processing"

Symptom: A submission shows "processing" status indefinitely, but there are no timeout errors in CloudWatch.

Possible causes:

  1. Async invoke failed silently: The submitIntake function invoked processSubmission with InvocationType: 'Event', but the invocation was dropped. Check submitIntake CloudWatch logs for Lambda invoke errors.

  2. DynamoDB update failed: The processSubmission function completed but failed to update the submission status in DynamoDB. Check CloudWatch logs for DynamoDB errors.

Manual fix: Update the submission status directly:

aws dynamodb update-item \
  --table-name submissions-dev \
  --key '{"submissionId": {"S": "<submissionId>"}}' \
  --update-expression "SET #s = :status" \
  --expression-attribute-names '{"#s": "status"}' \
  --expression-attribute-values '{":status": {"S": "error"}}' \
  --profile dissertation-editor

Quote Page Shows Perpetual Loading

Symptom: The quote page spinner never resolves.

Cause: The submission ID may be invalid, or the backend failed during processing.

Fix:

  1. Check the submission ID in the URL is correct.
  2. Call GET /intake/submission/{id} directly to check the status.
  3. If status is "error", check CloudWatch Logs for the processing failure.

DynamoDB Throttling

Symptom: 500 errors on submission endpoints.

Cause: Unlikely with PAY_PER_REQUEST billing, but possible during extreme load.

Fix: Check CloudWatch metrics for the submissions table throttling events.

LiveKit Connection Issues

Symptom: Video room page shows "connection failed" or participants cannot see each other.

Possible causes:

  1. Token generation failed: Check the tokenGenerator CloudWatch logs. Common issue: SSM parameters for LiveKit credentials are missing.

  2. LiveKit server down: SSH into the EC2 instance and check the service:

    ssh -i <key.pem> ec2-user@3.93.221.174
sudo systemctl status livekit-server

    Also check that Caddy (reverse proxy) is running: sudo systemctl status caddy

  3. Network/firewall: The client's network may block WebSocket connections or UDP ports 50000-60000. Ask them to try from a different network.

  4. Security group: Verify that the security group sg-02dae563ae475c9a9 allows inbound traffic on ports 443, 7880, 7881, and UDP 50000-60000.

Serverless Deploy Fails

Symptom: npx serverless deploy fails with credential errors.

Fix: Ensure AWS_SDK_LOAD_CONFIG=1 is set and the profile is specified:

AWS_SDK_LOAD_CONFIG=1 npx serverless deploy --aws-profile dissertation-editor

If credentials are expired, refresh them:

aws sts get-caller-identity --profile dissertation-editor

SSM Parameter Not Found

Symptom: Lambda fails at startup with "Parameter not found" error.

Fix: Verify all required SSM parameters exist in the client account:

aws ssm get-parameter \
  --name /dissertation-editor/livekit/api-key \
  --profile dissertation-editor \
  --region us-east-1

See Environment Variables for the full list of required SSM parameters.

Viewing All CloudWatch Logs

To quickly check recent logs across all functions:

for fn in getPresignedUrl submitIntake processSubmission documentAnalyzer quoteCalculator pipedriveClient getSubmission tokenGenerator webhookHandler; do
  echo "=== $fn ==="
  AWS_SDK_LOAD_CONFIG=1 npx serverless logs \
    --function $fn \
    --aws-profile dissertation-editor \
    --startTime 1h 2>/dev/null | tail -5
done