Sentry SDK Integration Guide¶

Overview¶

Carver Horizon now includes comprehensive error tracking and performance monitoring using Sentry SDK across all services:

Backend: Django API server + 4 Celery services (3 workers + 1 scheduler)
Frontend: Next.js admin dashboard (client, server, and edge runtimes)

Key Features: - Automated error capture with zero manual instrumentation - Service-level tagging for easy filtering by container - Workflow tracking with execution IDs for cross-service correlation - Environment-aware sampling rates - Production-ready security controls

Architecture¶

Single Project Strategy¶

The integration uses one Sentry project with service-level tagging instead of multiple projects:

Sentry Organization: Carver Horizon
└── Project: carver-horizon-monitoring
    ├── Environment: development
    ├── Environment: staging
    └── Environment: production

Benefits: - Centralized issue tracking - Cross-service error correlation - Simplified management and billing - Easier to track distributed workflows

Service Identification¶

Each Docker container is tagged with: - service: Service type (backend, worker_summary, worker_workflows, worker_general, scheduler, admin_dashboard) - container: Docker container name - environment: Deployment environment (development/staging/production) - release: Git commit SHA or version tag - queue: For Celery workers - which queues they process

Example error tags:

{
  "service": "worker_workflows",
  "container": "reg_monitor_worker_workflows",
  "environment": "production",
  "release": "1.2.3-abc123f",
  "queue": "rss_queue,horizon_queue"
}

Configuration¶

1. Create Sentry Project¶

Sign up at sentry.io if you don't have an account
Create a new project for Carver Horizon
Copy the DSN from Project Settings > Client Keys (DSN)
Format: https://[key]@[org].ingest.sentry.io/[project-id]

2. Configure Environment Variables¶

Add to your .env file:

# Required for basic error tracking
SENTRY_DSN=https://your-public-key@your-org.ingest.sentry.io/your-project-id

# Required for release tracking
SENTRY_RELEASE=dev  # Use Git commit SHA in production

# Optional: For source map upload in production (frontend only)
SENTRY_ORG=your-org-name
SENTRY_PROJECT=carver-horizon-monitoring
SENTRY_AUTH_TOKEN=sntrys_[your-token]

3. Rebuild and Restart Services¶

# Rebuild containers with new dependencies
docker-compose build backend worker_summary worker_workflows worker_general scheduler admin_dashboard

# Restart all services
docker-compose down
docker-compose up -d

Backend Configuration¶

Integrations Enabled¶

Django Integration: - Automatic view exception capture - Database query tracking - Middleware execution tracing - Django signals monitoring - Cache operation tracking

Celery Integration: - Task execution tracking - Automatic retry and failure capture - Task performance monitoring - Celery Beat schedule monitoring

Redis Integration: - Redis operation tracking

Logging Integration: - Captures ERROR level logs automatically - Creates Sentry events for INFO and above

Sampling Rates¶

Development: 100% (traces_sample_rate=1.0)
Production: 10% (traces_sample_rate=0.1)

Ignored Errors¶

The following errors are filtered out to reduce noise: - KeyboardInterrupt - BrokenPipeError - ConnectionError - redis.exceptions.ConnectionError - celery.exceptions.TimeoutError - requests.exceptions.Timeout - botocore.exceptions.ClientError - psycopg2.OperationalError

Custom Context¶

Workflow Tracking - All DAG executions include: - execution_id: Unique identifier for tracking workflows across services - workflow_type: Type of workflow (dag_processing, rss_feed, horizon_keyword) - workflow_stage: Stage in workflow (execution, feed_processing, keyword_processing) - strategy_name: DAG strategy being executed

Example:

# In DAG processing tasks
set_tag('execution_id', execution_id)
set_tag('workflow_type', 'rss_feed')
set_tag('workflow_stage', 'feed_processing')
set_context('workflow', {
    'feed_id': feed_id,
    'entry_count': len(entries)
})

Frontend Configuration¶

Integrations Enabled¶

Client-side: - Browser error tracking - Performance monitoring (Web Vitals) - Session replay (with privacy controls) - Network request tracking

Server-side: - SSR error tracking - API route error tracking

Edge runtime: - Edge function error tracking

Privacy Controls¶

maskAllText: true - All text is masked in session replays
blockAllMedia: true - All images/videos blocked in replays
send_default_pii: false - No personally identifiable information sent automatically

Sampling Rates¶

Development: - Traces: 100% - Session Replay: 10% - Error Replay: 100%

Production: - Traces: 10% - Session Replay: 10% - Error Replay: 100%

Trace Propagation¶

The frontend is configured to propagate traces to: - localhost (development) - http://localhost:\d+ (development API) - https://*.carveragents.ai (production)

This enables distributed tracing between frontend and backend.

Testing¶

Backend Testing¶

Test Command:

docker-compose exec backend python manage.py test_sentry

What it tests: 1. Message capture 2. Exception capture 3. Custom tags and context 4. Configuration display

Expected output:

Testing Sentry integration...
✓ Test message sent to Sentry
✓ Test exception sent to Sentry
✓ Test custom tags and context sent to Sentry

Current Sentry Configuration:
  DSN: https://****@****.ingest.sentry.io/****
  Environment: development
  Release: dev
  Service: backend
  Traces Sample Rate: 1.0

✓ Sentry integration test completed successfully

Frontend Testing¶

Test Page: Navigate to http://localhost:3000/test-sentry

What it tests: 1. Configuration status display 2. Message capture 3. Error capture 4. Event ID display

Note: This page is only accessible in development mode (blocked in production for security).

Celery Worker Testing¶

Create a test task that triggers an error:

# Via Django shell
docker-compose exec backend python manage.py shell

>>> from tasks import test_sentry_worker
>>> test_sentry_worker.delay()

Then check Sentry dashboard for the error event with correct service and queue tags.

Verification Checklist¶

After deployment, verify:

Backend Services¶

[ ] Errors from backend service appear with service=backend tag
[ ] Errors from worker_summary appear with service=worker_summary, queue=summary_queue
[ ] Errors from worker_workflows appear with service=worker_workflows, queue=rss_queue,horizon_queue
[ ] Errors from worker_general appear with service=worker_general, queue=celery
[ ] Errors from scheduler appear with service=scheduler tag
[ ] Request context is captured (URL, method, headers)
[ ] User context is captured for authenticated requests

Frontend Services¶

[ ] Client-side errors appear with service=admin_dashboard, runtime=client
[ ] Server-side errors appear with service=admin_dashboard_ssr, runtime=server
[ ] Edge errors appear with service=admin_dashboard_edge, runtime=edge
[ ] Browser and device info is captured
[ ] Session replay is available for errors

Cross-Service Features¶

[ ] Can search by execution_id to see related errors across services
[ ] Environment tags match across all services
[ ] Release versions are consistent
[ ] Workflow context is captured (feed_id, strategy_name, etc.)

Troubleshooting¶

Errors Not Appearing in Sentry¶

Check DSN configuration:

# Backend
docker-compose exec backend python manage.py shell
>>> from django.conf import settings
>>> print(settings.SENTRY_DSN)

# Frontend
docker-compose exec admin_dashboard env | grep SENTRY

Verify DSN format: - Must start with https:// - Must contain @ symbol - Format: https://[key]@[org].ingest.sentry.io/[project-id]

Check Sentry initialization:

# Backend logs should show:
docker-compose logs backend | grep "Sentry initialized"

# Should see:
# INFO Sentry initialized for environment: development, service: backend

High Event Volume / Quota Issues¶

Adjust sampling rates in production:

Backend (backend/regulatory_monitor/settings.py):

# Lower the traces_sample_rate for production
traces_sample_rate = 0.05 if ENVIRONMENT == 'production' else 1.0  # 5% instead of 10%

Frontend (all three config files):

tracesSampleRate: SENTRY_ENVIRONMENT === 'production' ? 0.05 : 1.0  // 5% instead of 10%

Source Maps Not Working in Production¶

Check environment variables:

echo $SENTRY_AUTH_TOKEN
echo $SENTRY_ORG
echo $SENTRY_PROJECT

Verify build logs:

docker-compose logs admin_dashboard | grep -i sentry

Should see: "Uploading source maps to Sentry"

Missing Service Tags¶

Check environment variables for the specific service:

docker-compose exec worker_summary env | grep SENTRY_SERVICE

Should show: SENTRY_SERVICE=worker_summary

DSN Validation Warnings¶

If you see warnings like:

Invalid Sentry DSN format (must start with https://)

Check your .env file: - DSN must start with https:// - No spaces or quotes around the DSN - Complete DSN from Sentry dashboard

Production Deployment¶

Pre-Deployment Checklist¶

[ ] SENTRY_DSN set in production environment
[ ] SENTRY_RELEASE set to Git commit SHA
[ ] SENTRY_AUTH_TOKEN configured (for frontend source maps)
[ ] SENTRY_ORG and SENTRY_PROJECT set
[ ] Sampling rates adjusted for production volume
[ ] Sentry alerts configured for critical errors
[ ] Team members added to Sentry project

Setting Release Version¶

Using Git commit SHA:

# In deployment script
export SENTRY_RELEASE=$(git rev-parse --short HEAD)

# Or full SHA
export SENTRY_RELEASE=$(git rev-parse HEAD)

Using version tags:

export SENTRY_RELEASE=v1.2.3

Creating Releases in Sentry¶

To enable release tracking features:

# Install Sentry CLI
curl -sL https://sentry.io/get-cli/ | bash

# Create release
sentry-cli releases new -p carver-horizon-monitoring $SENTRY_RELEASE

# Associate commits
sentry-cli releases set-commits $SENTRY_RELEASE --auto

# Finalize release
sentry-cli releases finalize $SENTRY_RELEASE

# Deploy to environment
sentry-cli releases deploys $SENTRY_RELEASE new -e production

Alerting Configuration¶

Recommended Sentry alerts:

New Issue Alert
Trigger: First time an issue is seen
Environment: production
Action: Notify team channel
High Error Rate
Trigger: More than 10 errors per minute
Environment: production
Action: Page on-call engineer
Performance Degradation
Trigger: P95 response time > 2s
Environment: production
Action: Notify team channel
Celery Task Failures
Trigger: Task failure rate > 5%
Environment: production
Service: worker_*
Action: Notify team channel

Security Considerations¶

PII Protection¶

send_default_pii=False prevents automatic PII capture
Session replays have maskAllText=true and blockAllMedia=true
No email addresses, names, or sensitive data sent automatically

DSN Security¶

SENTRY_DSN is public and safe to expose in frontend
SENTRY_AUTH_TOKEN is private and must be kept secret
Auth token only needed for source map upload (build time, not runtime)

Error Message Review¶

Periodically review error messages in Sentry to ensure: - No credentials or API keys exposed - No internal infrastructure details leaked - No user personal information visible

Custom Context Sanitization¶

When adding custom context, avoid including: - Passwords or tokens - Email addresses - Phone numbers - Credit card information - Any user-provided sensitive data

Costs and Quotas¶

Sentry Pricing¶

Developer Plan: Free (5,000 errors/month, 10,000 performance units/month)
Team Plan: $26/month per member (50,000 errors/month, 100,000 performance units/month)
Business Plan: $80/month per member (custom limits)

Optimizing Quota Usage¶

High Volume Services: Lower sampling rates for services that generate many events:

# In settings.py
if SENTRY_SERVICE == 'worker_crawl':
    traces_sample_rate = 0.01  # 1% sampling for high-volume worker
else:
    traces_sample_rate = 0.1   # 10% for other services

Ignore Known Issues: Add to ignore_errors list in settings.py

Error Grouping: Configure issue grouping rules in Sentry dashboard to reduce duplicate issues

Advanced Features¶

Custom Performance Monitoring¶

Track specific operations:

from sentry_sdk import start_transaction

with start_transaction(op="task", name="process_large_feed"):
    with start_span(op="db", description="Fetch entries"):
        entries = FeedEntry.objects.filter(...)

    with start_span(op="llm", description="Summarize content"):
        summary = generate_summary(entries)

User Feedback¶

Collect user feedback on errors:

import * as Sentry from '@sentry/nextjs';

// After error occurs
Sentry.showReportDialog({
  eventId: lastEventId,
  title: 'Looks like we had an issue',
  subtitle: 'Our team has been notified',
});

Breadcrumbs¶

Add custom breadcrumbs for debugging:

from sentry_sdk import add_breadcrumb

add_breadcrumb(
    category='workflow',
    message='Starting RSS feed processing',
    level='info',
    data={'feed_id': feed.id}
)

Files Modified¶

Backend¶

backend/requirements.txt - Added sentry-sdk dependency
backend/regulatory_monitor/settings.py - Sentry initialization
backend/apps/core/management/commands/test_sentry.py - Test command
backend/dag_processing/executor.py - Workflow tagging
backend/dag_processing/tasks.py - Strategy tagging
backend/tasks.py - RSS and Horizon tagging

Frontend¶

admin-dashboard/package.json - Added @sentry/nextjs dependency
admin-dashboard/next.config.js - Sentry webpack plugin
admin-dashboard/sentry.client.config.js - Client-side config
admin-dashboard/sentry.server.config.js - Server-side config
admin-dashboard/sentry.edge.config.js - Edge runtime config
admin-dashboard/pages/test-sentry.tsx - Test page

Docker¶

docker-compose.yml - Environment variables for all services
.env.example - Sentry configuration template

Documentation¶

docs/SENTRY_INTEGRATION.md - This file
docs/SENTRY_FRONTEND_SETUP.md - Frontend-specific setup
docs/SENTRY_IMPLEMENTATION_SUMMARY.md - Implementation timeline

Support¶

Sentry Documentation¶

Main docs: https://docs.sentry.io/
Django: https://docs.sentry.io/platforms/python/guides/django/
Celery: https://docs.sentry.io/platforms/python/guides/celery/
Next.js: https://docs.sentry.io/platforms/javascript/guides/nextjs/

Internal Support¶

For configuration issues: Check .env file and docker-compose.yml
For integration issues: Review logs with docker-compose logs [service]
For quota issues: Adjust sampling rates or upgrade Sentry plan
For testing: Use test commands and test page

Changelog¶

2025-11-12: Initial Sentry SDK integration - Implemented backend integration (Django + Celery) - Implemented frontend integration (Next.js) - Added custom workflow tagging - Configured production-ready security controls - Created test commands and documentation