Skip to content

System Architecture

Project Overview

Carver Horizon is a regulatory monitoring system that tracks regulatory updates from various sources (RSS feeds, HTML pages, keywords) and processes them through AI-powered workflows. The system consists of:

  • Backend: Django + Django Ninja API with Celery for background processing
  • Admin Dashboard: Next.js frontend for managing feeds, topics, and monitoring workflows
  • DAG Processing: Custom workflow engine for processing regulatory content through multi-step pipelines
  • Services: PostgreSQL, Redis, multiple Celery workers, Nginx

Backend Structure

Django Apps (by Domain)

  • apps/: Django apps organized by domain
  • core/: Base models, authentication (Firebase + JWT), RBAC, tags, crawl tracking
  • feeds/: RSS feeds, feed entries, topics, tags, workflow execution
  • horizon/: Keyword-based monitoring, SERP integration, summaries
  • config/: System configuration, schedules, prompt templates
  • summaries/: Summary generation workflows
  • users/: User management
  • extension/: Chrome extension integration
  • sme_playground/: SME prompt testing playground

DAG Processing Engine

Location: dag_processing/

The DAG processing engine executes multi-step workflows with: - Dependency management between steps - S3-based completion markers for idempotency - Cost tracking for LLM calls - Execution logging to database and S3 - Error handling and recovery

Key Files: - executor.py: Main workflow orchestrator - step_node.py: Step definition and registry - steps/: Individual processing steps (RSS processing, LLM extraction, etc.) - s3_utils.py: S3 integration for artifacts and completion tracking - hybrid_s3_manager.py: Hybrid S3/database storage for logs - llm_handler.py: LLM call management with cost tracking

Key Concepts: - Each step is a Python class inheriting from StepNode - Steps declare inputs/outputs and dependencies - Executor validates DAG and runs steps in topological order - Context is passed between steps via shared dictionary - Completion markers in S3 enable resumption after failures

Execution Flow: 

executor.execute() → validate DAG → build dependency graph →
execute steps in order → track completion in S3 →
log execution to database → return results with costs

Services Layer

Location: services/

Shared business logic services: - crawl_service.py: Web content crawling - crawl_engines/: Pluggable crawl engines (crawl4ai, newspaper4k) - rss_service.py: RSS feed processing - serp_service.py: Search engine integration - s3_service.py: AWS S3 operations

Background Tasks

Location: tasks.py

Celery task definitions for background processing

Celery Queue Architecture

The system uses multiple specialized queues for optimal concurrency:

  • crawl_queue: High concurrency (8 workers) for parallel web crawling
  • summary_queue: Low concurrency (1 worker) for sequential LLM summarization
  • rss_queue + horizon_queue: Workflow orchestration (1 worker)
  • celery: General background tasks (1 worker)
  • scheduler: Celery Beat for periodic task scheduling

Rationale: - Crawling can be highly parallel (network I/O bound) - LLM summarization requires sequential processing (cost control, rate limits) - Workflow orchestration coordinates other queues

API Structure

APIs are defined using Django Ninja: - apps/*/api.py: App-specific API endpoints - apps/*/v2_api.py: V2 API endpoints for some apps - apps/config/cost_analytics_v2_api.py: Cost tracking APIs - URL routing in regulatory_monitor/urls.py

Service Infrastructure

Docker Services (defined in docker-compose.yml): - postgres: PostgreSQL database - redis: Redis for Celery broker/backend - backend: Django application server - admin_dashboard: Next.js frontend - worker_crawl: High-concurrency crawling worker - worker_summary: Sequential LLM summarization worker - worker_workflows: Workflow orchestration worker - worker_general: General background tasks worker - scheduler: Celery Beat scheduler - nginx: Reverse proxy with SSL termination

Important Design Notes

  • The system uses "strategy" to identify different workflow types (e.g., "rss_monitoring")
  • Feed hashes are MD5 hashes of feed URLs for consistent S3 paths
  • Celery workers are separated by workload type for optimal concurrency
  • DAG execution is idempotent via S3 completion markers
  • All LLM calls go through the cost tracking system
  • Logs are written to both Django database and S3 for durability