Architecture

System Overview

Maverick is a lead-generation and client-management platform for insurance marketing. It processes contacts through a multi-stage pipeline: scraping, filtering, verification, batching, and upload to Email Bison campaigns.

graph TB
    Internet["Internet"]-->|HTTPS :443|Nginx["Nginx<br/>SSL Termination"]
    Nginx-->|:3000|Dashboard["Dashboard<br/>Next.js 14"]
    Nginx-->|:8000|API["API<br/>FastAPI + Uvicorn"]
    Nginx-->|:5555|Flower["Flower<br/>Task Monitor"]
    Nginx-->|:3003|Docs["Docs Site<br/>Fumadocs"]
 
    API-->Redis[("Redis<br/>Message Broker")]
    API-->Supabase[("Supabase<br/>PostgreSQL + Auth")]
 
    Redis-->W1["Scraping Worker<br/>Playwright Browser"]
    Redis-->W2["CSV Processing Worker"]
    Redis-->W3["Filtering Worker"]
    Redis-->W4["Verification Worker<br/>Debounce API"]
    Redis-->W5["Batching + Bison Worker"]
 
    W1-->|CSV files|W2
    W5-->Bison["Email Bison<br/>Campaign Platform"]
 
    API-->Pushgateway["Pushgateway"]-->Prometheus["Prometheus"]-->Grafana["Grafana"]

Docker Services

Production (docker-compose.production.yml)

Worker Configuration (Production):

Local (docker-compose.local.yml)

Worker Configuration (Local):

Celery Queue Architecture

Task Routing

Autodiscovered Task Modules

• app.workers.tasks.scraping
• app.workers.tasks.processing
• app.workers.tasks.filtering
• app.workers.tasks.verification
• app.workers.tasks.batching
• app.workers.tasks.email_bison_integration
• app.workers.tasks.webhook_processing
• app.workers.tasks.campaign_sync
• app.workers.tasks.tag_burnt_accounts
• app.workers.tasks.quarantine_burnt_accounts
• app.workers.tasks.sentiment
• app.workers.tasks.auto_reply
• app.workers.tasks.daily_metrics
• app.workers.tasks.infrastructure_orders
• app.workers.tasks.hall_of_fame_backfill
• app.workers.tasks.master_db_fill
• app.workers.tasks.daily_infra_reports
• app.workers.tasks.verification_maintenance
• app.workers.tasks.task_notifications
• app.workers.tasks.task_comment_notifications
• app.workers.tasks.task_slack_cleanup
• app.workers.tasks.slack_inbound
• app.workers.tasks.slack_event_dedup
• app.workers.tasks.audit_discover
• app.workers.tasks.audit_ingest_replies
• app.workers.tasks.audit_ingest_sequence_steps
• app.workers.tasks.audit_ingest_senders
• app.workers.tasks.audit_attribute
• app.workers.tasks.audit_aggregate_rankings
• app.workers.tasks.audit_generate_recommendations
• app.workers.tasks.audit_expire_recommendations
• app.workers.tasks.audit_render_briefings
• app.workers.tasks.audit_refresh
• app.workers.tasks.volume_tracking
• app.workers.tasks.account_stats_snapshot
• app.workers.tasks.reconcile_interested_leads
• app.workers.tasks.billing_audit
• app.workers.tasks.interested_signal_learner

Beat Schedule (Periodic Tasks)

Worker Defaults

Contact Pipeline

The core business process: acquire contacts, clean them, and upload to campaigns.

graph LR
    XD["Xpressdocs<br/>X-Dates Platform"]-->|"Playwright<br/>scraping queue"|Raw[("raw_contacts")]
    Raw-->|"csv_processing queue"|Parsed["Parsed + Inserted"]
    Parsed-->|"filtering queue"|Filtered[("filtered_contacts")]
    Filtered-->|"verification queue<br/>Debounce API"|Verified[("verified_contacts")]
    Verified-->|"batching queue"|Batches[("contact_batches")]
    Batches-->|"email_bison queue"|Bison["Email Bison<br/>Campaigns"]

Each stage is a separate Celery queue with dedicated workers. Stages communicate via database tables (not direct task chaining), so any stage can be re-run independently.

Port Map