Skip to content

About

Large-scale video meeting platform with LiveKit + mediasoup

Resources

Stars

6 stars

Watchers

0 watching

Forks

Repository files navigation

Large-Scale Video Meeting Platform

LiveKit + mediasoup Hybrid Architecture

Target: 100–500 interactive participants, 10k+ HLS viewers, <1s latency


Architecture Overview

This platform uses a hybrid SFU approach:

  • LiveKit handles primary WebRTC signaling and client connections
  • mediasoup provides backend processing pipeline (AI, recording, broadcasting)
  • Bridge relays selective streams from LiveKit → mediasoup for server-side operations

Key Features

  • Stage/viewer model (selective subscription)
  • Simulcast/SVC for adaptive quality
  • Real-time transcription (Whisper)
  • Custom recording (MP4, HLS)
  • RTMP/HLS broadcast to CDN
  • Comprehensive monitoring (Prometheus/Grafana)

Quick Start (Development)

Prerequisites

  • Docker & Docker Compose
  • Node.js 20+ (for local development)
  • 4+ CPU cores, 8GB+ RAM
  • Public IP or ngrok for TURN/ICE

1. Clone and Setup

# Clone repository
git clone <your-repo>
cd Meeting

# Copy environment file
cp .env.example .env

# Generate LiveKit keys
openssl rand -base64 32  # Use for LIVEKIT_API_KEY
openssl rand -base64 32  # Use for LIVEKIT_API_SECRET
openssl rand -base64 32  # Use for JWT_SECRET

# Edit .env with your values
nano .env

2. Configure Coturn

# Edit coturn config with your public IP
nano coturn/turnserver.conf
# Replace REPLACE_WITH_YOUR_PUBLIC_IP with your actual IP

3. Start Services

# Start all services
docker-compose up -d

# Check logs
docker-compose logs -f

# Verify services are healthy
docker-compose ps

4. Access Services

5. Create First Room

# Using curl
curl -X POST http://localhost:8080/api/rooms \
  -H "Content-Type: application/json" \
  -d '{"name": "test-room", "maxParticipants": 100}'

# Join room (frontend will generate join token)

Development Workflow

Running Bridge Locally

cd bridge
npm install
cp .env.example .env
# Configure .env
npm run dev

Running Frontend Locally

cd frontend
npm install
npm run dev
# Access at http://localhost:5173

Running Backend Locally

cd backend
npm install
npm run dev
# Access at http://localhost:8080

Production Deployment (Kubernetes)

Prerequisites

  • Kubernetes cluster (1.24+)
  • kubectl configured
  • Helm 3+
  • Persistent storage (for recordings)
  • Load balancer with UDP support
  • Domain with SSL certificates

1. Create Namespace

kubectl create namespace meeting
kubectl config set-context --current --namespace=meeting

2. Create Secrets

# LiveKit credentials
kubectl create secret generic livekit-secrets \
  --from-literal=api-key=$(openssl rand -base64 32) \
  --from-literal=api-secret=$(openssl rand -base64 32)

# Database credentials
kubectl create secret generic postgres-secrets \
  --from-literal=password=$(openssl rand -base64 32)

# S3 credentials
kubectl create secret generic s3-secrets \
  --from-literal=access-key=YOUR_ACCESS_KEY \
  --from-literal=secret-key=YOUR_SECRET_KEY

# JWT secret
kubectl create secret generic jwt-secret \
  --from-literal=secret=$(openssl rand -base64 32)

3. Deploy Infrastructure

# Apply Kubernetes manifests
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/configmap.yaml
kubectl apply -f k8s/redis.yaml
kubectl apply -f k8s/postgres.yaml
kubectl apply -f k8s/coturn.yaml

4. Deploy Application

# Deploy LiveKit
kubectl apply -f k8s/livekit.yaml

# Deploy mediasoup workers
kubectl apply -f k8s/mediasoup.yaml

# Deploy bridge
kubectl apply -f k8s/bridge.yaml

# Deploy backend API
kubectl apply -f k8s/backend.yaml

# Deploy frontend
kubectl apply -f k8s/frontend.yaml

# Deploy recorder
kubectl apply -f k8s/recorder.yaml

# Deploy AI processor
kubectl apply -f k8s/ai-processor.yaml

5. Deploy Monitoring

kubectl apply -f k8s/monitoring/prometheus.yaml
kubectl apply -f k8s/monitoring/grafana.yaml

6. Configure Ingress

# Install cert-manager for SSL
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Apply ingress
kubectl apply -f k8s/ingress.yaml

7. Verify Deployment

# Check all pods
kubectl get pods

# Check services
kubectl get svc

# Check ingress
kubectl get ingress

# View logs
kubectl logs -f deployment/livekit

Capacity Planning

Per-Node Estimates (100 participants)

LiveKit Node:

  • CPU: 8 vCPU (1 core per ~12 active publishers)
  • RAM: 8GB (80MB per participant)
  • Network: 100 Mbps uplink, 1 Gbps downlink
  • Storage: Minimal (state only)

Calculation:

  • 100 participants @ 720p (1 Mbps outgoing each)
  • Average 20% simultaneous speakers = 20 publishers
  • Each publisher sends 1 Mbps × 3 simulcast layers = 3 Mbps
  • Total uplink: 20 × 3 = 60 Mbps
  • Each participant receives ~10 visible tiles @ 500 Kbps = 5 Mbps
  • Total downlink: 100 × 5 = 500 Mbps

mediasoup Worker Node:

  • CPU: 4 vCPU (for 10 streams)
  • RAM: 4GB
  • Network: 50 Mbps (receiving from bridge only)

Recorder Node:

  • CPU: 8 vCPU (1 core per FFmpeg instance)
  • RAM: 2GB per stream
  • Storage: 100GB/hour for 720p (depends on recording duration)

AI Processor Node:

  • CPU: 8 vCPU or 1 GPU (T4/V100)
  • RAM: 16GB (for Whisper models)
  • Network: 10 Mbps

Scaling to 500 Participants

  • LiveKit: 5 nodes (100 participants each) with load balancer
  • mediasoup: 10 workers (50 streams each)
  • Bridge: 5 instances (1 per LiveKit node)
  • Recorder: 2 nodes (redundancy)
  • AI: 2 nodes (redundancy)

Total Egress Calculation:

  • 500 participants × 5 Mbps = 2.5 Gbps
  • With 10:1 viewer ratio = 5000 viewers × 2 Mbps (HLS) = 10 Gbps (offload to CDN)

Scaling to 1000+ Participants

  • Use LiveKit's multi-node routing
  • Implement viewer-only mode (receive-only, no publishing)
  • Reduce simultaneous tile limit (show top 12 speakers)
  • Enable VP9 SVC for better compression
  • Use CDN for HLS (reduce origin egress)

Load Testing

Install Test Tools

cd scripts
npm install puppeteer playwright

Run Load Test

# Test with 50 bots
./scripts/loadtest.sh --room test-room --bots 50 --duration 300

# Test with 200 bots (requires multiple machines)
./scripts/loadtest.sh --room test-room --bots 200 --duration 600

# Monitor during test
watch -n 1 'docker stats --no-stream'

Expected Results

50 Participants:

  • CPU: 40-50% (8 vCPU node)
  • RAM: 4-5GB
  • Network: 250 Mbps downlink
  • Packet Loss: <0.5%

100 Participants:

  • CPU: 70-80% (8 vCPU node)
  • RAM: 8-9GB
  • Network: 500 Mbps downlink
  • Packet Loss: <1%

Monitoring

Grafana Dashboards

  1. LiveKit Dashboard: Room stats, participant count, bitrate, packet loss
  2. mediasoup Dashboard: Worker CPU, transport count, RTP stats
  3. System Dashboard: Node CPU, RAM, network, disk I/O
  4. Application Dashboard: API latency, error rates, webhook events

Key Metrics to Watch

# Active participants
livekit_room_participants{room="main-room"}

# Bitrate per room (Mbps)
rate(livekit_room_bytes_out[1m]) * 8 / 1000000

# Packet loss rate
rate(livekit_packet_loss_total[1m]) / rate(livekit_packets_total[1m])

# CPU usage per mediasoup worker
process_cpu_seconds_total{job="mediasoup"}

# FFmpeg recording failures
ffmpeg_recording_errors_total

Alerts (Prometheus)

# High packet loss
- alert: HighPacketLoss
  expr: livekit_packet_loss_rate > 0.03
  for: 2m
  annotations:
    summary: "Packet loss >3% in room {{ $labels.room }}"

# High CPU
- alert: HighCPU
  expr: process_cpu_percent > 90
  for: 5m
  annotations:
    summary: "CPU usage >90% on {{ $labels.instance }}"

Security Considerations

Authentication Flow

  1. User logs in → Backend generates JWT
  2. User requests room access → Backend validates JWT
  3. Backend generates short-lived LiveKit token (5 min)
  4. Client connects to LiveKit with token
  5. LiveKit validates token signature

TURN Credentials

// Backend generates time-limited TURN credentials
const username = Math.floor(Date.now() / 1000) + 86400; // 24h expiry
const secret = 'your-coturn-secret';
const password = crypto.createHmac('sha1', secret)
  .update(username.toString())
  .digest('base64');

Rate Limiting

  • API: 100 req/min per IP
  • WebSocket: 10 connections per user
  • Room creation: 10 rooms/hour per user

CORS Configuration

// Backend
app.use(cors({
  origin: ['https://yourdomain.com'],
  credentials: true,
  maxAge: 86400
}));

Troubleshooting

Bridge Not Connecting

# Check LiveKit accessibility
curl -v ws://livekit:7880/rtc

# Check mediasoup worker logs
docker-compose logs mediasoup-worker

# Verify API credentials
echo $LIVEKIT_API_KEY | base64

No Video/Audio

  1. Check TURN server is running: telnet your-ip 3478
  2. Verify ICE candidates in browser DevTools
  3. Check firewall allows UDP ports 40000-65535
  4. Verify codec compatibility (VP8 supported on all browsers)

High Latency

  1. Check CPU usage (should be <80%)
  2. Verify network bandwidth (monitor with iftop)
  3. Reduce simulcast layers (from 3 to 2)
  4. Enable hardware acceleration (H.264 on GPU)

Recording Failures

  1. Check disk space: df -h
  2. Verify S3 credentials
  3. Check FFmpeg logs: docker-compose logs recorder
  4. Test FFmpeg manually: ffmpeg -i rtsp://... test.mp4

Implementation Checklist

Phase 1: Dev/Local Prototype (Week 1-2)

  • Set up Docker Compose environment
  • Configure LiveKit with test API keys
  • Implement basic bridge (single stream)
  • Create minimal React frontend (1-on-1 call)
  • Test end-to-end connectivity
  • Verify TURN server works behind NAT
  • Implement basic recording (local file)

Phase 2: Staging (Week 3-4)

  • Deploy to Kubernetes staging cluster
  • Configure persistent storage (PVC)
  • Implement full bridge (multiple streams, selective forwarding)
  • Add stage/viewer mode to frontend
  • Implement moderator controls
  • Add Prometheus metrics
  • Set up Grafana dashboards
  • Perform load test (50 bots)
  • Implement recording to S3
  • Add basic transcription (Whisper)

Phase 3: Production (Week 5-6)

  • Configure production Kubernetes cluster
  • Set up multi-region TURN servers
  • Implement HPA (Horizontal Pod Autoscaler)
  • Configure CDN for HLS output
  • Add authentication & authorization
  • Implement rate limiting
  • Set up SSL/TLS certificates
  • Configure backup & disaster recovery
  • Perform load test (200+ bots)
  • Set up alerting (PagerDuty/Slack)
  • Write operational runbooks
  • Conduct security audit
  • Plan capacity for launch

Phase 4: Optimization (Week 7-8)

  • Optimize simulcast layer selection
  • Implement adaptive bitrate based on CPU/bandwidth
  • Add connection quality indicators
  • Optimize FFmpeg encoding params
  • Implement smart grid layout (show active speakers)
  • Add screen share optimization (higher res/fps)
  • Optimize database queries
  • Implement Redis caching
  • Add CDN cache warming
  • Fine-tune TURN server performance

Next Steps

  1. Generate API Keys: Run openssl rand -base64 32 for all secrets
  2. Configure Network: Set ANNOUNCED_IP to your public IP
  3. Start Dev Environment: Run docker-compose up -d
  4. Test Locally: Access http://localhost:5173 and create test room
  5. Review Logs: Check docker-compose logs -f for errors
  6. Scale Testing: Follow load testing guide with 10+ bots
  7. Production Deploy: Follow Kubernetes deployment guide

Support & Resources


License

MIT (adjust as needed)

About

Large-scale video meeting platform with LiveKit + mediasoup

Resources

Stars

6 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

⌂ 换网址