README
ΒΆ
Golwarc - Go Web Crawler Master
A comprehensive Go-based crawler master framework with multiple database clients, caching systems, message queues, and various crawler implementations.
Architecture

The architecture follows a clean dependency injection pattern:
- main.go - Entry point that initializes the DI container
- inject/inject.go - Dependency injection container that conditionally initializes all packages
- services/ - Business logic layer that receives injected dependencies
- Packages - Independent modules for cache, database, crawlers, message queues, etc.
For detailed architecture documentation, see ARCHITECTURE.md.
Documentation
Comprehensive documentation is available to help you understand and use Golwarc effectively:
- Architecture - Detailed architecture and dependency injection pattern explanation
- Performance - Performance tips, optimization strategies, and benchmarks
- Security - Security best practices and guidelines
- Contributing - Guidelines for contributing to the project
- Changelog - Version history and release notes
- Troubleshooting - Common issues and solutions
Features
ποΈ Database Support
- MySQL - Full GORM integration with connection pooling
- PostgreSQL - GORM-based with transaction support
- ClickHouse - Optimized for analytical workloads
- BigTable - Google Cloud BigTable client
πΎ Caching
- LRU Cache - In-memory LRU cache with configurable size
- Redis - Full-featured Redis client with TTL, JSON support, and atomic operations
βοΈ Configuration
- Viper - YAML/JSON configuration with environment variable overrides
- Structured configuration for all components
- Hot-reload support
π¨ Message Queues
- Kafka - Producer and consumer with batch operations
- RabbitMQ - Full-featured client with exchange/queue management
π Workflow Orchestration
- Temporal - Workflow and activity management
- Signal and query support
- Worker creation and management
π·οΈ Crawler Implementations
Static Content Crawlers
- Colly - Fast and elegant scraper framework
- Spider - Custom crawler using goquery/cascadia
- Soup - Simple HTML parser
Dynamic Content Crawlers (JavaScript Support)
- Selenium - WebDriver-based browser automation
- Playwright - Multi-browser support (Chromium, Firefox, WebKit)
- Puppeteer - Chrome DevTools Protocol via chromedp
- Ferret - Declarative web scraping with FQL
π Models
Pre-built models for common scraping scenarios:
- Page - Web page data
- Product - E-commerce products
- Article - News articles and blog posts
Installation
git clone https://github.com/alonecandies/golwarc
cd golwarc
go mod download
Configuration
Copy the example configuration file and customize it:
cp configs/config.example.yaml configs/config.yaml
Edit configs/config.yaml with your database credentials, cache settings, and other configurations.
Quick Start
1. Initialize Logger
package main
import "github.com/alonecandies/golwarc/logger"
func main() {
// Initialize logger
logger.InitDefaultLogger()
defer logger.Sync()
log := logger.GetLogger()
log.Info("Hello from Golwarc!")
}
2. Load Configuration
import "github.com/alonecandies/golwarc/configs"
config, err := configs.LoadConfig("configs/config.yaml")
if err != nil {
panic(err)
}
3. Use Cache
import "github.com/alonecandies/golwarc/cache"
// LRU Cache
lruCache, err := cache.NewLRUCache(1000)
lruCache.Set("key", "value")
value, exists := lruCache.Get("key")
// Redis Cache
redisClient, err := cache.NewRedisClient(cache.RedisConfig{
Addr: "localhost:6379",
Password: "",
DB: 0,
})
redisClient.Set("key", "value", 10*time.Minute)
4. Database Operations
import "github.com/alonecandies/golwarc/database"
// MySQL
mysqlClient, err := database.NewMySQLClient(database.MySQLConfig{
Host: "localhost",
Port: 3306,
User: "root",
Password: "password",
Database: "golwarc",
})
defer mysqlClient.Close()
// Auto-migrate models
mysqlClient.Migrate(&models.Page{}, &models.Product{}, &models.Article{})
5. Web Crawling Examples
Using Colly (Static Content)
import "github.com/alonecandies/golwarc/crawlers"
client := crawlers.NewDefaultCollyClient()
client.OnHTML("h1", func(e *colly.HTMLElement) {
fmt.Println("Title:", e.Text)
})
client.OnHTML("a[href]", func(e *colly.HTMLElement) {
link := e.Attr("href")
fmt.Println("Link:", link)
e.Request.Visit(link)
})
client.Visit("https://example.com")
client.Wait()
Using Playwright (Dynamic Content)
import "github.com/alonecandies/golwarc/crawlers"
client, err := crawlers.NewPlaywrightClient(crawlers.PlaywrightConfig{
BrowserType: "chromium",
Headless: true,
})
defer client.Close()
client.Navigate("https://example.com")
client.Click("button#load-more")
client.WaitForSelector(".dynamic-content")
content, err := client.GetContent()
fmt.Println(content)
Using Ferret FQL (Declarative)
import "github.com/alonecandies/golwarc/crawlers"
client, err := crawlers.NewDefaultFerretClient()
// Extract all links
links, err := client.ExtractLinks("https://example.com")
// Extract products
products, err := client.ExampleExtractProducts("https://shop.example.com")
6. Message Queue Operations
Kafka
import "github.com/alonecandies/golwarc/message-queue"
producer := messagequeue.NewKafkaProducer(messagequeue.KafkaProducerConfig{
Brokers: []string{"localhost:9092"},
Topic: "crawl-results",
})
defer producer.Close()
producer.Produce(ctx, []byte("key"), []byte("message"))
RabbitMQ
import "github.com/alonecandies/golwarc/message-queue"
client, err := messagequeue.NewRabbitMQClient(messagequeue.RabbitMQConfig{
URL: "amqp://guest:guest@localhost:5672/",
})
defer client.Close()
client.DeclareQueue("tasks", true)
client.Publish(ctx, "tasks", []byte("task data"))
7. Temporal Workflow
import "github.com/alonecandies/golwarc/libs"
temporalClient, err := libs.NewTemporalClient(libs.TemporalConfig{
HostPort: "localhost:7233",
Namespace: "default",
})
defer temporalClient.Close()
// Start a workflow
run, err := temporalClient.StartWorkflow(ctx, "workflow-id", "task-queue", "workflowType", args)
Usage Examples
Complete Crawling Pipeline
package main
import (
"context"
"fmt"
"time"
"github.com/alonecandies/golwarc"
"github.com/alonecandies/golwarc/cache"
"github.com/alonecandies/golwarc/configs"
"github.com/alonecandies/golwarc/crawlers"
"github.com/alonecandies/golwarc/database"
"github.com/alonecandies/golwarc/models"
)
func main() {
// Initialize logger
golwarc.InitDefaultLogger()
defer golwarc.Sync()
// Load config
config, _ := configs.LoadConfig("configs/config.yaml")
// Setup cache
redisClient, _ := cache.NewRedisClient(cache.RedisConfig{
Addr: config.Cache.Redis.Addr,
Password: config.Cache.Redis.Password,
DB: config.Cache.Redis.DB,
})
defer redisClient.Close()
// Setup database
mysqlClient, _ := database.NewMySQLClient(database.MySQLConfig{
Host: config.Database.MySQL.Host,
Port: config.Database.MySQL.Port,
User: config.Database.MySQL.User,
Password: config.Database.MySQL.Password,
Database: config.Database.MySQL.Database,
})
defer mysqlClient.Close()
// Migrate models
mysqlClient.Migrate(&models.Page{})
// Create crawler
crawler := crawlers.NewDefaultCollyClient()
crawler.OnHTML("title", func(e *colly.HTMLElement) {
// Check cache
cached, _ := redisClient.Exists(e.Request.URL.String())
if cached {
return
}
// Save to database
page := &models.Page{
URL: e.Request.URL.String(),
Title: e.Text,
Status: 200,
}
mysqlClient.Create(page)
// Cache the result
redisClient.Set(e.Request.URL.String(), "processed", 24*time.Hour)
fmt.Printf("Saved: %s\n", e.Text)
})
crawler.Visit("https://example.com")
crawler.Wait()
}
Testing
# Run all tests
go test ./tests/...
# Run with verbose output
go test -v ./tests/...
# Run with coverage
go test -cover ./tests/...
# Run specific test package
go test ./tests/cache/
go test ./tests/configs/
go test ./tests/models/
Project Structure
golwarc/
βββ cache/ # Cache implementations
β βββ lru.go
β βββ redis.go
βββ configs/ # Configuration management
β βββ config.go
β βββ config.example.yaml
βββ crawlers/ # Crawler implementations
β βββ colly.go
β βββ spider.go
β βββ soup.go
β βββ selenium.go
β βββ playwright.go
β βββ puppeteer.go
β βββ ferret.go
βββ database/ # Database clients
β βββ mysql.go
β βββ postgresql.go
β βββ clickhouse.go
β βββ bigtable.go
βββ docker/ # Docker configuration
β βββ Dockerfile
β βββ docker-compose.yaml
βββ libs/ # Third-party integrations
β βββ temporal.go
βββ logger/ # Logging configuration
β βββ logger.go
βββ message-queue/ # Message queue clients
β βββ kafka.go
β βββ rabbitmq.go
βββ models/ # Data models
β βββ page.go
β βββ product.go
β βββ article.go
βββ scripts/ # Development scripts
β βββ dev.sh
βββ tests/ # Test suite
β βββ cache/
β β βββ cache_test.go
β βββ configs/
β β βββ config_test.go
β βββ models/
β βββ models_test.go
βββ .env.example # Environment variables template
βββ config.example.yaml # Configuration template
βββ go.mod
βββ go.sum
βββ main.go # Demo application
βββ README.md
External Dependencies
Browser Automation Tools
For Selenium, Playwright, and Puppeteer clients to work, you need:
Selenium
- Download ChromeDriver or GeckoDriver
- Or use a remote Selenium server (e.g., Selenium Grid)
Playwright
go run github.com/playwright-community/playwright-go/cmd/playwright install
Puppeteer (chromedp)
- Requires Chrome/Chromium installed on the system
Database Services
Ensure you have the following services running:
- MySQL (port 3306)
- PostgreSQL (port 5432)
- ClickHouse (port 9000)
- Redis (port 6379)
- Kafka (port 9092)
- RabbitMQ (port 5672)
- Temporal (port 7233)
Use Docker Compose for easy setup:
version: "3.8"
services:
mysql:
image: mysql:8.0
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: golwarc
redis:
image: redis:7-alpine
ports:
- "6379:6379"
kafka:
image: confluentinc/cp-kafka:latest
ports:
- "9092:9092"
# Add other services as needed
Testing
# Run all tests
go test ./...
# Run with coverage
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out
Performance Tips
- Use connection pooling - All database clients have built-in connection pooling
- Enable caching - Use Redis to cache frequently accessed data
- Batch operations - Use batch inserts for better throughput
- Rate limiting - Configure delays in crawlers to avoid overwhelming targets
- Concurrent crawling - Use Colly's async mode or Spider's concurrency settings
FAQ
General Questions
Q: What Go version is required?
A: Go 1.25 or later is required.
Q: Can I use this in production?
A: Yes! Golwarc is production-ready with comprehensive testing and error handling.
Q: Which crawler should I use?
A:
- Static sites: Use Colly (fastest, most efficient)
- JavaScript-heavy sites: Use Playwright or Puppeteer
- Complex scraping logic: Use Ferret with FQL
- Browser compatibility testing: Use Selenium
Q: Do I need all the external services (Redis, MySQL, etc.)?
A: No! Golwarc uses conditional initialization. Only configure the services you need.
Configuration
Q: How do I override config with environment variables?
A: Use the GOLWARC_ prefix:
export GOLWARC_DATABASE_MYSQL_HOST=production.db
export GOLWARC_CACHE_REDIS_ADDR=redis.example.com:6379
Q: Can I use environment variables exclusively?
A: Yes, either via .env file or export commands. The DI container will use defaults if config file is missing.
Database
Q: Can I use multiple databases simultaneously?
A: Yes! Configure all databases you need in config.yaml and they'll all be initialized.
Q: How do I migrate to a new database schema?
A: Use GORM's AutoMigrate:
container.MySQLClient.Migrate(&models.NewModel{})
Q: What about database connection pooling?
A: All database clients have connection pooling pre-configured with sensible defaults (10 idle, 100 max open connections).
Crawling
Q: How do I avoid getting blocked while crawling?
A:
- Add delays between requests
- Rotate user agents
- Use proxies (configure in crawler settings)
- Respect robots.txt
- Implement rate limiting
Q: Can I crawl sites that require login?
A: Yes, use Playwright or Selenium to handle authentication flows.
Q: How do I handle AJAX/dynamic content?
A: Use Playwright, Puppeteer, or Ferret instead of Colly/Soup.
Caching
Q: When should I use LRU vs Redis cache?
A:
- LRU: Single instance, development, simple caching needs
- Redis: Distributed systems, production, shared cache across services
Q: Can I use both caches together?
A: Yes! Use LRU as L1 cache and Redis as L2 for multi-tier caching.
Testing
Q: Why do tests skip?
A: Tests skip when external services aren't available. Use make docker-up to start all services.
Q: How do I run only unit tests?
A: Unit tests that don't require external services will run:
go test ./tests/configs/
go test ./tests/models/
Performance
Q: How many concurrent crawlers can I run?
A: Depends on your resources. Start with 5-10 and monitor CPU/memory. Use message queues (Kafka/RabbitMQ) for distributed crawling.
Q: How do I improve crawling speed?
A:
- Use Colly's async mode
- Increase concurrency settings
- Enable caching to avoid re-crawls
- Use connection pooling
- Batch database inserts
Troubleshooting
For common issues and solutions, see TROUBLESHOOTING.md.
Quick links:
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
License
MIT License - see LICENSE file for details
Acknowledgments
- Colly - Fast web scraping framework
- goquery - jQuery-like HTML parsing
- Playwright - Browser automation
- chromedp - Chrome DevTools Protocol
- Ferret - Declarative web scraping
- GORM - ORM library for Go
- Viper - Configuration management
- Zap - Fast structured logging
Support
For issues, questions, or contributions, please visit the GitHub repository.
Documentation
ΒΆ
There is no documentation for this package.