Request a URL processes the results by a pipeline:
- Produce a scheduled http request to a configured URL
- Process the response by calculating the response time
- Push the processed data to a provided consumer ( in memeory queue , Apache Kafka etc.)
- Consumes pushed messages from the Producer pipeline
- Persists them into any configured store ( in memory store, Prostgres, etc.)
- data: describes the data model that is being produced and consumed
- flyway: coniguration and db migration scripts. flyway is used to do the migrations
- monitor: the producer(heartbeat) & the consumer(collector) pipelines are implemented together with their recpective consumer and producer implemenetations:
- pipeline: abstractions for creating pipelines, stages, producers and consumers (based on : https://github.com/PacktPublishing/Hands-On-Software-Engineering-with-Golang/tree/master/Chapter07/pipeline)
- queue: a queue interface and the Kafka and In Memory Queue implementations. Kafka Produce and Consumer are mainly based on the examples from (https://pkg.go.dev/github.com/Shopify/sarama?tab=doc)
- store: a persistent store interface and the Postgres and In Memory Store implementations
- docker-compose.yml: Runs zoekeeper, kafka and postgres locally.
- env.sh: environment variables needed to run the system.
- Makefile: some targets to setup and run the application
$ make test
- Requirements: docker, docker-compose, >=go1.14, golangci-lint
- Update env.sh contents with the appropriate settings
$ make run-infra$ make migrate$ make run-producer$ make run-consumer
- Extend tests
- Add regex matcher stage to pipeline
- Improve offset handling in consumer pipelin
- Improve error handling
- Containerize produce and consumer
- Add cloud provider support (Aiven, Confluent etc.) for queue and store
- Support multiple url registrations
- Create kubernetes deployment manifests