Task manager with parallel processing functionality based on a YAML task definition file designed for Metafog's decentralised network. See examples to learn more.
Credits: https://github.com/go-task/task
This is a trimmed and adapted version of the above library in order to support parallel task processing on iteratables like numeric range, files and records in a file.
loop command is added to support the above feature.
Mac, Windows and Linux binaries are in bin folder. They are statically built without any dependancies.
Note: Binaries are bundled with Metafog Gateway installers.
Clone the repository.
$ git clone https://github.com/metafog/metafog-compose
$ cd metafog-compose
$ go mod tidy
$ go run cmd/task/task.go
tasks:
default:
cmds:
- loop:
range: [10, 20]
run: mytask
parallel: 0
mytask:
cmds:
- echo "My Task - Index:{{.INDX}}, Value:{{.ARG}}"
INDX is the iteration number starting 1.
ARG is the number within the range for the iteration.
tasks:
default:
cmds:
- loop:
range: [1, 100]
run: mytask
parallel: 5
mytask:
cmds:
- echo "My Task - Index:{{.INDX}}, Value:{{.ARG}}"
Here, mytask will be executed in parallel with concurrency of 5. Loop command will exit only after all tasks are executed.
Iterate through numbers - and run task.
ARG will be the number in the range.
- loop:
range: [1, 100]
run: task1
parallel: 0
Parameters: range: [<start>, <end>] (Both are inclusive)
Iterate through a list of values and run task.
ARG will be the value in the list.
- loop:
list: ["Orange", "Apple"]
run: task1
parallel: 0
Parameters: list: [<value>, ...]
Iterate through all files in the folder (ignore sub folders) and run task1.
ARG will be the file name.
- loop:
folder: /tmp/
run: task1
parallel: 0
Parameters: folder: <folder-path>
Iterate through each line in the file and run task1.
ARG will be the contents of the line.
- loop:
file: /tmp/urls.txt
run: task1
parallel: 2
Parameters: file: <file-path>
Run a timer with seconds interval and run task1.
ARG will be the time of that execution.
- loop:
timer: 3
run: task1
parallel: 2
Parameters: timer: <interval-in-seconds>
This will never end. You have to exit the process by Ctrl+C.
Monitor a folder for new files and run task1.
ARG will be the new file path.
- loop:
folder_watch: /tmp/
run: task1
parallel: 2
Parameters: folder_watch: <folder-path>
This will never end. You have to exit the process by Ctrl+C.
Subscribe to ActiveMQ topic and run task1.
ARG will be the message body.
- loop:
activemq: ["localhost:61613", "metafog"]
run: task1
parallel: 2
Parameters: activemq: [<connection-url>, <topic-name>]
This will never end. You have to exit the process by Ctrl+C.
Subscribe to Kafka messages and run task1.
ARG will be the message body.
- loop:
kafka: ["localhost:9092", "metafog-topic"]
run: task1
parallel: 2
Parameters: kafka: [<connection-url>, <topic-name>]
This will never end. You have to exit the process by Ctrl+C.
There are many examples for you to get started with metafog-compose.