fluentd-protobuf-proxy-example

Fluentd Protobuf Proxy Example

An example of using Fluentd as an HTTP-to-Kafka proxy for Protobuf messages. A client POSTs a Protobuf-encoded message over HTTP, Fluentd decodes it and forwards it to a Kafka topic, and a consumer reads it from Kafka.

producer.go ──HTTP POST (protobuf)──▶ Fluentd :8080 ──▶ Kafka topic hello_world ──▶ consumer.go

Pieces

• fluentd.conf: a protobuf_http source on port 8080 that decodes messages using the schemas in fluentd_protos/, and a kafka2 output that writes to the hello_world topic.
• proto/hello.proto: the hello_world.SayHello message (name string). The same file is mounted into Fluentd from fluentd_protos/.
• producer.go: sends one SayHello to http://127.0.0.1:8080/hello_world?msgtype=hello_world.SayHello&batch=false.
• consumer.go: a Sarama consumer group on 127.0.0.1:9093 that reads hello_world and decodes each record as SayHello. Run go run consumer.go -h to see its flags.
• Dockerfile.fluentd: Fluentd 1.16 with protoc and the two plugins. It stays on 1.16 because the protobuf plugin needs the WEBrick-based HTTP server that Fluentd 1.19 removed.
• docker-compose.yml: Fluentd and a single-node Kafka 3.9 in KRaft mode.

Run

Requires Go and Docker.

make up              # build Fluentd and start it with Kafka
make create-topic    # create the hello_world topic
make log             # optional: follow Fluentd's log

make run-consumer    # in one terminal
make run-producer    # in another; prints OK! on HTTP 200, and the consumer prints the name
make kafka-consume   # or inspect the raw topic with the Kafka console consumer

make down

To regenerate the Go code after editing the proto, run make proto-code-gen, and copy the .proto into fluentd_protos/ as well.

License

MIT