Previously we used the
nats utility to explore the various patterns of messaging in NATS, today we’ll write a bit of code, and in the following few posts, we’ll expand this code to show how to scale it up and make it resilient.
We’ll write a tool that tails a log file and publishes it over NATS to a receiver that writes it to a file. The intent is that several nodes use this log file Publisher and a central node consume and saves the data to a file per Publisher with log rotation in the central node.
I should be clear that there are already solutions to this problem, I am not saying you should solve this problem by writing your own. It’s a good learning experience, though because it’s quite challenging to do right in a reliable and scalable manner.
- Producers can be all over the world in many locations
- It’s challenging to scale problem as you do not always control the rate of production and have an inherent choke point on the central receiver
- You do not want to lose any logs, so we probably need persistence
- Ordering matters, there’s no point in getting your logs in random order
- Scaling consumers horizontally while having order guarantees is difficult, additionally you need to control who writes to log files
- Running a distributed network with all the firewall implications in enterprises is very hard
So we’ll look if we can build a log forwarder and receiver that meet these criteria, in the process we’ll explore the previous sections in depth.
We’ll use Go for this but the NATS ecosystem supports almost 40 languages today, you’re spoiled for choice.[Read More]