Choria Blog

Yesterday we wrote our first end to end file tail tool and a consumer for it. It was fatally flawed though in that we could not scale it horizontally and had to run one consumer per log file. This design won’t work for long.

The difficulty lies in the ordering of messages. We saw that NATS supports creating consumer groups and that it load-shares message handling in a random manner between available workers. The problem is if we have more than one worker, there are no longer any ordering guarantees as many workers are concurrently handling messages each processing them at different rates. We’d end up with 1 file having many writers and soon it will be shuffled.

We’re fortunate in that the ordering requirements aren’t necessary for all of the messages, we only really need to order messages from a single host. Host #1 messages can be mixed with host #2, as long as all of the host 1 messages are in the right order. This behaviour is a big plus and lets us solve the problem. We can have 100 workers receiving logs, as long as 1 worker always handles logs from one Producer.

Read on for the solution!

Previously we used the nats utility to explore the various patterns of messaging in NATS, today we’ll write a bit of code, and in the following few posts, we’ll expand this code to show how to scale it up and make it resilient.

We’ll write a tool that tails a log file and publishes it over NATS to a receiver that writes it to a file. The intent is that several nodes use this log file Publisher and a central node consume and saves the data to a file per Publisher with log rotation in the central node.

I should be clear that there are already solutions to this problem, I am not saying you should solve this problem by writing your own. It’s a good learning experience, though because it’s quite challenging to do right in a reliable and scalable manner.

• Producers can be all over the world in many locations
• It’s challenging to scale problem as you do not always control the rate of production and have an inherent choke point on the central receiver
• You do not want to lose any logs, so we probably need persistence
• Ordering matters, there’s no point in getting your logs in random order
• Scaling consumers horizontally while having order guarantees is difficult, additionally you need to control who writes to log files
• Running a distributed network with all the firewall implications in enterprises is very hard

So we’ll look if we can build a log forwarder and receiver that meet these criteria, in the process we’ll explore the previous sections in depth.

We’ll use Go for this but the NATS ecosystem supports almost 40 languages today, you’re spoiled for choice.

In our previous posts, we did a thorough introduction to messaging patterns and why you might want to use them, today let’s get our hands dirty by setting up a NATS Server and using it to demonstrate these patterns.

Setting up

Grab one of the zip archives from the NATS Server Releases page, inside you’ll find nats-server, it’s easy to run:

$ ./nats-server -D
[26002] 2020/03/17 15:30:25.104115 [INF] Starting nats-server version 2.2.0-beta
[26002] 2020/03/17 15:30:25.104238 [DBG] Go build version go1.13.6
[26002] 2020/03/17 15:30:25.104247 [INF] Git commit [not set]
[26002] 2020/03/17 15:30:25.104631 [INF] Listening for client connections on 0.0.0.0:4222
[26002] 2020/03/17 15:30:25.104644 [INF] Server id is NA3J5WPQW4ELF6AJZW5G74KAFFUPQWRM5HMQJ5TBGRBH5RWL6ED4WAEL
[26002] 2020/03/17 15:30:25.104651 [INF] Server is ready
[26002] 2020/03/17 15:30:25.104671 [DBG] Get non local IPs for "0.0.0.0"
[26002] 2020/03/17 15:30:25.105421 [DBG]   ip=192.168.88.41

The -D tells it to log verbosely, at this point you’ll see you have port 4222 open on 0.0.0.0.

Also grab a copy of the nats CLI and place this in your path, this can be found in the JetStream Releases page, you can quickly test the connection:

$ nats rtt
nats://localhost:4222:

   nats://127.0.0.1:4222: 251.289µs
       nats://[::1]:4222: 415.944µs

Above shows all you need to have a NATS Server running in development and production use is not much more complex, to be honest, once it’s running this can happily serve over 50,000 Choria nodes depending on your hardware (a $40 Linode would do)

Yesterday, in our previous post, we did a light compare between HTTP and Middleware based architectures for Microservices. Today we’ll start focusing on Middleware based architectures and show some detail about the why’s and the patterns available to developers.

Why Middleware Based

The Middleware is a form of Transport, rather than addresses you think in named channels and the endpoints within that channel decides how they consume the messages, 1:1, 1:n or randomly-selected 1:n.

The goal is to promote an architecture that is scalable, easy to manage and easy to operate. These goals are achieved by:

• Promoting application design that breaks complex applications into simple single-function building blocks that’s easy to develop, test and scale
• Application building blocks are not tightly coupled and can scale independently of other building blocks
• The middleware layer implementation is transparent to the application – network topologies, routing, ACLs etc. can change without application code change
• The brokers provide a lot of the patterns you need for scaling – load balancing, queuing, persistence, eventual consistency, failover, etc.
• Mature brokers are designed to be scalable and highly available – very complex problems that you do not want to attempt to solve on your own
• Fewer moving parts, less coupled to infrastructure layers and scalable across multiple clusters

There are many other reasons, but for me, these are the big-ticket items – especially the 2nd one.

Back in 2011, I wrote a series of posts on Common Messaging Patterns Using Stomp; these posts were very popular, I figured it’s time for a bit of refresh focusing on NATS - the Middleware Choria uses as it’s messaging transport.

Today there are 2 prevailing architectures in Microservices based infrastructure - HTTP based and Middleware based Microservices. I’ll do a quick overview of the two patterns highlighting some of the pros and cons here; first, we look at the more familiar HTTP based and then move to Middleware based. In follow-up posts, we’ll explore the Middleware communication patterns in detail and show some code.

Note though the context here is not to say one is better than the other or to convince you to pick a particular style, I am also not exhaustively comparing the systems - that would be impossible to do well.

Today the prevailing architecture of choice is HTTP based, and it’s demonstrably a very good and very scalable choice. I want to focus on using Middleware to achieve similar outcomes and what other problems they can solve and how - the aim is to share information and not to start a product/architecture comparison debate.

Series Index

• Series introduction
• Description of the major messaging patterns supported by NATS
• Running NATS Server and exploring patterns from the CLI
• Creating a basic log distribution tool
• Adding scalability to the log distribution tool
• Introducing JetStream
• Interacting with the JetStream API
• Adjusting the log distribution tool for JetStream
• Reliable delivery and Acknowledgement Patterns

Open Policy Agent is a CNCF incubating project that allow you to define Policy as code. It’s widely used in various projects like Istio, Kubernetes and more.

It allows you to express authorization policies - like our Action Policy - in a much more flexible way.

Building on the work that was done for aaasvc, I’ve added a rego engine to the choria server, which will allow us to do most of what actionpolicy allows, as well as:

• Assertions based on the arguments sent to actions
• Assertions based on other request fields like TTL and Collective
• Assertions based on if the server is set to provisioning mode or not

Read below the fold for our initial foray into OPA policies and what might come next.

Choria configuration, despite efforts with Puppet module and so, is still very challenging. Just knowing what settings are available has been a problem.

We tried to hide much of the complexity behind Puppet models but for people who don’t conform to the norm it’s been a challenge.

I eventually want to move to a new configuration format - perhaps HCL? - but this is a massive undertaking both for me and users. For now we’ve made some effort to give insights to all the known configuration settings on the CLI and in our documentation.

First we’ll publish a generated configuration reference in CONFIGURATION.md - for now it’s in the Git repository we’ll move it to the doc site eventually.

As of the upcoming version of Choria Server you’ll be able to query the CLI for any setting using regular expressions. The list will show descriptions, data types, validation rules, default values, deprecation hints and URLs to additional information.

And get a list:

$ choria tool config puppet -l
plugin.choria.puppetca_host
plugin.choria.puppetca_port
plugin.choria.puppetdb_host
plugin.choria.puppetdb_port
plugin.choria.puppetserver_host
plugin.choria.puppetserver_port

These references are extracted from the Go code - something that I never imagine is possible - read on for details on how that is done.

I’ve previously blogged about a niche system that enables Mass Provisioning Choria Servers, it is quite scary and tend to be specific to large customers so it’s off by default and require customer specific builds to enable.

I’ve had some time with this system and it’s proven very reliable and flexible, I’ve had many 100s of thousands of nodes under management of this provisioner and can kick 10s of thousands of nodes into provisioning state and it will happily do thousands a minute.

The concept is sound and the next obvious step is to make it available to FOSS users in our FOSS builds. To get there is a long road one that I think will take us toward Kubernetes deployed Choria Broker, CA and Choria Provisioners and eventually a Puppet free deployment scenario.

It’s a long road and step one is about looking at how we can safely enable provisioning in the open source builds. Read on for the details of what was enabled in Choria Server 0.13.0 that was released on 12th of December.

Open Policy Agent is a CNCF incubating project that allow you to define Policy as code. It’s widely used in various projects like Istio, Kubernetes and more.

It allows you to express authorization policies - like our Action Policy - in a much more flexible way.

I’ve wanted to explore what is next for our general Authorization piece and I think this gets us to a very good place - and OPA have massive adoption so it’s always good to adopt widely used standards.

Using Open Policy we’ll be able to do a number of things we’ve never been able to do - but get asked about regularly:

• Make sure requests have filters associated to avoid huge blast radius
• Assertions based on the arguments sent to actions
• Assertions based on other request fields like TTL and Collective

And just generally be much more expressive about it.

Read below the fold for our initial foray into OPA policies and what might come next.

It’s been a while since we had release announcements and it’s been a surprisingly busy period.

The main focus here has been on a number of stability and bug fixes, we’ve had some users dig in really deep into various aspects of the system and a number of bugs were squashed.

Past the quality of life stuff I have started reworking Choria Server Provisioning which will set us on a path to having a good Puppet free story, I have some POCs lying around of a Kubernetes based Broker, CA, and Provisioner that will give a really smooth path forward - provisioning is now compiled in to the FOSS stack by default and can be enabled using a JWT token, more on that in a future post.

We also include a Tech Preview of NATS JetStream support and significantly moved our event formats over to Cloud Events v1.0 format.

Thanks especially go to Alexander Hermes for his deep dive into all aspects of the client side playbooks. Deep dives into a product and filing some tickets, discussing the model on slack etc it hugely time consuming and very often this kind of community contribution flies under the radar but I find it more valuable than code, huge props to Alexander.

Other shout outs to Ben Robert, Yury Bushmelev, Romain Tartière and Vincent Janelle