Choria Blog

Choria supports a distributed authentication model as well as a centralised model using our Choria AAA Service. A Puppet user uses the distribution method by default.

In distributed mode every client has a certificate, signs his request with it and the certificate becomes the identity. The servers will verify using their RPC Authorization system if that certificate (id) can perform an action.

In the centralised setup each client do not have a certificate but it has a JWT token obtained from a sign-in service often using choria login. The JWT holds the identity, policies, permissions and more. The AAA Service signs requests using its certificate allowing clients to publish signed requests. Effectively the signing step gets outsourced to a trusted 3rd party. Before signing a request a policy is evaluated on the AAA Service to determine if the request should be allowed.

The AAA Service was introduced in 2019 and we’ve improved on it in 2020 by allowing a client certificate free operation.

The Certificate Free operation was a big win, however it came at a considerable cost of requiring additional Choria Brokers to take client connections.

We made a number of improvements in Release 0.6.0, read the full entry for details.

Today we’re releasing the next Choria Server and a few Puppet modules. Primarily this is a bug fix and general improvement release with few real big ticket user facing items.

We have a major breaking change relating to our Package Repositories. For most people who use our public repositories nothing will change, but those using internal mirrors should probably read the full post for details. In short, we are moving from Package Cloud to our own infrastructure hosted in EU, UK and US. Our packages and repositories are now signed using our own keys.

We’ve had some great feedback on Choria Governors and we’ve improved the CLI tooling a bit, we’ve also added a new Puppet Type and Provider to manage these. Thanks to users who have been testing these new features.

We have an opt-in new feature that should significantly improve the default broadcast based discovery system. Usually we wait for 2 seconds for discovery results, but in most cases most discovery results came in within the first few 100ms. By setting plugin.choria.discovery.broadcast.windowed_timeout=1 in your client configuration file we now do a windowed discovery that will terminate if after the last received result no more results were received in 300ms. In most cases this will be a massive improvement in UX. Please test it, we aim to flip this to default on in near future.

We’ve had a big set of refactors on the Debian packaging and should have functioning Debian Bullseye packages for this release. There’s also been a few improvements to the Debian packages in general.

We have started the process of supporting a new style of agent called a Choria Service. These services will be used to perform AAA signing over the NATS protocol, to facilitate DDL free clients thanks to central Schema Registries and more. Today this is mainly under the cover improvements but expect big changes coming soon in areas of client deployment simplification.

Thanks to Romain Tartière, Romuald Conty and Tim Meusel for their contributions to this release

This is the first release since April, and it’s a massive release bringing many enhancements and new features.

We are introducing Choria Streams - a Stream Processing framework built into the Choria Broker powered by NATS JetStream. I wrote a blog post about this Introducing Choria Streams that’s worth a read.

Additionally, we added Choria Key-Value Store, Choria Governor and Choria Message Submit all powered by Choria Streams and each in their own right a big feature.

Other major enhancements are that we now support Websockets for the network connections between Servers, Broker and Go clients.

Autonomous Agents now have a data layer meaning within an Autonomous Agent data can be fetched from stores like other Key-Value stores and this data can be accessed by Watchers at run time. We expose node facts to Autonomous Agents in the data layer. Additionally, we support watching Choria Key-Value Store for changes which updates the data layer and trigger transitions. Exec Watchers also support Governors to create orchestration-free rolling upgrades etc.

We made huge improvements to Provisioning, we blogged about this in Provisioning HA and Security. There you can also see we support Leader Election against Choria Streams as a library feature.

On the documentation front we added a big section about Choria Streams but also received permission to Open Source some documentation that shows how a very large - millions of nodes - Choria deployment might look. This is a proven design in active use in production for a few years already. We are busy building another such network at the moment, and a lot of the enhancements in Provisioning is as a result of this work. Find the document at Large Scale Design.

Thanks to Chris Boulton, Romain Tartière, Tim Meusel, Dominic Vallejo, Vincent Janelle and Franciszek Klajn for their contributions to this release

The Choria Provisioner is a niche component that can onboard Choria Servers into a Choria environment without needing Puppet or other CM. I often refer to this as light-bulb mode, ie. a IoT device style on-boarding rather than traditional CM.

I’ve written in the past about this in Mass Provisioning Choria Servers for background.

Today I want to talk about upcoming changes to significantly improve this process from a security and reliability perspective and talk a bit about what is next.

Read on for more details.

Choria Broker is based on the excellent NATS Server technology, this technology has been instrumental to moving Choria from its MCollective roots where 1 000 managed nodes required a big hardware investment to where we are today with a $40 Linode being enough to manage 50 000 nodes in an easy to manage and run single binary package.

NATS Server recently introduced a new capability called NATS JetStream and today I want to show a bit where we are with making that available to Choria users as Choria Streams.

JetStream is a Streaming Server that uses a WAL to create an append-only log of messages. Messages get stored to disk or memory, can be replicated within a cluster and can later be consumed by different consumers using any of the 40+ programming languages supported by NATS.

By embedding this technology in the Choria Broker we enable a number of use cases around our Metadata processing features, Autonomous Agents, CloudEvents as produced by Choria Scout, and we also introduce 2 major new features: Choria Key-Value Store and Choria Concurrency Governor.

This will all be available in our upcoming 0.23.0 release.

Read the full entry for an overview of where we are.

Till recently our documentation had a mix of visual styles for diagrams - mixing icons from Cisco, AWS etc - I recently wanted to document the supported network topologies and realised I need a more unified visual style for the documentation.

For some time now I am using diagrams.net to generate diagrams for blog posts and such, this tool is ok for diagrams but what really sets it apart for me is that even when exporting a PNG file it can embed the diagram vector source in the resulting PNG image.

This means any image on the website can simply be loaded and edited as a vector in the diagram editor, this is huge for ease of maintenance of the website, docs etc.

After some googling I found the Affinity symbol set - a public domain icon set in SVG format. Using these I came up with set of on-brand colored icons for our various components you can see below.

See the full post for links to assets and libraries for diagrams.net.

Because Choria allows you to manage nodes spread all around the world, and because you might be working from your laptop, far away from the (bad) Wi-Fi access points that connects you through (bad) PLC to the (bad) internet connection from the (not bad) island you are on, you may experience inconvenient latency and unreliabilities.

The reason is quite simple: while the Choria servers maintain a permanent connection with the message broker, the Choria client has to establish a new connection with the middleware for each request. Latency and packet loss do not help with establishing TLS encrypted connections in a timely fashion.

But good news everyone! NATS — the messaging system Choria is built on — has built-in support for so-called leaf nodes which offer a solution to this problem.

We’re pleased to announce the next set of Choria releases, these are bug and feature releases.

We’re starting to add the concept of a Service to Choria, a Service is a special kind of Agent that rather than requiring discovery and handling multiple results will only ever have 1 response. The Agents hosted as Services will form a load balanced group with High Availability and Reliability being the focus.

We will use these to create node inventory services, configuration services for Scout and eventually also move our AAA signing over to this format so that no TCP ports are needed other than the brokers. Foundational level features are being released today, but we are still working on the big picture here.

We have recreated the long broken choria plugin doc command and move the choria tool generate commands also into choria plugin, for plugin doc invoking the mco equivalent will call into Choria, but the old generate commands in mco was too different so invoking those will now fail inviting you to invoke choria plugin generate instead. Plugin documentation has been reformatted to look a bit nicer and now also support generating Markdown format output.

We updated our underlying NATS Server to version 2.2.2 which brings many stability and feature improvements. The main feature is a system called JetStream that is already enabled within Choria - though more on that at a later stage as we refine our particular use cases. If you wish to explore JetStream within Choria please reach out to us on the usual community channels.

A huge feature for us is that Websocket support has landed in the NATS. Today we do not yet expose these ports in Choria but I’d love to hear from the community who would prefer this rather than our traditional TCP ports.

Read all about NATS 2.2.2 on its announcement blog post.

Special thanks to Romain Tartière for his contributions in this release.

We’re pleased to announce the next set of Choria releases, these are mainly bug fixes, but we have a few important changes to the Choria Server and Broker.

We have a new Registration plugin that will send all the data needed for discovery, previous supported plugin only read a specific file regularly, the new plugin will send all the active state - facts, classes, collectives and more. This is a first step towards building our own discovery database to replace our use of PuppetDB in the long run.

To configure the inventory_content registration plugin you can set:

choria::server_config:
  registration: inventory_content
  plugin.choria.registration.inventory_content.target: mcollective.ingest.discovery.%{facts.fqdn}
  plugin.choria.registration.inventory_content.compression: true

Replacing mcollective.ingest.discovery.%{facts.fqdn} with your subject of choice. The intention is to ingest this into our Streaming server - more detail below.

The Choria Broker is starting to use the NATS Account system to create isolation between different organisational units, today we move all clients and nodes into a choria account as a first step. If you are upgrading a cluster of Choria Brokers expect to see some errors related to this account being unknown. Once your entire cluster is upgraded it will resolve. There might be some short network splits during this time.

Additionally, we now enable a new system account that will have events published in it for:

• connects and disconnects
• authentication errors
• server shutdowns
• regular server states

There are also a number of broker system level APIs for building reports and more. See the full post for details.

We’re starting to expose a NATS JetStream based Streaming system, which we’ll call Choria Streaming, to ingest registration, scout status, system events and more for downstream processing and analysis.

This is a huge topic, one that we’re still working on for Choria framing so more details on that later, this release adds a number of configuration items related to that already.

The Puppet modules are now able to configure something called Leaf Nodes to facilitate access to Choria from remote offices and, especially, high latency destinations. A blog post will be published this week covering that.

Special thanks to Romain Tartière, Trey Dockendorf, Tim Meusel and Mark Frost for their contributions in this release.

Hot on the heels of our January release we have a few small bug fixes to the previous release, and a number of very significant improvements to the discovery and configuration subsystems.

This is again a big release, and we suggest you do careful testing of your client applications in your testing environments after reading the Upgrade Notes in this post.

The focus of this release has been around Discovery and Configuration, we’ll let the planned module changes bake a bit longer to ensure we’re 100% stable where we are now before we undertake the next big change. Discovery features no fewer than 3 new discovery methods, we have the start of Data Providers in Compound Filters and exciting new project based configuration, read the full post for details.

Special thanks to Vincent Janelle, Romain Tartière and Ben Roberts for their contributions in this release.