releases

It’s been almost 5 months since our last release, not because nothing has been happening but because so much has been happening, good problems to have!

So this is a bit of a massive release, however I think the bulk of the changes will not affect our typical Puppet based users.

Choria Registry

This introduces first work of a new Choria Registry. We have a long-standing pain point around managing DDL files on clients, it’s a technical requirement to describe remote services but it’s just a pain to maintain, Puppet helps but for clients in CI, desktops etc, the DDL requirement is just too much.

Choria Server now has an option to act as a Registry where it can read it’s local DDL directory and serve that up to clients on demand. When a client tries to access a new agent it has never accessed before it will ask the registry for the DDL describing that agent. It will also do so regularly to ensure the local cache is still accurate.

This means that we can now have truly single-file client deployments. With just the choria binary and a running Registry that choria client can interact with the entire fleet and do everything it wants. This is a great improvement for deployment of client machines and making Choria more generally useful without Configuration Management.

The Choria Server can be a Registry, running multiple Servers with registry enabled will create a failure tolerant HA cluster of registry servers.

This is a brand-new feature, so I am not yet documenting it publicly, but I am keen to talk to users who wish to help in validating this before we look to supporting this more widely.

Non mTLS communications

The major work here that contributed to the 20 000 line code change in Choria Server is that we now support a secure non mTLS mode of communication. This is of no consequence for Puppet users so if that’s you feel free to skip this section.

With a typical deployment we use the Puppet CA to create a fully managed and closed mTLS based network. For some enterprises replicating that with their internal PKI infrastructure is nearly impossible. So we looked to, optionally, move away from a pure mTLS mode to a mixed setup where we use ED25519 keypair and signed JWTs to provide equivalent security.

Essentially we now have formalized our use of JWT into a new tokens package where servers and clients have their own JWT. We hope to move entirely over to this model in time as we were able to create a greatly enhanced security model:

• Servers are restricted to only certain collectives, attempting to enter non defined collectives will be denied by the broker
• Servers are restricted to only server traffic flows. A server token cannot make a request to any other server, enforced by the broker
• Servers have a default deny permission set allow specific access to Streams, Governors, Hosting Services and being able to be a Submission Server
• Clients have private reply channels, clients cannot view each others replies
• In addition to Open Policy Agent a set of default deny permissions allowing access to use Streams, administer Streams, use Elections, view Events, use Governors etc

Using these settings moves us to a much more secure and private setup where even between 2 Choria Users traffic is now isolated and secure and this introduces the first of a security model around our adoption of Choria Streams. We cannot replicate these policies using just certificates. We hope to move even Puppet users to this model in future but that’s a big undertaking to get right without additional services.

To enable these features one needs to deploy AAA Service and Provisioner - and both of those had recent releases supporting this mode.

As mentioned this is not really a thing that Puppet users should worry about however those in large enterprises who deploy in non-Puppet ways should keep an eye out for incoming documentation around this feature.

Package Repository Changes

As notified back in September we are moving away from Packagecloud to our own package hosting infrastructure. I am keeping the Packagecloud infrastructure up for a while but this release and all future ones will not be uploaded there to promote users moving to the new infrastructure.

Thanks to Romain Tartière, Steffy Fort, Tim Meusel and Alexander Olofsson for their contributions to this release

Today we’re releasing the next Choria Server and a few Puppet modules. Primarily this is a bug fix and general improvement release with few real big ticket user facing items.

We have a major breaking change relating to our Package Repositories. For most people who use our public repositories nothing will change, but those using internal mirrors should probably read the full post for details. In short, we are moving from Package Cloud to our own infrastructure hosted in EU, UK and US. Our packages and repositories are now signed using our own keys.

We’ve had some great feedback on Choria Governors and we’ve improved the CLI tooling a bit, we’ve also added a new Puppet Type and Provider to manage these. Thanks to users who have been testing these new features.

We have an opt-in new feature that should significantly improve the default broadcast based discovery system. Usually we wait for 2 seconds for discovery results, but in most cases most discovery results came in within the first few 100ms. By setting plugin.choria.discovery.broadcast.windowed_timeout=1 in your client configuration file we now do a windowed discovery that will terminate if after the last received result no more results were received in 300ms. In most cases this will be a massive improvement in UX. Please test it, we aim to flip this to default on in near future.

We’ve had a big set of refactors on the Debian packaging and should have functioning Debian Bullseye packages for this release. There’s also been a few improvements to the Debian packages in general.

We have started the process of supporting a new style of agent called a Choria Service. These services will be used to perform AAA signing over the NATS protocol, to facilitate DDL free clients thanks to central Schema Registries and more. Today this is mainly under the cover improvements but expect big changes coming soon in areas of client deployment simplification.

Thanks to Romain Tartière, Romuald Conty and Tim Meusel for their contributions to this release

This is the first release since April, and it’s a massive release bringing many enhancements and new features.

We are introducing Choria Streams - a Stream Processing framework built into the Choria Broker powered by NATS JetStream. I wrote a blog post about this Introducing Choria Streams that’s worth a read.

Additionally, we added Choria Key-Value Store, Choria Governor and Choria Message Submit all powered by Choria Streams and each in their own right a big feature.

Other major enhancements are that we now support Websockets for the network connections between Servers, Broker and Go clients.

Autonomous Agents now have a data layer meaning within an Autonomous Agent data can be fetched from stores like other Key-Value stores and this data can be accessed by Watchers at run time. We expose node facts to Autonomous Agents in the data layer. Additionally, we support watching Choria Key-Value Store for changes which updates the data layer and trigger transitions. Exec Watchers also support Governors to create orchestration-free rolling upgrades etc.

We made huge improvements to Provisioning, we blogged about this in Provisioning HA and Security. There you can also see we support Leader Election against Choria Streams as a library feature.

On the documentation front we added a big section about Choria Streams but also received permission to Open Source some documentation that shows how a very large - millions of nodes - Choria deployment might look. This is a proven design in active use in production for a few years already. We are busy building another such network at the moment, and a lot of the enhancements in Provisioning is as a result of this work. Find the document at Large Scale Design.

Thanks to Chris Boulton, Romain Tartière, Tim Meusel, Dominic Vallejo, Vincent Janelle and Franciszek Klajn for their contributions to this release

We’re pleased to announce the next set of Choria releases, these are bug and feature releases.

We’re starting to add the concept of a Service to Choria, a Service is a special kind of Agent that rather than requiring discovery and handling multiple results will only ever have 1 response. The Agents hosted as Services will form a load balanced group with High Availability and Reliability being the focus.

We will use these to create node inventory services, configuration services for Scout and eventually also move our AAA signing over to this format so that no TCP ports are needed other than the brokers. Foundational level features are being released today, but we are still working on the big picture here.

We have recreated the long broken choria plugin doc command and move the choria tool generate commands also into choria plugin, for plugin doc invoking the mco equivalent will call into Choria, but the old generate commands in mco was too different so invoking those will now fail inviting you to invoke choria plugin generate instead. Plugin documentation has been reformatted to look a bit nicer and now also support generating Markdown format output.

We updated our underlying NATS Server to version 2.2.2 which brings many stability and feature improvements. The main feature is a system called JetStream that is already enabled within Choria - though more on that at a later stage as we refine our particular use cases. If you wish to explore JetStream within Choria please reach out to us on the usual community channels.

A huge feature for us is that Websocket support has landed in the NATS. Today we do not yet expose these ports in Choria but I’d love to hear from the community who would prefer this rather than our traditional TCP ports.

Read all about NATS 2.2.2 on its announcement blog post.

Special thanks to Romain Tartière for his contributions in this release.

We’re pleased to announce the next set of Choria releases, these are mainly bug fixes, but we have a few important changes to the Choria Server and Broker.

We have a new Registration plugin that will send all the data needed for discovery, previous supported plugin only read a specific file regularly, the new plugin will send all the active state - facts, classes, collectives and more. This is a first step towards building our own discovery database to replace our use of PuppetDB in the long run.

To configure the inventory_content registration plugin you can set:

choria::server_config:
  registration: inventory_content
  plugin.choria.registration.inventory_content.target: mcollective.ingest.discovery.%{facts.fqdn}
  plugin.choria.registration.inventory_content.compression: true

Replacing mcollective.ingest.discovery.%{facts.fqdn} with your subject of choice. The intention is to ingest this into our Streaming server - more detail below.

The Choria Broker is starting to use the NATS Account system to create isolation between different organisational units, today we move all clients and nodes into a choria account as a first step. If you are upgrading a cluster of Choria Brokers expect to see some errors related to this account being unknown. Once your entire cluster is upgraded it will resolve. There might be some short network splits during this time.

Additionally, we now enable a new system account that will have events published in it for:

• connects and disconnects
• authentication errors
• server shutdowns
• regular server states

There are also a number of broker system level APIs for building reports and more. See the full post for details.

We’re starting to expose a NATS JetStream based Streaming system, which we’ll call Choria Streaming, to ingest registration, scout status, system events and more for downstream processing and analysis.

This is a huge topic, one that we’re still working on for Choria framing so more details on that later, this release adds a number of configuration items related to that already.

The Puppet modules are now able to configure something called Leaf Nodes to facilitate access to Choria from remote offices and, especially, high latency destinations. A blog post will be published this week covering that.

Special thanks to Romain Tartière, Trey Dockendorf, Tim Meusel and Mark Frost for their contributions in this release.

Hot on the heels of our January release we have a few small bug fixes to the previous release, and a number of very significant improvements to the discovery and configuration subsystems.

This is again a big release, and we suggest you do careful testing of your client applications in your testing environments after reading the Upgrade Notes in this post.

The focus of this release has been around Discovery and Configuration, we’ll let the planned module changes bake a bit longer to ensure we’re 100% stable where we are now before we undertake the next big change. Discovery features no fewer than 3 new discovery methods, we have the start of Data Providers in Compound Filters and exciting new project based configuration, read the full post for details.

Special thanks to Vincent Janelle, Romain Tartière and Ben Roberts for their contributions in this release.

We have a number of releases today that will be the start of big changes in our modules. These releases will hopefully have a minor impact on users, but the next release or two will require some Hiera changes, so it’s worth keeping an eye on these. For the next while testing in your labs and dev environments is essential.

This is the beginning of a big push to once again simplify our deployment story. Choria started as a trivial way to install MCollective but things have changed quite a lot since then and unfortunately entropy has had its effect on our modules.

In addition to these changes we also have some pretty amazing additions to the Choria Servers.

Read on for the background and details of what’s to come.

On the community side we’ve set up a GitHub Discussions group for those who are not keen on signing up to Slack.

Special thanks to Tim Meusel, Vincent Janelle, Vadym Chepkov, Vladislav Kuspits and Romain Tartière for their contributions in this release.

We have a number of small releases today, mainly quality of life changes - performance improvements and such.

The only major work here is around our Autonomous Agent feature, this lets you build managed finite state machines that can manage components on your machines without RPC interaction. This underpins our Scout checks and helps in IoT scenarios etc.

Today we’re adding 2 new watchers, an Apple HomeKit Button and a Timer. The HomeKit button is interesting in home automation scenarios where a Choria Autonomous Agent can appear to your Apple devices as a button that you can toggle from your Apple Home apps. Combined with the timer it’s possible to create an override button for HVAC, Fans etc that interrupts a normal managed schedule for a while. For example when watching a movie I don’t like having my extractor fan on, using any Apple device I can now set a 2 hour override, after 2 hours normal scheduled activity resumes so I don’t need to remember to re-enable the extractor.

In future releases we’ll add a Timer based maintenance window to Scout checks using the timer watcher.

We’re starting to work on supporting Puppet 7, progress is being made (thanks Tim!) but I think we have some way to go.

Special thanks to Tim Meusel and Romuald Conty for their contributions in this release.

Today we have quite a bumper release with significant updates for Choria Scout and the first step in improvements for AAA Service managed clients.

We added numerous Choria Scout CLI tools - choria scout status, choria scout trigger, choria scout maintenance and choria scout resume. These allow you to manage a fleet of Choria nodes that are performing Scout checks.

$ choria scout status dev1.example.net
+-----------------------+-------+------------+-------------------------------+
| NAME                  | STATE | LAST CHECK | HISTORY                       |
+-----------------------+-------+------------+-------------------------------+
| mailq                 | OK    | 1m20s      | OK OK OK OK                   |
| ntp_peer              | OK    | 1m32s      | OK OK OK OK OK OK OK OK OK OK |
| pki                   | OK    | 2m28s      | OK OK OK OK OK OK OK OK OK OK |
| puppet_failures       | OK    | 2m3s       | OK OK OK OK WA WA CR CR OK OK |
| puppet_run            | OK    | 24s        | OK OK OK                      |
| swap                  | OK    | 4m23s      | OK OK OK OK OK OK OK          |
| zombieprocs           | OK    | 2m23s      | OK OK OK OK OK OK OK OK OK OK |
| goss                  | OK    | 3m12s      | OK OK OK                      |
| heartbeat             | OK    | 57s        | OK OK OK OK OK OK OK OK OK OK |
+-----------------------+-------+------------+-------------------------------+

The choria req utility got a new --table format option and all the result rendering code got extracted into a reusable package.

[rip@dev1]% choria req package status package=zsh --table
Discovering nodes .... 2

2 / 2    0s [====================================================================] 100%

+------------------+--------+------------------+-------+------+------------+---------+
| SENDER           | ARCH   | ENSURE           | EPOCH | NAME | RELEASE    | VERSION |
+------------------+--------+------------------+-------+------+------------+---------+
| dev2.example.net | x86_64 | 5.0.2-34.el7_8.2 | 0     | zsh  | 34.el7_8.2 | 5.0.2   |
| dev1.example.net | x86_64 | 5.0.2-34.el7_8.2 | 0     | zsh  | 34.el7_8.2 | 5.0.2   |
+------------------+--------+------------------+-------+------+------------+---------+

We improved generated Go clients significantly by allowing them to have typical progress bars, choria req like result formatting, result parsing helpers, improved logging and faster discovery. These features are show cased in the new choria scout commands that are built entirely by using abilities of the generated clients. We also significantly simplified the code for choria req by using the same features.

We have nice menu based zsh completion, you can generate a completion script using choria completion --zsh, we’re looking for a contributor who can build a nice moden bash based completion script as our old one is a bit long in the tooth.

Shout out to Romain Tartière and Mike Newton for their contribution

We had a release quite recently but I wanted to release a number of Scout related features to early adopters, these releases are mainly focussed on Scout but includes a few bug fixes and new builds for Ubuntu Focal (20.04 LTS).

The big item here is that we have integrated Goss into the Scout framework and it can now run validations regularly. See the Scout Goss blog post for details.

You’ll also notice a new agent - scout - on your nodes, this gives API access to interact with Scout checks on Choria servers.

Additionally, we are starting to work on our documentation for Scout, an initial cut of this is also published today, this shows our Puppet integration, Prometheus integration and a bit about the events.

Thanks to Romain Tartière for contributions to these releases.

Read on for the full details.