Choria Blog

Hot on the heels of our January release we have a few small bug fixes to the previous release, and a number of very significant improvements to the discovery and configuration subsystems.

This is again a big release, and we suggest you do careful testing of your client applications in your testing environments after reading the Upgrade Notes in this post.

The focus of this release has been around Discovery and Configuration, we’ll let the planned module changes bake a bit longer to ensure we’re 100% stable where we are now before we undertake the next big change. Discovery features no fewer than 3 new discovery methods, we have the start of Data Providers in Compound Filters and exciting new project based configuration, read the full post for details.

Special thanks to Vincent Janelle, Romain Tartière and Ben Roberts for their contributions in this release.

We have a number of releases today that will be the start of big changes in our modules. These releases will hopefully have a minor impact on users, but the next release or two will require some Hiera changes, so it’s worth keeping an eye on these. For the next while testing in your labs and dev environments is essential.

This is the beginning of a big push to once again simplify our deployment story. Choria started as a trivial way to install MCollective but things have changed quite a lot since then and unfortunately entropy has had its effect on our modules.

In addition to these changes we also have some pretty amazing additions to the Choria Servers.

Read on for the background and details of what’s to come.

On the community side we’ve set up a GitHub Discussions group for those who are not keen on signing up to Slack.

Special thanks to Tim Meusel, Vincent Janelle, Vadym Chepkov, Vladislav Kuspits and Romain Tartière for their contributions in this release.

We have a number of small releases today, mainly quality of life changes - performance improvements and such.

The only major work here is around our Autonomous Agent feature, this lets you build managed finite state machines that can manage components on your machines without RPC interaction. This underpins our Scout checks and helps in IoT scenarios etc.

Today we’re adding 2 new watchers, an Apple HomeKit Button and a Timer. The HomeKit button is interesting in home automation scenarios where a Choria Autonomous Agent can appear to your Apple devices as a button that you can toggle from your Apple Home apps. Combined with the timer it’s possible to create an override button for HVAC, Fans etc that interrupts a normal managed schedule for a while. For example when watching a movie I don’t like having my extractor fan on, using any Apple device I can now set a 2 hour override, after 2 hours normal scheduled activity resumes so I don’t need to remember to re-enable the extractor.

In future releases we’ll add a Timer based maintenance window to Scout checks using the timer watcher.

We’re starting to work on supporting Puppet 7, progress is being made (thanks Tim!) but I think we have some way to go.

Special thanks to Tim Meusel and Romuald Conty for their contributions in this release.

Today we have quite a bumper release with significant updates for Choria Scout and the first step in improvements for AAA Service managed clients.

We added numerous Choria Scout CLI tools - choria scout status, choria scout trigger, choria scout maintenance and choria scout resume. These allow you to manage a fleet of Choria nodes that are performing Scout checks.

$ choria scout status dev1.example.net
+-----------------------+-------+------------+-------------------------------+
| NAME                  | STATE | LAST CHECK | HISTORY                       |
+-----------------------+-------+------------+-------------------------------+
| mailq                 | OK    | 1m20s      | OK OK OK OK                   |
| ntp_peer              | OK    | 1m32s      | OK OK OK OK OK OK OK OK OK OK |
| pki                   | OK    | 2m28s      | OK OK OK OK OK OK OK OK OK OK |
| puppet_failures       | OK    | 2m3s       | OK OK OK OK WA WA CR CR OK OK |
| puppet_run            | OK    | 24s        | OK OK OK                      |
| swap                  | OK    | 4m23s      | OK OK OK OK OK OK OK          |
| zombieprocs           | OK    | 2m23s      | OK OK OK OK OK OK OK OK OK OK |
| goss                  | OK    | 3m12s      | OK OK OK                      |
| heartbeat             | OK    | 57s        | OK OK OK OK OK OK OK OK OK OK |
+-----------------------+-------+------------+-------------------------------+

The choria req utility got a new --table format option and all the result rendering code got extracted into a reusable package.

[rip@dev1]% choria req package status package=zsh --table
Discovering nodes .... 2

2 / 2    0s [====================================================================] 100%

+------------------+--------+------------------+-------+------+------------+---------+
| SENDER           | ARCH   | ENSURE           | EPOCH | NAME | RELEASE    | VERSION |
+------------------+--------+------------------+-------+------+------------+---------+
| dev2.example.net | x86_64 | 5.0.2-34.el7_8.2 | 0     | zsh  | 34.el7_8.2 | 5.0.2   |
| dev1.example.net | x86_64 | 5.0.2-34.el7_8.2 | 0     | zsh  | 34.el7_8.2 | 5.0.2   |
+------------------+--------+------------------+-------+------+------------+---------+

We improved generated Go clients significantly by allowing them to have typical progress bars, choria req like result formatting, result parsing helpers, improved logging and faster discovery. These features are show cased in the new choria scout commands that are built entirely by using abilities of the generated clients. We also significantly simplified the code for choria req by using the same features.

We have nice menu based zsh completion, you can generate a completion script using choria completion --zsh, we’re looking for a contributor who can build a nice moden bash based completion script as our old one is a bit long in the tooth.

Shout out to Romain Tartière and Mike Newton for their contribution

Choria introduced a Centralized AAA model in 2019 that alleviate the need for managing certificates of every user and allow you to integrate Choria into your enterprise identity providers for Authentication, Auditing and Authorization.

For controlled environments this model is a huge boom, but there was one annoying thing - the need to still issue a TLS certificate to communicate with Choria Brokers. In this mode, these certificates do not form part of the security model of Choria but was nonetheless required to exist, you could share them but that was frowned upon.

In our next release we will introduce a new broker type that significantly simplifies the AAA security model by allowing clients holding no certificates to interact, safely, with Choria networks.

In the Scout Announcement blog post I mentioned we are looking to integrate Goss into Scout and I wanted to post an update on that.

Background

Goss is something similar to serverspec - it lets you write unit tests about your nodes actual state rather than code used to build it. Goss definitions are written in YAML or JSON and supports Go templating for customization.

This model is well suited for the purposes of monitoring since you can write really in depth sets of validations and treat them as a single unit.

Goss is written in Go, very fast and thanks to a lot of work I did recently embeddable in other software.

We had a release quite recently but I wanted to release a number of Scout related features to early adopters, these releases are mainly focussed on Scout but includes a few bug fixes and new builds for Ubuntu Focal (20.04 LTS).

The big item here is that we have integrated Goss into the Scout framework and it can now run validations regularly. See the Scout Goss blog post for details.

You’ll also notice a new agent - scout - on your nodes, this gives API access to interact with Scout checks on Choria servers.

Additionally, we are starting to work on our documentation for Scout, an initial cut of this is also published today, this shows our Puppet integration, Prometheus integration and a bit about the events.

Thanks to Romain Tartière for contributions to these releases.

Read on for the full details.

We have a number of releases to announce today, the focus is general quality of life improvements in addition to the features to support out larger Choria Server release that included our announcement of Choria Scout.

With these releases you can create Scout checks on your machines using:

choria::scout_check{"check_typhon":
    plugin            => "/usr/lib64/nagios/plugins/check_procs",
    arguments         => '-C typhon -c {{ o "warn" 1 }}:{{ o "crit" 1 }}',
    remediate_command => "service typhon restart",
}

In addition to this we have fixed mco puppet runall when using Choria Server, I know quite a few people have wanted to see the return of this utility.

Thanks to Romain Tartière for contributions to these releases.

Yesterday I introduced a new Choria component called Scout which helps you build scalable monitoring pipelines. Today, we’ll look a bit at what makes a Scout install and how it is built.

In a follow up post I’ll dive a bit into Autonomous Agents - an infrequently used but very powerful building block found in Choria.

Overview

We’re happy to announce a new project called Choria Scout - a highly scalable system health monitoring framework and monitoring data pipeline released under the Apache 2.0 license.

Initially we support the ability to execute Nagios compatible plugins on Choria managed nodes with results sent centrally in a standard CloudEvents format, and optionally, integrated into Prometheus.

These are framework level building blocks that will in time be used to create a full monitoring stack built on Choria technologies. Checks and value overrides can already be configured using our Puppet modules. You can also use these building blocks to build entirely custom solutions for your own needs.

Scout will be a cloud native project with central components capable of being hosted on Kubernetes and using data formats supported by commercial clouds and projects like KNative. It will have a focus on integration, open data exchange and extensibility.

Despite being cloud native we will of course support monitoring anything where Choria, or the upcoming Scout agent, can run which includes traditional baremetal, VMs, containers and pods and small devices.