New Project Visuals

Till recently our documentation had a mix of visual styles for diagrams - mixing icons from Cisco, AWS etc - I recently wanted to document the supported network topologies and realised I need a more unified visual style for the documentation.

For some time now I am using diagrams.net to generate diagrams for blog posts and such, this tool is ok for diagrams but what really sets it apart for me is that even when exporting a PNG file it can embed the diagram vector source in the resulting PNG image.

This means any image on the website can simply be loaded and edited as a vector in the diagram editor, this is huge for ease of maintenance of the website, docs etc.

After some googling I found the Affinity symbol set - a public domain icon set in SVG format. Using these I came up with set of on-brand colored icons for our various components you can see below.

See the full post for links to assets and libraries for diagrams.net.

[Read More]

Reducing connection overhead for branch office scenarios

Because Choria allows you to manage nodes spread all around the world, and because you might be working from your laptop, far away from the (bad) Wi-Fi access points that connects you through (bad) PLC to the (bad) internet connection from the (not bad) island you are on, you may experience inconvenient latency and unreliabilities.

The reason is quite simple: while the Choria servers maintain a permanent connection with the message broker, the Choria client has to establish a new connection with the middleware for each request. Latency and packet loss do not help with establishing TLS encrypted connections in a timely fashion.

But good news everyone! NATS — the messaging system Choria is built on — has built-in support for so-called leaf nodes which offer a solution to this problem.

[Read More]

April 2021 Releases

We’re pleased to announce the next set of Choria releases, these are bug and feature releases.

We’re starting to add the concept of a Service to Choria, a Service is a special kind of Agent that rather than requiring discovery and handling multiple results will only ever have 1 response. The Agents hosted as Services will form a load balanced group with High Availability and Reliability being the focus.

We will use these to create node inventory services, configuration services for Scout and eventually also move our AAA signing over to this format so that no TCP ports are needed other than the brokers. Foundational level features are being released today, but we are still working on the big picture here.

We have recreated the long broken choria plugin doc command and move the choria tool generate commands also into choria plugin, for plugin doc invoking the mco equivalent will call into Choria, but the old generate commands in mco was too different so invoking those will now fail inviting you to invoke choria plugin generate instead. Plugin documentation has been reformatted to look a bit nicer and now also support generating Markdown format output.

We updated our underlying NATS Server to version 2.2.2 which brings many stability and feature improvements. The main feature is a system called JetStream that is already enabled within Choria - though more on that at a later stage as we refine our particular use cases. If you wish to explore JetStream within Choria please reach out to us on the usual community channels.

A huge feature for us is that Websocket support has landed in the NATS. Today we do not yet expose these ports in Choria but I’d love to hear from the community who would prefer this rather than our traditional TCP ports.

Read all about NATS 2.2.2 on its announcement blog post.

Special thanks to Romain Tartière for his contributions in this release.

[Read More]

March 2021 Releases

We’re pleased to announce the next set of Choria releases, these are mainly bug fixes, but we have a few important changes to the Choria Server and Broker.

We have a new Registration plugin that will send all the data needed for discovery, previous supported plugin only read a specific file regularly, the new plugin will send all the active state - facts, classes, collectives and more. This is a first step towards building our own discovery database to replace our use of PuppetDB in the long run.

To configure the inventory_content registration plugin you can set:

choria::server_config:
  registration: inventory_content
  plugin.choria.registration.inventory_content.target: mcollective.ingest.discovery.%{facts.fqdn}
  plugin.choria.registration.inventory_content.compression: true

Replacing mcollective.ingest.discovery.%{facts.fqdn} with your subject of choice. The intention is to ingest this into our Streaming server - more detail below.

The Choria Broker is starting to use the NATS Account system to create isolation between different organisational units, today we move all clients and nodes into a choria account as a first step. If you are upgrading a cluster of Choria Brokers expect to see some errors related to this account being unknown. Once your entire cluster is upgraded it will resolve. There might be some short network splits during this time.

Additionally, we now enable a new system account that will have events published in it for:

  • connects and disconnects
  • authentication errors
  • server shutdowns
  • regular server states

There are also a number of broker system level APIs for building reports and more. See the full post for details.

We’re starting to expose a NATS JetStream based Streaming system, which we’ll call Choria Streaming, to ingest registration, scout status, system events and more for downstream processing and analysis.

This is a huge topic, one that we’re still working on for Choria framing so more details on that later, this release adds a number of configuration items related to that already.

The Puppet modules are now able to configure something called Leaf Nodes to facilitate access to Choria from remote offices and, especially, high latency destinations. A blog post will be published this week covering that.

Special thanks to Romain Tartière, Trey Dockendorf, Tim Meusel and Mark Frost for their contributions in this release.

[Read More]

February 2021 Releases

Hot on the heels of our January release we have a few small bug fixes to the previous release, and a number of very significant improvements to the discovery and configuration subsystems.

This is again a big release, and we suggest you do careful testing of your client applications in your testing environments after reading the Upgrade Notes in this post.

The focus of this release has been around Discovery and Configuration, we’ll let the planned module changes bake a bit longer to ensure we’re 100% stable where we are now before we undertake the next big change. Discovery features no fewer than 3 new discovery methods, we have the start of Data Providers in Compound Filters and exciting new project based configuration, read the full post for details.

Special thanks to Vincent Janelle, Romain Tartière and Ben Roberts for their contributions in this release.

[Read More]

January 2021 Releases

We have a number of releases today that will be the start of big changes in our modules. These releases will hopefully have a minor impact on users, but the next release or two will require some Hiera changes, so it’s worth keeping an eye on these. For the next while testing in your labs and dev environments is essential.

This is the beginning of a big push to once again simplify our deployment story. Choria started as a trivial way to install MCollective but things have changed quite a lot since then and unfortunately entropy has had its effect on our modules.

In addition to these changes we also have some pretty amazing additions to the Choria Servers.

Read on for the background and details of what’s to come.

On the community side we’ve set up a GitHub Discussions group for those who are not keen on signing up to Slack.

Special thanks to Tim Meusel, Vincent Janelle, Vadym Chepkov, Vladislav Kuspits and Romain Tartière for their contributions in this release.

[Read More]

November 2020 Releases

We have a number of small releases today, mainly quality of life changes - performance improvements and such.

The only major work here is around our Autonomous Agent feature, this lets you build managed finite state machines that can manage components on your machines without RPC interaction. This underpins our Scout checks and helps in IoT scenarios etc.

Today we’re adding 2 new watchers, an Apple HomeKit Button and a Timer. The HomeKit button is interesting in home automation scenarios where a Choria Autonomous Agent can appear to your Apple devices as a button that you can toggle from your Apple Home apps. Combined with the timer it’s possible to create an override button for HVAC, Fans etc that interrupts a normal managed schedule for a while. For example when watching a movie I don’t like having my extractor fan on, using any Apple device I can now set a 2 hour override, after 2 hours normal scheduled activity resumes so I don’t need to remember to re-enable the extractor.

In future releases we’ll add a Timer based maintenance window to Scout checks using the timer watcher.

We’re starting to work on supporting Puppet 7, progress is being made (thanks Tim!) but I think we have some way to go.

Special thanks to Tim Meusel and Romuald Conty for their contributions in this release.

[Read More]

Choria Server 0.17.0

Today we have quite a bumper release with significant updates for Choria Scout and the first step in improvements for AAA Service managed clients.

We added numerous Choria Scout CLI tools - choria scout status, choria scout trigger, choria scout maintenance and choria scout resume. These allow you to manage a fleet of Choria nodes that are performing Scout checks.

$ choria scout status dev1.example.net
+-----------------------+-------+------------+-------------------------------+
| NAME                  | STATE | LAST CHECK | HISTORY                       |
+-----------------------+-------+------------+-------------------------------+
| mailq                 | OK    | 1m20s      | OK OK OK OK                   |
| ntp_peer              | OK    | 1m32s      | OK OK OK OK OK OK OK OK OK OK |
| pki                   | OK    | 2m28s      | OK OK OK OK OK OK OK OK OK OK |
| puppet_failures       | OK    | 2m3s       | OK OK OK OK WA WA CR CR OK OK |
| puppet_run            | OK    | 24s        | OK OK OK                      |
| swap                  | OK    | 4m23s      | OK OK OK OK OK OK OK          |
| zombieprocs           | OK    | 2m23s      | OK OK OK OK OK OK OK OK OK OK |
| goss                  | OK    | 3m12s      | OK OK OK                      |
| heartbeat             | OK    | 57s        | OK OK OK OK OK OK OK OK OK OK |
+-----------------------+-------+------------+-------------------------------+

The choria req utility got a new --table format option and all the result rendering code got extracted into a reusable package.

[rip@dev1]% choria req package status package=zsh --table
Discovering nodes .... 2

2 / 2    0s [====================================================================] 100%

+------------------+--------+------------------+-------+------+------------+---------+
| SENDER           | ARCH   | ENSURE           | EPOCH | NAME | RELEASE    | VERSION |
+------------------+--------+------------------+-------+------+------------+---------+
| dev2.example.net | x86_64 | 5.0.2-34.el7_8.2 | 0     | zsh  | 34.el7_8.2 | 5.0.2   |
| dev1.example.net | x86_64 | 5.0.2-34.el7_8.2 | 0     | zsh  | 34.el7_8.2 | 5.0.2   |
+------------------+--------+------------------+-------+------+------------+---------+

We improved generated Go clients significantly by allowing them to have typical progress bars, choria req like result formatting, result parsing helpers, improved logging and faster discovery. These features are show cased in the new choria scout commands that are built entirely by using abilities of the generated clients. We also significantly simplified the code for choria req by using the same features.

We have nice menu based zsh completion, you can generate a completion script using choria completion --zsh, we’re looking for a contributor who can build a nice moden bash based completion script as our old one is a bit long in the tooth.

Shout out to Romain Tartière and Mike Newton for their contribution

[Read More]

Choria AAA Improvements

Choria introduced a Centralized AAA model in 2019 that alleviate the need for managing certificates of every user and allow you to integrate Choria into your enterprise identity providers for Authentication, Auditing and Authorization.

For controlled environments this model is a huge boom, but there was one annoying thing - the need to still issue a TLS certificate to communicate with Choria Brokers. In this mode, these certificates do not form part of the security model of Choria but was nonetheless required to exist, you could share them but that was frowned upon.

In our next release we will introduce a new broker type that significantly simplifies the AAA security model by allowing clients holding no certificates to interact, safely, with Choria networks.

[Read More]

Scout Goss Integration

In the Scout Announcement blog post I mentioned we are looking to integrate Goss into Scout and I wanted to post an update on that.

Background

Goss is something similar to serverspec - it lets you write unit tests about your nodes actual state rather than code used to build it. Goss definitions are written in YAML or JSON and supports Go templating for customization.

This model is well suited for the purposes of monitoring since you can write really in depth sets of validations and treat them as a single unit.

Goss is written in Go, very fast and thanks to a lot of work I did recently embeddable in other software.

[Read More]
scout