August 2022 Releases

It’s time for a minor release of Choria and a few related components, not a huge release as we are preparing a big refactor but still some user facing improvements - especially to the recently added App Builder component.

New Election features

We’ve used leader elections for a while internally for various components, these are backed by Choria Streams.

In this release we added an choria election command, using this command you can:

  • Manage, view, evict elections and leaders
  • Designate a particular node as a leader by managing a file under election using choria election file
  • Run a command under leader election using choria election run

See the Election documentation.

New Governor features

Governors can traditionally only control maximum concurrent executions, use this to limit how many nodes are actively running Puppet for example.

With a small change we were able to make them support a mode of maximum executions per period.

Imagine you have 10 nodes running the same cron job, if you need the job to run only on one of the servers every hour - essentially creating a failover pool - you can use the new --max-per-period flag.

See the Governor documentation.

Go Versions

The Go team released 1.19 last night, this means going forward we will support only go 1.18 and 1.19. The github.com/choria-io/go-choria tag v0.26.1 will be the last to work on Go 1.17.

Special thanks to Tim Meusel for his contributions to this release!

[Read More]

June 2022 Releases

It’s time for our next release and this one is packed full of goodies after quite a long development period. While our previous release was more about internal plumbing this one brings exciting user visible quality of life improvements and significant new features.

App Builder

We recently introduced App Builder, a operations tool for building custom CLI commands that wrap many tools into one single app.

In Choria we extend the App Builder to have the ability to interact with Key-Value Stores, initiate RPC requests and perform discovery.

To use this simply change your app symlink from appbuilder to choria.

See the App Builder Documentation for full reference of the extensions Choria adds.

choria Command UX

We forked the Kingpin CLI parser into a Choria project called Fisk that extends Kingpin in various ways and introduce some breaking changes. We now use Fisk in all our tools.

As part of moving to Fisk the help output from choria has been changed significantly to be shorter and easier to navigate with less superfluous information on every page.

We also add a new cheat feature to the CLI that gives you access to cheat sheet style help, here’s an example:

$ choria cheat req
# further documentation https://choria.io/docs/concepts/cli/

# request the status of a service
choria req service status service=httpd

...

Run choria cheat for a list of available cheats, contributions to these would be greatly appreciated!

Improved Testing

We now have full integration testing where real running clusters are built and real network round trips are tested, this is a significant step forward in achieving reliability in the long run.

We also now do daily tests of installing Choria using Puppet on every support Linux Distro.

Operating Systems

We now publish packages for EL9 and Ubuntu 22.04. Debian packages are tagged by their distribution name to help with mirroring.

We removed Ubuntu Xenial, Debian Stretch and EL6.

Contributors

Special thanks to Jonathan Matthews, Vincent Janelle, Nicolas Le Gaillart, Lena Schneider and Romain Tartière for their contributions.

[Read More]

Introducing App Builder

Today I am pleased to announce a new operations tool, it’s a small little thing, but it can be a huge help, I hope you will love it.

Operations teams tend to use a large selection of shell scripts, piped commands, kubectl invocations and more in their day to day job.

To a large extend these are tribal knowledge and something that is a big hurdle for new members of the team. The answer is often to write wiki pages capturing run books that has these commands documented.

This does not scale well and does not stay up to date.

What if there was a CLI tool that encapsulated all of these commands in a single, easy to use and easy to discover command.

The appbuilder project lets you build exactly that by specifying a model for your CLI application in a YAML file and then building custom interfaces on the fly.

There will be 2 versions of this, the standalone version mentioned above that has only non Choria related features, we will then use it as a library in the normal Choria release where it will gain discovery, rpc and kv support. You will see that in the next Choria release (0.26.0).

There is an optional video introducing the idea behind it.

See the full entry for an example of this tool in use and what it is all about.

View the Documentation and GitHub Repo for more

[Read More]

Supported OS and Go versions update

Just a small headsup to alert users of a few changes in the Operating Systems we support.

Operating Systems

We support Operating System packages primarily for our Puppet users, and, so we tend to follow along with their deprecations.

Yesterday Puppet announced they will drop support for Ubuntu Xenial which prompted me to do the same here. Xenial has been a long and painful distribution to support, I am eager to not have to deal with it anymore. At the same time we are removing support for Debian Stretch and EL6 (though we have not done packages for EL6 for a long time). We will not build packagers for these Operating Systems and future releases will not have any RPMs or DEBs published for them.

We will in the near term future archive the repositories that was used to serve these Operating Systems.

Golang

The Golang team announced Go 1.18 this week, in line with Go support policies we therefore also dropped support for Go 1.16 and made some breaking changes that would prevent Go 1.16 from compiling the code base. Nightly Choria builds are already done using 1.18 to give us ample time to test the new compiler.

These updates will slowly ripple through Puppet modules and other related projects also.

Stream Replicator 0.5.0

In the past we’ve had a project called Stream Replicator that was used to copy data between independent NATS Streaming Server instances. I’ve needed an updated version of this, see the full text for links to a brand new ground-up rewrite of this tool that support JetStream.

At a basic level the system simply takes all data in one Stream found in a cluster and copies it all to another stream in, potentially, another cluster. We maintain order and, it’s a long-running process so the 2 streams are kept up to date.

The streams can have different configurations - different storage types, different retention periods, different replication factors and more, even different subject spaces.

That’s the easy part, the harder part is where we meet some Choria specific needs. Choria Servers support sending chunky packets of metadata containing lots of metrics and metadata about the nodes. In a single Data Center sending this data frequently all is fine, but when you are operating 100s of thousands of nodes no central metadata store can realistically keep up with the demand this place on it. At a medium scale this can equal to many MB/sec.

The Choria Stream Replicator supports inspecting the data streams, tracking individual senders and sampling data out of the stream - sending data for any given node once per hour, while the node itself publishes every 5 minutes.

Using this method one can construct tree structures - city-level data centers feeding regional aggregators which in turn feed a large central data store. 5 minute freshness at the city level, 30 minutes at the region and hourly at the central.

Further, while replicating and sampling data the Stream Replicator will track nodes and send small advisories about new nodes, nodes not recently seen and nodes that are deemed retired. By ingesting these advisories regionally or centrally a real time view of global node availability can be built without the cost of actually processing node level data.

Choria Fleet Streams

Given the above diagram, the Replicator supports:

  1. Choria Fleet Nodes publish their metadata every 300 seconds
  2. Choria Data Adapters place the data in the CHORIA_REGISTRATION stream with per-sender identifying information
  3. Stream Replicator reads all messages in the CHORIA_REGISTRATION Stream
  4. Sampling is applied and advisories are sent to the CHORIA_REGISTRATION_ADVISORIES stream about node movements and health
  5. Sampled Fleet Node metadata is replicated to central into the CHORIA_REGISTRATION stream
  6. All advisories are replicated to central into the CHORIA_REGISTRATION_ADVISORIES stream without sampling

I gave a talk detailing this pattern at Cfgmgmt Camp 2019 that might explain the concept further.

This is quite niche stuff, though the Replicator would be generically useful, it’s tailored to the needs of our Large Scale Choria Deploy reference architecture.

[Read More]

February 2022 Releases

It’s been almost 5 months since our last release, not because nothing has been happening but because so much has been happening, good problems to have!

So this is a bit of a massive release, however I think the bulk of the changes will not affect our typical Puppet based users.

Choria Registry

This introduces first work of a new Choria Registry. We have a long-standing pain point around managing DDL files on clients, it’s a technical requirement to describe remote services but it’s just a pain to maintain, Puppet helps but for clients in CI, desktops etc, the DDL requirement is just too much.

Choria Server now has an option to act as a Registry where it can read it’s local DDL directory and serve that up to clients on demand. When a client tries to access a new agent it has never accessed before it will ask the registry for the DDL describing that agent. It will also do so regularly to ensure the local cache is still accurate.

This means that we can now have truly single-file client deployments. With just the choria binary and a running Registry that choria client can interact with the entire fleet and do everything it wants. This is a great improvement for deployment of client machines and making Choria more generally useful without Configuration Management.

The Choria Server can be a Registry, running multiple Servers with registry enabled will create a failure tolerant HA cluster of registry servers.

This is a brand-new feature, so I am not yet documenting it publicly, but I am keen to talk to users who wish to help in validating this before we look to supporting this more widely.

Non mTLS communications

The major work here that contributed to the 20 000 line code change in Choria Server is that we now support a secure non mTLS mode of communication. This is of no consequence for Puppet users so if that’s you feel free to skip this section.

With a typical deployment we use the Puppet CA to create a fully managed and closed mTLS based network. For some enterprises replicating that with their internal PKI infrastructure is nearly impossible. So we looked to, optionally, move away from a pure mTLS mode to a mixed setup where we use ED25519 keypair and signed JWTs to provide equivalent security.

Essentially we now have formalized our use of JWT into a new tokens package where servers and clients have their own JWT. We hope to move entirely over to this model in time as we were able to create a greatly enhanced security model:

  • Servers are restricted to only certain collectives, attempting to enter non defined collectives will be denied by the broker
  • Servers are restricted to only server traffic flows. A server token cannot make a request to any other server, enforced by the broker
  • Servers have a default deny permission set allow specific access to Streams, Governors, Hosting Services and being able to be a Submission Server
  • Clients have private reply channels, clients cannot view each others replies
  • In addition to Open Policy Agent a set of default deny permissions allowing access to use Streams, administer Streams, use Elections, view Events, use Governors etc

Using these settings moves us to a much more secure and private setup where even between 2 Choria Users traffic is now isolated and secure and this introduces the first of a security model around our adoption of Choria Streams. We cannot replicate these policies using just certificates. We hope to move even Puppet users to this model in future but that’s a big undertaking to get right without additional services.

To enable these features one needs to deploy AAA Service and Provisioner - and both of those had recent releases supporting this mode.

As mentioned this is not really a thing that Puppet users should worry about however those in large enterprises who deploy in non-Puppet ways should keep an eye out for incoming documentation around this feature.

Package Repository Changes

As notified back in September we are moving away from Packagecloud to our own package hosting infrastructure. I am keeping the Packagecloud infrastructure up for a while but this release and all future ones will not be uploaded there to promote users moving to the new infrastructure.

Thanks to Romain Tartière, Steffy Fort, Tim Meusel and Alexander Olofsson for their contributions to this release

[Read More]

Provisioning Improvements

The typical Choria Deployment method is to use Puppet to provisioning everything on the managed nodes. This works fine for those users, however on Large Scale this just does not really work.

Large Enterprises have a vastly varied infrastructure, and you simply do not find Puppet in use across all tiers. We therefore support provisioning Choria in a way that’s entirely configuration management free.

Essentially this is the “IoT Light-bulb” mode, you start a Choria Server and in short period of time its figured out how to provision itself, connected to the provisioning infrastructure and were on-boarded.

The Choria Provisioner can provision thousands of nodes a minute, is highly available and extendible and can integrate with enterprise CAs.

In August we blogged about some enhancements to make this processes better, today we follow up with further improvements. Read on for full details.

[Read More]

AAA Improvements

Choria supports a distributed authentication model as well as a centralised model using our Choria AAA Service. A Puppet user uses the distribution method by default.

In distributed mode every client has a certificate, signs his request with it and the certificate becomes the identity. The servers will verify using their RPC Authorization system if that certificate (id) can perform an action.

In the centralised setup each client do not have a certificate but it has a JWT token obtained from a sign-in service often using choria login. The JWT holds the identity, policies, permissions and more. The AAA Service signs requests using its certificate allowing clients to publish signed requests. Effectively the signing step gets outsourced to a trusted 3rd party. Before signing a request a policy is evaluated on the AAA Service to determine if the request should be allowed.

The AAA Service was introduced in 2019 and we’ve improved on it in 2020 by allowing a client certificate free operation.

The Certificate Free operation was a big win, however it came at a considerable cost of requiring additional Choria Brokers to take client connections.

We made a number of improvements in Release 0.6.0, read the full entry for details.

[Read More]

September 2021 Releases

Today we’re releasing the next Choria Server and a few Puppet modules. Primarily this is a bug fix and general improvement release with few real big ticket user facing items.

We have a major breaking change relating to our Package Repositories. For most people who use our public repositories nothing will change, but those using internal mirrors should probably read the full post for details. In short, we are moving from Package Cloud to our own infrastructure hosted in EU, UK and US. Our packages and repositories are now signed using our own keys.

We’ve had some great feedback on Choria Governors and we’ve improved the CLI tooling a bit, we’ve also added a new Puppet Type and Provider to manage these. Thanks to users who have been testing these new features.

We have an opt-in new feature that should significantly improve the default broadcast based discovery system. Usually we wait for 2 seconds for discovery results, but in most cases most discovery results came in within the first few 100ms. By setting plugin.choria.discovery.broadcast.windowed_timeout=1 in your client configuration file we now do a windowed discovery that will terminate if after the last received result no more results were received in 300ms. In most cases this will be a massive improvement in UX. Please test it, we aim to flip this to default on in near future.

We’ve had a big set of refactors on the Debian packaging and should have functioning Debian Bullseye packages for this release. There’s also been a few improvements to the Debian packages in general.

We have started the process of supporting a new style of agent called a Choria Service. These services will be used to perform AAA signing over the NATS protocol, to facilitate DDL free clients thanks to central Schema Registries and more. Today this is mainly under the cover improvements but expect big changes coming soon in areas of client deployment simplification.

Thanks to Romain Tartière, Romuald Conty and Tim Meusel for their contributions to this release

[Read More]

August 2021 Releases

This is the first release since April, and it’s a massive release bringing many enhancements and new features.

We are introducing Choria Streams - a Stream Processing framework built into the Choria Broker powered by NATS JetStream. I wrote a blog post about this Introducing Choria Streams that’s worth a read.

Additionally, we added Choria Key-Value Store, Choria Governor and Choria Message Submit all powered by Choria Streams and each in their own right a big feature.

Other major enhancements are that we now support Websockets for the network connections between Servers, Broker and Go clients.

Autonomous Agents now have a data layer meaning within an Autonomous Agent data can be fetched from stores like other Key-Value stores and this data can be accessed by Watchers at run time. We expose node facts to Autonomous Agents in the data layer. Additionally, we support watching Choria Key-Value Store for changes which updates the data layer and trigger transitions. Exec Watchers also support Governors to create orchestration-free rolling upgrades etc.

We made huge improvements to Provisioning, we blogged about this in Provisioning HA and Security. There you can also see we support Leader Election against Choria Streams as a library feature.

On the documentation front we added a big section about Choria Streams but also received permission to Open Source some documentation that shows how a very large - millions of nodes - Choria deployment might look. This is a proven design in active use in production for a few years already. We are busy building another such network at the moment, and a lot of the enhancements in Provisioning is as a result of this work. Find the document at Large Scale Design.

Thanks to Chris Boulton, Romain Tartière, Tim Meusel, Dominic Vallejo, Vincent Janelle and Franciszek Klajn for their contributions to this release

[Read More]