machines

When we first introduced our Lifecycle Events we mentioned efforts by cloudevents.io to standardize a format and set of transports but that it was a bit early in their life to adopt it.

Cloud Events is now a CNCF Incubator project and they reached CloudEvents 1.0 which means there is now a standard to work around that is stable. A good introduction to the concept can be found in The CloudEvents Spec Seeks to Bring Uniformity to Event Data Descriptions.

Since then adoption of this format has exploded - a whole bunch of companies and projects support ingesting and routing CloudEvents, examples: solo.io with their event powered API gateway, OpenFaaS, Serverless.com Event Gateway, Oracle Cloud, Azure, Knative and many many more.

So for us the time is ripe to adjust the format of our events to support CloudEvents 1.0. As of the next release of the Choria Server the Life Cycle events and Autonomous Agent Events are all been transitioned to CloudEvents format.

Today we’re launching a significant new feature that allow you to create a kind of automation that run on your nodes and do not need RPC interactions to initiate actions. We call it Choria Autonomous Agents or Choria Machine, it’s available as a preview feature in Choria Server 0.11.0

These run forever and continuously interact with your node, they keep working if the node is disconnected from the middleware and do not require a central component to function.

This release is a feature preview release, there are significant shortcomings and missing features but it’s already functional. We are launching this feature very early to solicit feedback and ideas to help us prioritize future work.

Overview

The typical orchestrations that people have done with MCollective or Choria has always taken the form of a conductor that tells the fleet what to do every step of the way.

This works fine for a lot of things especially if you use features like Sub Collectives to create isolated network-near groups where you run a daemon that orchestrates just the little cluster. It’s work so well in fact that this has always just been acceptable.

Unfortunately there are number of draw backs to this:

• It requires a lot of network traffic as one entity communicate constantly with the fleet
• It does not scale to complex tasks
• The orchestrator, network and brokers are all single point of failures, any failure anywhere means the managed component is not managed anymore
• The central orchestrator can get very complex as it might need to maintain lots of state for every node

Mark Burgess has a little anecdote about this, the Mayor of a city does not constantly tell every street sweeper where and how to do their job, the sweepers are trained to do their thing on their own and so a city scales by applying this concept on every level.

For years I have tried to build some form of autonomous agent that let us describe a system being managed and it will constantly be managed. The conceptual component is a Finite State Machine - nothing new about this - but I always had concerns about visibility and operability. Recent advances in tools like Prometheus, but also my own work in events from the Choria daemons, have made this much more viable.

I think of this a bit like a Kubernetes Operator but for anything in any environment.