developing

MCollective has for a long time been extendible using purely Ruby means. This was fine in an earlier age where it seemed Ruby was going to rule the ops space but turned out that isn’t how things ended up, and even then there were lots of real interest in extending it using Python for example.

In the last 2 weeks this conversation came up again and in many respects the situation was worse now since the new Choria Server, being written in Go, is not extendible by external plugins, we supported the old Ruby agents but that was it. We had plans to support Tengo and Yaegi as ways to add agents to Choria but neither of those got past POC.

There were major hurdles to actually doing this in the new Go system:

• no action_policy type authorization system
• no data plugins
• no data aggregators
• limited DDLs
• do not have inputs and outputs properly mapped
• have no DDL validators
• have no way to set default data in requests

These missing features all worked fine for the Ruby agents though - since the MCollective compatability layer would just start up a subset of old MCollective on demand and so gain access to its versions of all of the above features. We did not need any of it.

In the last 3 weeks we addressed most of these missing features in the pure go daemon:

• We have shellsafe, ipv4address, ipv6address, ipaddress and regex validators. Analysis showed this hits 90%+ of what was ever used
• We have data aggregators for average, summary (including booleans) and a new chart one. Analysis showed these would hit 90%+ of what was ever used
• We have greatly extended the DDLs with full input, output and aggregate awareness
• We can set defaults in both inputs and outputs and it’s all type aware
• We can do full validation of requests based on the DDLs
• We have a action_policy plugin thats 1:1 compatible except for compound statements (planned)
• We can generate ruby DDLs from JSON ones

You’ll notice we are making deliberate choices about what we support here rather than expose a 100 extension points to the user - over extendibility was a real problem in the past - we now favour a batteries included approach where what people need is always available, new validators or aggregators would be via PR submission to the Go code and everyone benefit.

Above points represent a huge push in features, at this point if we add new ways to write agents they would get all these features for free and suddenly the prospect of doing just that is a lot more palatable. But why stop at supporting a specific language like tengo? Why not support all languages - especially with new movement in things like webasm?

That’s exactly what I did in a new feature called External Agents and it will be available in the next release, read on for the full details.