Usage
CLI
The CLI tools have extensive –help output which will not be reproduced here, sample commands will be shown here but please refer to the help for full details.
Execution Flow
With the default Puppet Tasks runner from Puppet Inc the tasks are copied from your shell to remote nodes and run there. The flow in Choria is significantly different so I will show the steps that are followed when you execute a Task
- The metadata is fetched from your Puppet Server and the CLI is created
- Inputs are validated against the task signature
- The task files specification are sent to all your nodes and they will attempt to download the task files
- Request that nodes should run the Task - once it’s confirmed that the entire node set has the exact same task files and commands downloaded and cached
- Wait for up to 60 seconds for the Task to complete everywhere
At this point the task is either done or still executing - either way - you have a summary output that includes the task ID.
You can later retrieve task results, statuses and metadata from the nodes using the ID.
Deploying a Task
Tasks are delivered using Puppet modules much like anything else in the Puppet world. Uniquely to Tasks you only have to put the files on your Puppet Server module paths, you do not need to include any classes etc.
At present Choria will only consult your production environment for tasks.
You can therefore use puppet module, r10k or librarian puppet to place your modules in the production environment and that should be enough for them to be used by Choria
Discovering Tasks
A basic list of tasks can be fetched from the CLI:
$ mco tasks
Known tasks in the production environment
exec gcompute::instance gcompute::reset
gcompute::snapshot gcontainer::resize gpubsub::publish
gsql::clone gsql::passwd gstorage::upload
choria::ping puppet_conf
Use mco task <TASK> to see task help
Pass option --detail to see task descriptions
Passing –detail will request their descriptions, but please note that due to shortcomings in the Puppet API this is quite slow as a REST request has to be made for every task.
You can view details about one specific task:
$ mco tasks puppet_conf
Retrieving task metadata for task puppet_conf from the Puppet Server
puppet_conf - Inspect puppet agent configuration settings
Task Parameters:
action The operation (get, set) to perform on the configuration setting (Enum[get, set])
section The section of the config file. Defaults to main (Optional[String[1]])
setting The name of the config entry to set/get (String[1])
value The value you are setting. Only required for set (Optional[String[1]])
Task Files:
init.rb 1231 bytes
Use 'mco tasks run puppet_conf' to run this task
Running a Task
Tasks are always run dissociated from the mcollectived, you can therefore use them to restart MCollective, do system upgrades and other long running tasks.
Lets look at the task input first
$ mco tasks run puppet_conf
The action option is mandatory
The setting option is mandatory
Please run with --help for detailed help
$ mco tasks run puppet_conf --help
Puppet Task Orchestrator
Usage:
mco tasks run <TASK NAME> [OPTIONS]
Runs a task in the background and wait up to 50 seconds for it to complete.
....
Application Options
--action ACTION The operation (get, set) to perform on the configuration setting (Enum[get, set])
--section SECTION The section of the config file. Defaults to main (Optional[String[1]])
--setting SETTING The name of the config entry to set/get (String[1])
--value VALUE The value you are setting. Only required for set (Optional[String[1]])
--summary Only show a overall summary of the task
--background Do not wait for the task to complete
--input INPUT JSON input to pass to the task
-c, --config FILE Load configuration from file rather than default
-v, --verbose Be verbose
-h, --help Display this screen
I’ve removed the bulk of the Help output here, the thing to note are the parameters like –action, these are inputs defined by the task exposed on the CLI.
In most cases Choria will attempt to convert a string like 1 into an Integer when that is what the task need so for most tasks you can pass the options as normal. You can though also use a YAML or JSON file as input, for example:
{
"action":"set",
"section":"user",
"setting":"modulepath",
"value":"/tmp/modules"
}
You could save this file to input.json and run mco tasks run puppet_conf –input @input.json and it will use these settings, here we’ll show supplying them on the CLI though:
$ mco tasks run puppet_conf --action set --section user --setting modulepath --value /tmp/modules -I node1.example.net
Retrieving task metadata for task puppet_conf from the Puppet Server
Attempting to download and run task puppet_conf on 1 nodes
Downloading and verifying 1 file(s) from the Puppet Server to all nodes: ✓ 1 / 1
Running task puppet_conf and waiting up to 60 seconds for it to complete
Summary for task 45250c07824f5922be68468d08f6b76c
Task Name: puppet_conf
Caller: choria=rip.mcollective
Completed: 1
Running: 0
Successful: 1
Failed: 0
Average Run Time: 1.39s
Here as is typical for MCollective it will only show failures, you can later retrieve the output the command produced:
$ mco tasks status 45250c07824f5922be68468d08f6b76c -v -I node1.example.net
Discovering hosts using the choria method .... 1
node1.example.net
{"status":"/tmp/modules","setting":"modulepath","section":"user"}
Summary for task 45250c07824f5922be68468d08f6b76c
Task Name: puppet_conf
Caller: choria=rip.mcollective
Completed: 1
Running: 0
Successful: 1
Failed: 0
Average Run Time: 1.39s
Overview metadata about a task can be retrieved:
$ mco tasks status 45250c07824f5922be68468d08f6b76c --metadata -I node1.example.net
Requesting task metadata for request 45250c07824f5922be68468d08f6b76c
* [ ============================================================> ] 1 / 1
node1.example.net
puppet_conf by choria=rip.mcollective at 2018-03-19 13:50:51
completed: yes runtime: 1.39 stdout: yes stderr: no
Finished processing 1 / 1 hosts in 171.36 ms
And finally you can retrieve all the output and statuses as JSON:
$ mco tasks status 45250c07824f5922be68468d08f6b76c --json -I node1.example.net | jq .
{
"items": [
{
"host": "node1.example.net",
"task": "puppet_conf",
"callerid": "choria=rip.mcollective",
"exitcode": 0,
"stdout": {
"status": "/tmp/modules",
"setting": "modulepath",
"section": "user"
},
"stderr": "",
"completed": true,
"runtime": 1.386015831,
"start_time": 1521467451
}
],
"stats": {
"names": [
"puppet_conf"
],
"callers": [
"choria=rip.mcollective"
],
"completed": 1,
"running": 0,
"task_not_known": 0,
"wrapper_failure": 0,
"success": 1,
"failed": 0,
"average_runtime": 1.386015831,
"noresponses": []
},
"summary": {
"nodes": 1,
"taskid": "45250c07824f5922be68468d08f6b76c",
"completed": true,
"success": true
}
}
Discovery Integration
In the examples above it was kind of annoying to keep typing the hostname via -I, and it would not work really if you had many nodes or wanted to address only nodes that completed a task (or not).
Choria provides a discovery data source that exposes a lot of data about a task to the MCollective discovery system.
Let’s see what data the plugin provides and then we’ll use what we saw to run a command on all nodes where a previous one failed:
$ mco plugin doc data/bolt_task
bolt_task
=========
Information about past Bolt Task
Author: R.I.Pienaar <rip@devco.net>
Version: 0.6.0
License: Apache-2.0
Timeout: 1
Home Page: https://choria.io
This data plugin let you extract information about a previously
run Bolt Task for use in discovery and elsewhere.
To run a task on nodes where one previously failed:
mco tasks run myapp::update -S "bolt_task('ae561842dc7d5a9dae94f766dfb3d4c8').exitcode > 0"
QUERY FUNCTION INPUT:
Description: The Task ID to retrieve
Prompt: Task ID
Type: string
Validation: ^[a-z,0-9]{32}$
Length: 32
QUERY FUNCTION OUTPUT:
caller:
Description: The user who invoked the task
Display As: Invoked by
completed:
Description: Did the task complete running
Display As: Completed
exitcode:
Description: The exitcode from the task
Display As: Exit Code
known:
Description: If this is a known task on this node
Display As: Known Task
runtime:
Description: How long the task took to run
Display As: Runtime
spool:
Description: Where on disk the task status is stored
Display As: Spool
start_time:
Description: When the task was started, seconds since 1970 in UTC time
Display As: Start Time
stderr:
Description: The STDERR output from the task
Display As: STDERR
stdout:
Description: The STDOUT output from the task
Display As: STDOUT
task:
Description: The name of the task that was run
Display As: Task
wrapper_error:
Description: Error output from the wrapper command
Display As: Wrapper Error
wrapper_pid:
Description: The PID of the wrapper that runs the task
Display As: Wrapper PID
wrapper_spawned:
Description: Did the wrapper start successfully
Display As: Wrapper Spawned
So there’s a ton of information about past tasks, lets assume we ran a Task to upgrade some software and where the upgrade failed we have to do some remediation.
The original task that failed has id ae561842dc7d5a9dae94f766dfb3d4c8:
$ mco tasks run acme::recover -S "bolt_task('ae561842dc7d5a9dae94f766dfb3d4c8').exitcode > 0"
This will query your network for all nodes where the task had a exitcode of greater than 0 and then it will run this acme::recover task on them.
Playbooks
As of version 0.9.0 of the choria/choria module you can use Tasks from within Choria Playbooks.
Running a Task
The integration is intended to be used by other Playbooks, let’s look at using the Package Task to retrieve the version of the puppet-agent package:
plan acme::agent_status () {
$nodes = choria::discover(
"agents" => ["bolt_tasks"]
)
$results = choria::run_playbook("choria::tasks::run",
"nodes" => $nodes,
"task" => "package",
"silent" => true,
"inputs" => {
"name" => "puppet-agent",
"action" => "status"
}
)
$results.reduce({}) |$memo, $result| {
$memo + {$result.host => $result["data"]["stdout"].parsejson}
}
}
Here we use the choria::tasks::run playbook to perform the same basic Flow that the CLI does - get metadata, validate input, download task files, run task and wait up to 60 seconds.
The examples shown were made using our Vagrant Demo environment
We use the Choria::Results set to create a simplified result status, this might look like this:
$ mco playbook run acme::agent_status
Notice: Scope(<module>/acme/plans/agent_status.pp, 2): Discovered 3 node(s) in node set task_nodes
Info: Scope(<module>/choria/plans/tasks/download_files.pp, 12): Downloading files for task 'package' onto 3 nodes
Notice: Scope(<module>/choria/plans/tasks/download_files.pp, 14): About to run task: mcollective task
Notice: Scope(<module>/choria/plans/tasks/download_files.pp, 14): Starting request for bolt_tasks#download against 3 nodes
Notice: Scope(<module>/choria/plans/tasks/download_files.pp, 14): Successful request de3266400e165b76aa165c1f29f1063c for bolt_tasks#download in 1.01s against 3 node(s)
Info: Scope(<module>/choria/plans/tasks/run.pp, 43): Running task 'package' on 3 nodes
Notice: Scope(<module>/choria/plans/tasks/run.pp, 45): About to run task: mcollective task
Notice: Scope(<module>/choria/plans/tasks/run.pp, 45): Starting request for bolt_tasks#run_and_wait against 3 nodes
Notice: Scope(<module>/choria/plans/tasks/run.pp, 45): Successful request ca87087b91f950b6a226dd52d33f7c42 for bolt_tasks#run_and_wait in 3.62s against 3 node(s)
Plan acme::agent_status ran in 5.93 seconds: OK
Result:
{
"choria1.choria": {
"status": "up to date",
"version": "5.5.1-1.el7"
},
"choria0.choria": {
"status": "up to date",
"version": "5.5.1-1.el7"
},
"puppet.choria": {
"status": "up to date",
"version": "5.5.1-1.el7"
}
}
The choria::tasks::run playbook supports options for batching requests and more, please review the comments in its code.
This is a normal task invocation, so you could for example use mco tasks status ca87087b91f950b6a226dd52d33f7c42 to later look at this task from the CLI.
Retrieving task Status
You can also retrieve the status for a specific task in a Playbook:
# ....
$results = choria::run_playbook("choria::tasks::status",
"nodes" => $nodes,
"task_id" => "ca87087b91f950b6a226dd52d33f7c42",
)
# ....
Waiting for a task to complete
And finally if you started a task across a set of nodes and this task might run for a very long time you can pause your Playbook while this is happening:
# ...
choria::run_playbook("choria::tasks::wait",
"nodes" => $nodes,
"task_id" => "ca87087b91f950b6a226dd52d33f7c42",
)
# ...
Here the Playbook system will regularly fetch the status and wait for all nodes to complete. By default it will check 90 times with a 20 second sleep - so for 30 minutes. You can influence these parameters using tries and sleep arguments to the choria::tasks::wait playbook.