Recent versions of Choria all have Choria Streams ready to be used, though I suggest starting to use it only since version 0.23.0 and newer.
Traditionally Choria would allocate some memory per connection, little bits per subscription - but really you’d be fine with less than 1GB for 10s of thousands of connections.
Streams changes the equation a bit where now we need to persist data to disk, we need to use a lot of network resources to replicate data and obviously some CPU to manage all this.
On my network with 10s of nodes Memory and CPU is no problem, 1GB is plenty. In larger networks we’d say go for 8GB RAM, 4 to 8 cores of fast CPUs and fast SSD based disks.
If you want to publish data to Streams and Replicate that data 3 times - that means your data will traverse the network at least 4 times - could be even more. 1GB of messages can easily translate to 5GB or more of network usage. So you need to be careful with what kind of network you provision. Especially if you use network attached storage.
Direct attached storage is best, network attached block storage would also work - but keep in mind the considerations above. Under heavy load we’d depend a lot on fast IOPS.
There is no rule, you should test your workloads and be very sure to have monitoring on your servers that considers these new dimensions.
If you have a clustered Choria Broker cluster you need to enable Choria Streams on all nodes in the cluster using the Hiera data below:
choria::broker::stream_store: /var/lib/choria/ choria::broker::advisory_retention: 30d choria::broker::advisory_replicas: 1 choria::broker::event_retention: 30d choria::broker::event_replicas: 1 choria::broker::machine_retention: 30d choria::broker::machine_replicas: 1
When enabled Choria Streams will automatically create the following streams:
|CHORIA_EVENTS||Archive of Choria Cloud Events that you would see using
|CHORIA_MACHINE||Cloud Events published by the Choria Autonomous Agent system, including Scout check states|
|CHORIA_STREAM_ADVISORIES||Audit logs of Streams API access|
The full reference of available configuration options can be seen using
choria tool config network.stream,
not all settings are currently settable in Puppet.
Choria embeds key parts of the NATS management tool, for example
nats pub can be found in
nats broker pub.
By default it uses the Choria configuration and no extra settings are needed.
choria broker account info should report something like this:
$ choria broker account info Connection Information: Client ID: 7186 Client IP: 10.244.1.142 RTT: 57.874176ms Headers Supported: true Maximum Payload: 1.0 MiB Connected Cluster: CHORIA Connected URL: nats://choria.example.net:4222 Connected Address: 10.244.1.140:4222 Connected Server ID: NCRRT2R2R5O5FF3AMXO6VHKEH2ADITSCTVARZHXOKF2SISD4CUOTUWTS Connected Server Name: choria.example.net JetStream Account Information: Memory: 0 B of 0 B Storage: 1.2 GiB of 0 B Streams: 6 of 0 Consumers: 3 of 0
And the list of Streams can be seen:
$ choria broker stream report Obtaining Stream stats ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Stream Report │ ├──────────────────────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬──────────────────────────┤ │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ ├──────────────────────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼──────────────────────────┤ │ CHORIA_EVENTS │ File │ 0 │ 3,288 │ 1.0 MiB │ 0 │ 0 │ c1, c2, c3* │ │ CHORIA_STREAM_ADVISORIES │ File │ 0 │ 14,356 │ 14 MiB │ 0 │ 0 │ c1* │ │ CHORIA_MACHINE │ File │ 1 │ 142,753 │ 397 MiB │ 0 │ 0 │ c1, c2, c3* │ ╰──────────────────────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴──────────────────────────╯
As Choria Streams is a managed instance of NATS JetStream you can use the NATS docs for guidance on the various programming APIs.
We have sections here for various, tailored, features we built for the Choria Ecosystem using this technology.