1
0
Fork 0
mirror of https://github.com/postmannen/ctrl.git synced 2024-12-14 12:37:31 +00:00
ctrl/README.md

809 lines
32 KiB
Markdown
Raw Normal View History

2021-01-25 14:18:11 +00:00
# steward
2021-09-10 08:28:50 +00:00
Command & Control something that are either allways online or occasionally available online ? like Servers or IOT platforms where the link for reaching them can be a not-always-connected satellite, GSM, or even reliable fiber links that might fail ?
2021-04-16 19:19:19 +00:00
2021-09-10 08:28:50 +00:00
The idea behind Steward is to help out with exactly that, and let you handle the state of your servers/containers/nodes.
- [steward](#steward)
- [What is it ?](#what-is-it-)
- [Overview](#overview)
- [Inspiration](#inspiration)
- [Why](#why)
- [Publishing and Subscribing processes](#publishing-and-subscribing-processes)
- [Publisher](#publisher)
- [Subscriber](#subscriber)
- [Logical structure](#logical-structure)
- [Terminology](#terminology)
- [Features](#features)
- [Messages in order](#messages-in-order)
- [Messages not in order](#messages-not-in-order)
- [Error messages from nodes](#error-messages-from-nodes)
- [Message handling and threads](#message-handling-and-threads)
- [Timeouts and retries](#timeouts-and-retries)
- [Flags and configuration file](#flags-and-configuration-file)
- [Request Methods](#request-methods)
- [REQCliCommand](#reqclicommand)
- [REQnCliCommand](#reqnclicommand)
- [REQnCliCommandCont](#reqnclicommandcont)
- [REQTailFile](#reqtailfile)
- [REQHttpGet](#reqhttpget)
- [REQHello](#reqhello)
- [REQErrorLog](#reqerrorlog)
- [Request Methods used for reply messages](#request-methods-used-for-reply-messages)
- [REQToConsole](#reqtoconsole)
- [REQToFileAppend](#reqtofileappend)
- [REQToFile](#reqtofile)
- [Errors reporting](#errors-reporting)
- [Prometheus metrics](#prometheus-metrics)
- [Other](#other)
- [Howto](#howto)
- [Build and Run](#build-and-run)
- [Options for running](#options-for-running)
- [How to Run](#how-to-run)
- [Nkey Authentication](#nkey-authentication)
- [nats-server (the message broker)](#nats-server-the-message-broker)
- [Server config with nkey authentication](#server-config-with-nkey-authentication)
- [Steward](#steward-1)
- [Message fields explanation](#message-fields-explanation)
- [How to send a Message](#how-to-send-a-message)
- [Send to socket with netcat](#send-to-socket-with-netcat)
- [The current `operation`'s that are available are](#the-current-operations-that-are-available-are)
- [Sending a command from one Node to Another Node](#sending-a-command-from-one-node-to-another-node)
- [Example JSON for appending a message of type command into the `socket` file](#example-json-for-appending-a-message-of-type-command-into-the-socket-file)
- [Specify more messages at once do](#specify-more-messages-at-once-do)
- [Send the same message to several hosts by using the toHosts field](#send-the-same-message-to-several-hosts-by-using-the-tohosts-field)
- [Send an Op Command message for process listing with custom timeout and amount of retries](#send-an-op-command-message-for-process-listing-with-custom-timeout-and-amount-of-retries)
- [Send and Op Command to stop a subscriber on a node](#send-and-op-command-to-stop-a-subscriber-on-a-node)
- [Send and Op Command to start a subscriber on a node](#send-and-op-command-to-start-a-subscriber-on-a-node)
- [Concepts/Ideas](#conceptsideas)
- [Naming](#naming)
- [Subject](#subject)
- [Complete subject example](#complete-subject-example)
- [TODO](#todo)
2021-09-16 03:31:04 +00:00
- [Implements a replyOpts message field](#implements-a-replyopts-message-field)
- [Implements a repeat and repeatInterval message field](#implements-a-repeat-and-repeatinterval-message-field)
- [Split out the different opCmd's into it's own request types](#split-out-the-different-opcmds-into-its-own-request-types)
2021-09-13 11:24:28 +00:00
- [Move the processes map into it's own go routine](#move-the-processes-map-into-its-own-go-routine)
2021-09-10 08:28:50 +00:00
- [Services at startup of Steward. Could be implemented by having a local folder of messages to go through at startup. What is needed](#services-at-startup-of-steward-could-be-implemented-by-having-a-local-folder-of-messages-to-go-through-at-startup-what-is-needed)
- [A carrier type of message, that are able to forward a message on behalf of others, and are not directly ment for itself](#a-carrier-type-of-message-that-are-able-to-forward-a-message-on-behalf-of-others-and-are-not-directly-ment-for-itself)
- [Workflow request type](#workflow-request-type)
- [Add Op option the remove messages from the queue on nodes](#add-op-option-the-remove-messages-from-the-queue-on-nodes)
- [Disclaimer](#disclaimer)
2021-04-16 19:19:19 +00:00
2021-03-11 16:14:43 +00:00
## What is it ?
2021-04-16 19:19:19 +00:00
Command And Control anything like Servers, Containers, VM's or others by creating and sending messages with methods who will describe what to do. Steward will then take the responsibility for making sure that the message are delivered to the receiver, and that the method specified are executed with the given parameters defined. An example of a message.
2021-02-03 08:06:37 +00:00
2021-04-15 05:52:42 +00:00
```json
[
{
"directory":"cli_command__result",
"fileName": "some-file-name.result",
2021-04-15 05:52:42 +00:00
"toNode": "ship1",
"data": ["bash","-c","sleep 5 & tree ./"],
2021-04-15 05:52:42 +00:00
"method":"REQCliCommand",
"replyMethod":"REQToFileAppend",
"ACKTimeout":5,
2021-04-15 05:52:42 +00:00
"retries":3,
"replyACKTimeout":5,
"replyRetries":3,
"methodTimeout": 10
2021-04-15 05:52:42 +00:00
}
]
```
If the receiver `toNode` is down when the message was sent, it will be retried until delivered within the criterias set for `timeouts` and `retries`.
## Overview
Send Shell Commands, HTTP Get, or Tail log files to control your servers by passing a message that will have guaranteed delivery if/when the subsribing node is available. Or for example send logs or metrics from an end node back to a central log subscriber. The result of the method executed will be delivered back to you from the node you sent it from.
2021-08-13 14:21:35 +00:00
Steward uses Nats as message passing architecture for the commands back and forth from nodes, where delivery is guaranteed, and where all of the processes in the system are running concurrently so if something breaks or some process is slow it will not affect the handling and delivery of the other messages in the system.
2021-02-03 07:28:21 +00:00
2021-08-13 14:21:35 +00:00
By default the system guarantees that the order of the messages are handled by the subscriber in the order they where sent. There have also been implemented a special type `NOSEQ` which will allow messages within that process to be handles in a not sequential manner. This is handy for jobs that will run for a long time, and where other messages are not dependent on it's result.
2021-03-11 16:14:43 +00:00
2021-08-13 14:21:35 +00:00
A node can be a server running any host operating system, a container living in the cloud somewhere, a rapsberry pi, or something else that needs to be controlled that have an operating system installed.
2021-03-26 09:52:36 +00:00
## Inspiration
2021-04-15 05:52:42 +00:00
The idea for how to handle processes, messages and errors are based on Joe Armstrongs idea behind Erlang described in his Thesis <https://erlang.org/download/armstrong_thesis_2003.pdf>.
2021-08-13 14:21:35 +00:00
Joe's documents describes how to build a system where everything is based on sending messages back and forth between processes in Erlang, and where everything is done concurrently. I used those ideas as inspiration for building a fully concurrent system to control servers or container based systems by passing messages between processes asynchronously to execute methods, handle errors if they occur, or handle the retrying if something fails.
2021-04-15 05:52:42 +00:00
Steward is written in programming language Go with NATS as the message broker.
2021-03-26 09:52:36 +00:00
## Why
2021-02-03 08:06:37 +00:00
2021-08-13 14:21:35 +00:00
With existing solutions there is often either a push or a pull kind of setup to control the nodes.
2021-02-03 08:06:37 +00:00
2021-02-17 12:14:19 +00:00
In a push setup the commands to be executed is pushed to the receiver, but if a command fails because for example a broken network link it is up to you as an administrator to detect those failures and retry them at a later time until it is executed successfully.
2021-02-03 08:06:37 +00:00
2021-03-11 16:14:43 +00:00
In a pull setup an agent is installed at the Edge unit, and the configuration or commands to execute locally are pulled from a central repository. With this kind of setup you can be pretty certain that sometime in the future the node will reach it's desired state, but you don't know when. And if you want to know the current state you will need to have some second service which gives you that information.
2021-02-03 08:06:37 +00:00
2021-08-13 14:21:35 +00:00
In it's simplest form the idea about using an event driven system as the core for management of Edge units is that the sender/publisher are fully decoupled from the receiver/subscriber. We can get an acknowledge if a message is received or not, and with this functionality we will at all times know the current state of the receiving end.
2021-02-03 07:28:21 +00:00
2021-04-15 05:52:42 +00:00
## Publishing and Subscribing processes
2021-03-26 09:48:05 +00:00
All parts of the system like processes, method handlers, messages, error handling are running concurrently.
If one process hangs on a long running message method it will not affect the rest of the system.
### Publisher
2021-08-13 14:21:35 +00:00
- A message in valid format is appended to the in socket.
2021-03-26 09:48:05 +00:00
- The message is picked up by the system and put on a FIFO ringbuffer.
2021-08-13 14:21:35 +00:00
- The method type of the message is checked, a subject is created based on the content of the message, and a publisher process to handle the message type for that specific receiving node is started if it does not exist.
2021-03-26 09:48:05 +00:00
- The message is then serialized to binary format, and sent to the subscriber on the receiving node.
- If the message is expected to be ACK'ed by the subcriber then the publisher will wait for an ACK if the message was delivered. If an ACK was not received within the defined timeout the message will be resent. The amount of retries are defined within the message.
### Subscriber
2021-04-15 05:52:42 +00:00
- The receiving end will need to have a subscriber process started on a specific subject and be allowed handle messages from the sending nodes to execute the method defined in the message.
2021-03-26 09:48:05 +00:00
- When a message have been deserialized, it will lookup the correct handler for the method type specified within the message, and execute that handler.
2021-04-15 05:52:42 +00:00
- If the output of the method called is supposed to be returned to the publiser it will do so by using the replyMethod specified, and pick up the next message in the queue.
2021-03-26 09:48:05 +00:00
### Logical structure
2021-07-01 05:42:45 +00:00
![overview](doc/steward.svg)
2021-03-11 16:14:43 +00:00
## Terminology
2021-02-17 12:14:19 +00:00
- Node: Something with an operating system that have network available. This can be a server, a cloud instance, a container, or other.
2021-04-15 05:52:42 +00:00
- Process: A message handler that knows how to handle messages of a given subject concurrently.
- Message:
- Command: Something to be executed on the message received. An example can be a shell command.
- Event: Something that have happened. An example can be transfer of syslog data from a host.
2021-03-11 16:14:43 +00:00
## Features
### Messages in order
2021-03-11 16:14:43 +00:00
- By default the system guarantees that the order of the messages are handled by the subscriber in the order they where sent. So if a network link is down when the message is being sent, it will automatically be rescheduled at the specified interval with the given number of retries.
2021-04-15 05:52:42 +00:00
These types of messages have method starting with `REQ<Method name>`
### Messages not in order
2021-04-15 05:52:42 +00:00
- There have been implemented a special method type `REQn<Method name>` which will allow messages to be handled within that process in a not sequential manner. This is handy for jobs that will run for a long time, and where other messages are not dependent on it's result.
### Error messages from nodes
2021-03-11 16:14:43 +00:00
- Error messages will be sent back to the central error handler upon failure on a node.
### Message handling and threads
2021-08-13 14:21:35 +00:00
- The handling of all messages is done by spawning up a process for handling the message in it's own thread. This allows us to individually down to the message level keep the state for each message both in regards to ACK's, error handling, send retries, and rerun of a method for a message if the first run was not successful.
2021-03-11 16:14:43 +00:00
- Processes for handling messages on a host can be restarted upon failure, or asked to just terminate and send a message back to the operator that something have gone seriously wrong. This is right now just partially implemented to test that the concept works.
2021-08-13 14:21:35 +00:00
- Publisher Processes on a node for handling new messages for new nodes will automatically be spawned when needed if it does not already exist.
2021-03-11 16:14:43 +00:00
- Publishing processes will potentially be able to send to all nodes. It is the subscribing nodes who will limit from where and what they will receive from.
2021-03-11 16:14:43 +00:00
- Messages not fully processed or not started yet will be automatically handled in chronological order if the service is restarted since the current state of all the messages being processed are stored on the local node in a key value store until they are finished.
2021-03-11 16:14:43 +00:00
- All messages processed by a publisher will be written to a log file as they are processed, with all the information needed to recreate the same message if needed, or it can be used for auditing.
2021-03-05 12:10:46 +00:00
2021-03-11 16:14:43 +00:00
- All handling down to the process and message level are handled concurrently. So if there are problems handling one message sent to a node on a subject it will not affect the messages being sent to other nodes, or other messages sent on other subjects to the same host.
2021-03-01 09:16:07 +00:00
2021-03-11 16:14:43 +00:00
- Message types of both ACK and NACK, so we can decide if we want or don't want an Acknowledge if a message was delivered succesfully.
Example: We probably want an ACK when sending some CLICommand to be executed, but we don't care for an acknowledge (NACK) when we send an "hello I'm here" event.
2021-03-29 12:00:28 +00:00
### Timeouts and retries
- Default timeouts to wait for ACK messages and max attempts to retry sending a message are specified upon startup. This can be overridden on the message level.
- Timeout's can be specified on both the message, and the method. With other words a message can have a timeout, and for example if the method it will trigger is a shell command it can have it's own timeout so processes can have a timeout if they get stuck.
### Flags and configuration file
Steward supports both the use of flags/arguments set at startup, and the use of a config file. But how it is used might be a little different than how similar use is normally done.
A default config file will be created at first startup if one does not exist, with standard defaults values set. Any value also provided via a flag will also be written to the config file. If Steward is restarted the current content of the config file will be used as the new defaults. Said with other words, if you restart Steward without any flags specified the values of the last run will be read from the config file and used.
If new values are provided via flags they will take precedence over the ones currently in the config file, and they will also replace the current value in the config file, making it the default for the next restart.
The config file can also be edited directly, making the use of flags not needed.
If just getting back to standard default for all config options needed, then delete the current config file, restart Steward, and a new config file with all the options set to it's default values will be created.
2021-04-13 15:48:25 +00:00
### Request Methods
2021-08-13 14:21:35 +00:00
#### REQCliCommand
2021-04-13 15:59:30 +00:00
2021-04-13 15:48:25 +00:00
Run CLI command on a node. Linux/Windows/Mac/Docker-container or other.
2021-06-08 18:52:45 +00:00
Will run the command given, and return the stdout output of the command when the command is done.
2021-08-13 14:21:35 +00:00
#### REQnCliCommand
Run CLI command on a node. Linux/Windows/Mac/Docker-container or other.
Will run the command given without the execution order guarantee, and return the stdout output of the command when the command is done.
2021-06-08 18:52:45 +00:00
#### REQnCliCommandCont
Run CLI command on a node. Linux/Windows/Mac/Docker-container or other.
Will run the command given, and return the stdout output of the command continously while the command runs.
2021-04-13 15:59:30 +00:00
#### REQTailFile
2021-08-13 14:21:35 +00:00
Tail log files on some node, and get the result for each new line read sent back in a reply message until timeout is reached.
2021-04-13 15:48:25 +00:00
2021-04-13 15:59:30 +00:00
#### REQHttpGet
2021-04-13 15:48:25 +00:00
Scrape web servers, and get the html sent back in a reply message.
2021-04-13 15:59:30 +00:00
#### REQHello
2021-04-13 15:48:25 +00:00
Get Hello messages from all running nodes.
2021-04-13 15:59:30 +00:00
#### REQErrorLog
2021-04-13 15:48:25 +00:00
Central error logger.
2021-06-09 07:40:35 +00:00
### Request Methods used for reply messages
2021-04-13 15:59:30 +00:00
#### REQToConsole
Print the output of the reply message to the console.
#### REQToFileAppend
Append the output of the reply message to a log file specified with the `directory` and `fileName` fields.
2021-04-13 15:59:30 +00:00
#### REQToFile
Write the output of the reply message to a log file specified with the `directory` and `fileName` fields.
2021-04-13 15:59:30 +00:00
### Errors reporting
- Report errors happening on some node in to central error handler.
### Prometheus metrics
2021-03-11 16:14:43 +00:00
- Prometheus exporters for Metrics
2021-02-10 22:59:47 +00:00
### Other
2021-03-11 16:14:43 +00:00
- More will come. In active development.
2021-02-10 22:59:47 +00:00
2021-02-10 18:02:57 +00:00
## Howto
### Build and Run
2021-09-09 07:54:33 +00:00
Steward is written in go, so you need Go installed to compile it. You can get go at <https://golang.org/dl/>.
clone the repository `git clone https://github.com/RaaLabs/steward.git`.
Then cd `./steward/cmd` and do `go build -o steward`._1FJeI4_rrR6mY_Y496rsi5
2021-09-09 07:57:41 +00:00
Run the application with `env CONFIG_FOLDER </myconfig/folder/here> ./steward`
2021-09-09 07:54:33 +00:00
You can get all the options with `./steward --help`
2021-02-10 18:02:57 +00:00
### Options for running
The location of the config file are given via an env variable at startup (default "./etc/).
`env CONFIG_FOLDER </myconfig/folder/here>`
2021-02-25 12:42:34 +00:00
```text
2021-09-09 07:54:33 +00:00
// The configuration folder on disk
ConfigFolder string
// The folder where the socket file should live
SocketFolder string
// TCP Listener for sending messages to the system
TCPListener string
// The folder where the database should live
DatabaseFolder string
// some unique string to identify this Edge unit
NodeName string
// the address of the message broker
BrokerAddress string
// nats connect retry
NatsConnectRetryInterval int
// The number of the profiling port
ProfilingPort string
// host and port for prometheus listener, e.g. localhost:2112
PromHostAndPort string
// set to true if this is the node that should receive the error log's from other nodes
DefaultMessageTimeout int
// default amount of retries that will be done before a message is thrown away, and out of the system
DefaultMessageRetries int
// Publisher data folder
SubscribersDataFolder string
// central node to receive messages published from nodes
CentralNodeName string
// Path to the certificate of the root CA
RootCAPath string
// Full path to the NKEY's seed file
NkeySeedFile string
// The host and port to expose the data folder
ExposeDataFolder string
// Timeout for error messages
ErrorMessageTimeout int
// Retries for error messages.
ErrorMessageRetries int
// Make the current node send hello messages to central at given interval in seconds
StartPubREQHello int
// Start the central error logger.
// Takes a comma separated string of nodes to receive from or "*" for all nodes.
StartSubREQErrorLog bool
// Subscriber for hello messages
StartSubREQHello bool
// Subscriber for text logging
StartSubREQToFileAppend bool
// Subscriber for writing to file
StartSubREQToFile bool
// Subscriber for Echo Request
StartSubREQPing bool
// Subscriber for Echo Reply
StartSubREQPong bool
// Subscriber for CLICommandRequest
StartSubREQCliCommand bool
// Subscriber for REQnCliCommand
StartSubREQnCliCommand bool
// Subscriber for REQToConsole
StartSubREQToConsole bool
// Subscriber for REQHttpGet
StartSubREQHttpGet bool
// Subscriber for tailing log files
StartSubREQTailFile bool
// Subscriber for continously delivery of output from cli commands.
StartSubREQnCliCommandCont bool
2021-02-10 18:02:57 +00:00
```
### How to Run
#### Nkey Authentication
Nkey's can be used for authentication, and you use the `nkeySeedFile` flag to specify the seed file to use.
2021-09-09 07:54:33 +00:00
Read more in the sections below on how to generate nkey's.
2021-04-16 13:27:28 +00:00
#### nats-server (the message broker)
2021-04-06 09:02:58 +00:00
The broker for messaging is Nats-server from <https://nats.io>. Download, run it, and use the `-brokerAddress` flag on Steward to point to it.
2021-03-12 05:01:26 +00:00
2021-04-16 13:27:28 +00:00
There is a lot of different variants of how you can setup and confiure Nats. Full mesh, leaf node, TLS, Authentication, and more. You can read more about how to configure the Nats broker called nats-server at <https://nats.io/>.
##### Server config with nkey authentication
```config
port: 4222
tls {
cert_file: "/Users/bt/tmp/autocert/ww.steward.raalabs.tech/ww.steward.raalabs.tech.crt"
key_file: "/Users/bt/tmp/autocert/ww.steward.raalabs.tech/ww.steward.raalabs.tech.key"
}
authorization: {
users = [
{
# central
nkey: <USER_NKEY_HERE>
permissions: {
publish: {
allow: ["ww.>","errorCentral.>"]
}
subscribe: {
allow: ["ww.>","errorCentral.>"]
}
}
}
{
# mixer
nkey: <USER_NKEY_HERE>
permissions: {
publish: {
allow: ["central.>"]
}
subscribe: {
allow: ["central.>","mixer.>"]
}
}
}
{
# node10
nkey: <USER_NKEY_HERE>
permissions: {
publish: {
allow: ["ww.central.>","errorCentral.>","ww.morningconductor.>"]
}
subscribe: {
allow: ["ww.central.>","ww.morningconductor.>"]
}
}
}
]
}
```
The official docs for nkeys can be found here <https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth>.
Generate private (seed) and public (user) key pair:
`nk -gen user -pubout`
Generate a public (user) key from a private (seed) key file called `seed.txt`.
`nk -inkey seed.txt -pubout > user.txt`
More example configurations for the nats-server are located in the `doc` folder in this repository.
2021-04-16 13:27:28 +00:00
#### Steward
To set the location of the config folder other than default, you should use the ENV variable `CONFIG_FOLDER`.
2021-09-09 07:57:41 +00:00
`env CONFIG_FOLDER=./etc/ ./steward --nodeName="central" --centralNodeName="central"`
2021-09-09 07:54:33 +00:00
Or using default config folder location on some central server which will act as your command and control server.
2021-02-10 18:02:57 +00:00
2021-09-09 07:54:33 +00:00
`./steward --nodeName="central" --centralNodeName="central"`
2021-02-10 18:02:57 +00:00
One the nodes out there
2021-09-09 07:54:33 +00:00
`./steward ./steward --nodeName="ship1" --centralNodeName="central"` & `./steward --node="ship2"` and so on.
2021-02-10 18:02:57 +00:00
2021-04-06 09:02:58 +00:00
Use the `-help` flag to get all possibilities.
2021-04-16 13:27:28 +00:00
A complete example to start a central node called `central`.
`./steward --nodeName="central" --defaultMessageRetries=3 --defaultMessageTimeout=5 -subscribersDataFolder="./data" --centralNodeName="central" -startSubREQErrorLog="*" -subscribersDataFolder="./var" -brokerAddress="127.0.0.1:4222"`
And start another node that will be managed via central.
`./steward --nodeName="ship1" -startPubREQHello=200 --centralNodeName="central" -promHostAndPort=":12112" -brokerAddress="127.0.0.1:4222"`
2021-03-11 16:14:43 +00:00
### Message fields explanation
```go
// The node to send the message to
toNode
2021-09-09 07:54:33 +00:00
// ToNodes to specify several hosts to send message to in the
// form of an slice/array.
toNodes
// The actual data in the message. This is typically where we
// specify the cli commands to execute on a node, and this is
// also the field where we put the returned data in a reply
// message.
2021-03-11 16:14:43 +00:00
data
2021-09-09 07:54:33 +00:00
// Method, what request type to use, like REQCliCommand, REQHttpGet..
2021-03-11 16:14:43 +00:00
method
2021-04-13 15:48:25 +00:00
// ReplyMethod, is the method to use for the reply message.
// By default the reply method will be set to log to file, but
// you can override it setting your own here.
replyMethod
// Initial message Reply ACK wait timeout
ACKTimeout
2021-03-11 16:14:43 +00:00
// Normal Resend retries
retries
2021-04-13 15:48:25 +00:00
// The ACK timeout of the new message created via a request event.
replyACKTimeout
2021-03-11 16:14:43 +00:00
// The retries of the new message created via a request event.
2021-04-13 12:37:17 +00:00
replyRetries
2021-04-09 17:23:08 +00:00
// Timeout for long a process should be allowed to operate
2021-04-13 15:48:25 +00:00
methodTimeout
2021-04-09 17:23:08 +00:00
// Directory is a string that can be used to create the
//directory structure when saving the result of some method.
// For example "syslog","metrics", or "metrics/mysensor"
// The type is typically used in the handler of a method.
2021-04-13 15:48:25 +00:00
directory
// FileName is used to be able to set a wanted name
2021-04-09 17:23:08 +00:00
// on a file being saved as the result of data being handled
// by a method handler.
fileName
2021-04-09 17:23:08 +00:00
// operation are used to give an opCmd and opArg's.
2021-04-13 15:48:25 +00:00
operation
2021-04-09 17:23:08 +00:00
```
2021-04-13 15:48:25 +00:00
### How to send a Message
2021-09-09 07:54:33 +00:00
The API for sending a message from one node to another node is by pasting a structured JSON object into the socket file file called `steward.sock` which by default lives in the `./tmp` directory, or by starting the TCPListener, or via http (not implented yet). The message will be picked up, umarshaled, and if OK it will be sent as a message to the node specified in the `toNode` field.
2021-04-13 15:48:25 +00:00
The `method` is what defines what the event will do.
2021-04-09 17:23:08 +00:00
The `Operation` field is a little bit special. This field is used with the `REQOpCommand` to specify what operation command to run, and also it's arguments.
2021-09-09 07:54:33 +00:00
#### Send to socket with netcat
`nc -U ./tmp/steward.sock < myMessage.json`
2021-04-13 15:48:25 +00:00
#### The current `operation`'s that are available are
2021-04-09 17:23:08 +00:00
To stop a process of a specific type on a node.
```json
...
"method":"REQOpCommand",
"operation":{
"opCmd":"stopProc",
"opArg": {
"method": "REQHttpGet",
"kind": "subscriber",
2021-06-08 12:04:29 +00:00
"receivingNode": "ship2",
"id":127
2021-04-09 17:23:08 +00:00
}
},
...
```
To get a list of all running processes on a node.
```json
...
"method":"REQOpCommand",
"operation":{
"opCmd":"ps"
},
...
```
To start a process of a specified type on a node.
```json
"method":"REQOpCommand",
"operation":{
"opCmd":"startProc",
"opArg": {
"method": "REQHttpGet",
"allowedNodes": ["central","node1"]
}
},
2021-03-11 16:14:43 +00:00
```
2021-02-10 18:02:57 +00:00
2021-06-08 18:52:45 +00:00
and another example
```json
[
{
"directory":"opcommand_logs",
"fileName": "some.log",
2021-06-08 18:52:45 +00:00
"toNode": "ship2",
"data": [],
"method":"REQOpCommand",
"operation":{
"opCmd":"startProc",
"opArg": {
"method": "REQnCliCommandCont",
"allowedNodes": ["central"]
}
},
"replyMethod":"REQToFileAppend",
"ACKTimeout":3,
"retries":3,
"replyACKTimeout":3,
"replyRetries":3,
"MethodTimeout": 7
}
]
```
2021-03-11 16:14:43 +00:00
NB: Both the keys and the values used are case sensitive.
2021-02-25 12:42:34 +00:00
2021-02-10 18:02:57 +00:00
#### Sending a command from one Node to Another Node
2021-09-09 07:54:33 +00:00
##### Example JSON for appending a message of type command into the `socket` file
2021-02-10 18:02:57 +00:00
2021-02-10 22:59:47 +00:00
```json
2021-02-10 18:02:57 +00:00
[
{
2021-04-15 05:52:42 +00:00
"directory":"/var/steward/cli-command/executed-result",
2021-09-09 07:54:33 +00:00
"fileName": "some.log",
2021-02-10 18:02:57 +00:00
"toNode": "ship1",
2021-04-06 09:02:58 +00:00
"data": ["bash","-c","sleep 3 & tree ./"],
"method":"REQCliCommand",
"ACKTimeout":10,
2021-04-06 09:02:58 +00:00
"retries":3,
"methodTimeout": 4
2021-02-10 18:02:57 +00:00
}
]
```
2021-09-09 07:54:33 +00:00
##### Specify more messages at once do
2021-02-10 18:02:57 +00:00
2021-02-10 22:59:47 +00:00
```json
2021-02-10 18:02:57 +00:00
[
{
"directory":"cli-command-executed-result",
2021-09-09 07:54:33 +00:00
"fileName": "some.log",
2021-02-10 18:02:57 +00:00
"toNode": "ship1",
2021-04-06 09:02:58 +00:00
"data": ["bash","-c","sleep 3 & tree ./"],
"method":"REQCliCommand",
"ACKTimeout":10,
2021-04-06 09:02:58 +00:00
"retries":3,
"methodTimeout": 4
2021-02-10 18:02:57 +00:00
},
{
"directory":"cli-command-executed-result",
2021-09-09 07:54:33 +00:00
"fileName": "some.log",
2021-02-10 18:02:57 +00:00
"toNode": "ship2",
2021-04-06 09:02:58 +00:00
"data": ["bash","-c","sleep 3 & tree ./"],
"method":"REQCliCommand",
"ACKTimeout":10,
2021-04-06 09:02:58 +00:00
"retries":3,
"methodTimeout": 4
2021-02-10 18:02:57 +00:00
}
]
```
2021-09-09 07:54:33 +00:00
##### Send the same message to several hosts by using the toHosts field
```json
[
{
"directory": "httpget",
"fileName": "finn.no.html",
"toNode": "",
"toNodes": ["central","ship2"],
"data": ["https://finn.no"],
"method":"REQHttpGet",
"replyMethod":"REQToFile",
"ACKTimeout":5,
"retries":3,
"methodTimeout": 5
}
]
```
##### Send an Op Command message for process listing with custom timeout and amount of retries
2021-02-25 12:42:34 +00:00
```json
[
{
"directory":"opcommand_logs",
"fileName": "some.log",
2021-04-09 17:23:08 +00:00
"toNode": "ship2",
"data": [],
"method":"REQOpCommand",
"operation":{
"opCmd":"ps"
},
"ACKTimeout":3,
2021-04-09 17:23:08 +00:00
"retries":3,
"replyACKTimeout":3,
2021-04-13 12:37:17 +00:00
"replyRetries":3,
2021-09-09 07:54:33 +00:00
"methodTimeout": 7
2021-04-09 17:23:08 +00:00
}
]
```
2021-09-09 07:54:33 +00:00
##### Send and Op Command to stop a subscriber on a node
2021-04-09 17:23:08 +00:00
```json
[
{
"directory":"opcommand_logs",
"fileName": "some.log",
2021-04-09 17:23:08 +00:00
"toNode": "ship2",
"data": [],
"method":"REQOpCommand",
"operation":{
"opCmd":"stopProc",
"opArg": {
"method": "REQHttpGet",
"kind": "subscriber",
"receivingNode": "ship2"
}
},
"ACKTimeout":3,
2021-04-09 17:23:08 +00:00
"retries":3,
"replyACKTimeout":3,
2021-04-13 12:37:17 +00:00
"replyRetries":3,
2021-09-09 07:54:33 +00:00
"methodTimeout": 7
2021-04-09 17:23:08 +00:00
}
]
```
2021-09-09 07:54:33 +00:00
##### Send and Op Command to start a subscriber on a node
2021-04-09 17:23:08 +00:00
```json
[
{
"directory":"opcommand_logs",
"fileName": "some.log",
2021-04-09 17:23:08 +00:00
"toNode": "ship2",
"data": [],
2021-04-06 09:02:58 +00:00
"method":"REQOpCommand",
2021-04-09 17:23:08 +00:00
"operation":{
"opCmd":"startProc",
"opArg": {
"method": "REQHttpGet",
"allowedNodes": ["central","node1"]
}
},
"ACKTimeout":3,
2021-04-06 09:02:58 +00:00
"retries":3,
"replyACKTimeout":3,
2021-04-13 12:37:17 +00:00
"replyRetries":3,
2021-09-09 07:54:33 +00:00
"methodTimeout": 7
2021-02-10 18:02:57 +00:00
}
]
```
2021-04-16 13:27:28 +00:00
Tail a log file on a node, and save the result of the tail centrally at the directory specified.
```json
[
{
"directory": "./my/log/files/",
"fileName": "some.log",
2021-04-16 13:27:28 +00:00
"toNode": "ship2",
"data": ["./test.log"],
"method":"REQTailFile",
"ACKTimeout":5,
"retries":3,
"methodTimeout": 200
}
]
```
2021-04-06 09:02:58 +00:00
You can save the content to myfile.JSON and append it to the `socket` file.
2021-02-10 18:02:57 +00:00
2021-04-06 09:02:58 +00:00
`nc -U ./steward.sock < example/toShip1-REQCliCommand.json`
2021-02-19 17:00:47 +00:00
2021-03-11 16:14:43 +00:00
## Concepts/Ideas
### Naming
#### Subject
2021-04-06 09:02:58 +00:00
`<nodename>.<method>.<command/event>`
2021-03-11 16:14:43 +00:00
Nodename: Are the hostname of the device. This do not have to be resolvable via DNS, it is just a unique name for the host to receive the message.
Command/Event: Are type of message sent. `CommandACK`/`EventACK`/`CommandNACK`/`EventNACK`. Description of the differences are mentioned earlier.\
Info: The command/event which is called a MessageType are present in both the Subject structure and the Message structure. The reason for this is that it is used both in the naming of a subject, and in the message for knowing what kind of message it is and how to handle it.
Method: Are the functionality the message provide. Example could be `CLICommand` or `Syslogforwarding`
##### Complete subject example
2021-04-06 09:02:58 +00:00
For Hello Message to a node named "central" of type Event and there is No Ack.
2021-03-11 16:14:43 +00:00
2021-04-06 09:02:58 +00:00
`central.REQHello.EventNACK`
2021-03-11 16:14:43 +00:00
2021-04-06 09:02:58 +00:00
For CliCommand message to a node named "ship1" of type Command and it wants an Ack.
2021-03-11 16:14:43 +00:00
2021-04-06 09:02:58 +00:00
`ship1.REQCliCommand.CommandACK`
2021-03-11 16:14:43 +00:00
## TODO
2021-09-16 03:31:04 +00:00
### Implements a replyOpts message field
To be used for specifying options that can be needed for reply methods like external calls, etc.
### Implements a repeat and repeatInterval message field
To specify how many times a message should be run and at what interval
### Split out the different opCmd's into it's own request types
2021-09-13 11:24:28 +00:00
### Move the processes map into it's own go routine
Use channels to update and get values, and get rid of the use of mutex's.
### Services at startup of Steward. Could be implemented by having a local folder of messages to go through at startup. What is needed
2021-08-30 04:44:57 +00:00
- A Handler that writes to this folder.
- A Handler that can check what is in this folder.
- A Handler to remove messages from this folder.
### A carrier type of message, that are able to forward a message on behalf of others, and are not directly ment for itself
2021-08-30 04:44:57 +00:00
2021-06-08 02:02:23 +00:00
### Workflow request type
Implement a workflow request type that will allow steps to be executed, and rollback if something failed during execution.
2021-06-14 04:49:47 +00:00
### Add Op option the remove messages from the queue on nodes
If messages have been sent, and not picked up by a node it might make sense to have some method to clear messages on a node. This could either be done by message ID, and/or time duration.
2021-03-26 09:48:05 +00:00
## Disclaimer
2021-03-11 16:14:43 +00:00
2021-04-16 13:27:28 +00:00
All code in this repository are to be concidered not-production-ready, and the use is at your own responsibility. The code are the attempt to concretize the idea of a purely async management system where the controlling unit is decoupled from the receiving unit, and that that we know the state of all the receiving units at all times.
2021-03-11 16:14:43 +00:00
2021-03-26 09:48:05 +00:00
Also read the license file for further details.