README.md 7.52 KB
Newer Older
Bram Daams's avatar
Bram Daams committed
1
2
[![pypy badge](https://img.shields.io/pypi/v/sch.svg)](https://pypi.python.org/pypi/)

Bram Daams's avatar
Bram Daams committed
3
# SmartCronHelper
Bram Daams's avatar
Bram Daams committed
4
5
![sch logo](https://gitlab.science.ru.nl/uploads/-/system/project/avatar/3732/sch3.png)

Bram Daams's avatar
Bram Daams committed
6
A cron shell wrapper for registering and updating cron jobs automatically in
Bram Daams's avatar
Bram Daams committed
7
[healthchecks](https://healthchecks.io) or your [own hosted copy of Healthchecks](https://github.com/healthchecks/healthchecks).
Bram Daams's avatar
Bram Daams committed
8

Bram Daams's avatar
Bram Daams committed
9
> WARNING: once setup and configured, the code in this package runs as user specified in the cron jobs and is wrapped around the cron job commands. Errors in this package could prevent your cron jobs from being executed.
Bram Daams's avatar
Bram Daams committed
10

Bram Daams's avatar
Bram Daams committed
11
12

## Installation
Bram Daams's avatar
Bram Daams committed
13
Install sch system wide with pip
Bram Daams's avatar
Bram Daams committed
14
``` console
Bram Daams's avatar
Bram Daams committed
15
$ sudo pip3 install sch
Bram Daams's avatar
Bram Daams committed
16
17
```

Bram Daams's avatar
Bram Daams committed
18
A `sch` cli should now be availble:
Bram Daams's avatar
Bram Daams committed
19
``` console
Bram Daams's avatar
Bram Daams committed
20
21
$ which sch
/usr/local/bin/sch
Bram Daams's avatar
Bram Daams committed
22
```
Bram Daams's avatar
Bram Daams committed
23
24
25

`sch --version` should return something like:
``` console
Bram Daams's avatar
Bram Daams committed
26
sch, version 0.7.1
Bram Daams's avatar
Bram Daams committed
27
```
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
## Command line usage
See the `--help` option for usage:
``` console
Usage: sch [OPTIONS] COMMAND [ARGS]...

  sch - A cron shell wrapper for registering and updating cron jobs
  automatically in Healthchecks. The Healthchecks project api_url and
  api_key should be configured in /etc/sch.conf.

Options:
  --version                 Show the version and exit.
  -c, --shell_command TEXT  Command to execute. This how Cron executes 'sch'
                            when it is set as SHELL.
  --help                    Show this message and exit.

Commands:
  list  List checks for the configured Healthchecks project.
```

### `list` command
``` console
Usage: sch list [OPTIONS]

  List checks for the configured Healthchecks project.

Options:
  -l, --localhost / -a, --all     List checks that originate from this host
                                  (default) or list all checks.
  -s, --status [up|down|grace|started|pause|new]
                                  Show only checks that have the specified
                                  status.
  --help                          Show this message and exit.
```
Bram Daams's avatar
Bram Daams committed
61

Bram Daams's avatar
Bram Daams committed
62
63
Example output
``` console
Bram Daams's avatar
Bram Daams committed
64
$ sch list
Bram Daams's avatar
Bram Daams committed
65
66
67
68
69
70
71
Status  Last ping       Name                                    
------- --------------- ----------------------------------------
up      2 minutes ago   disk-check                              
up      4 hours ago     restic                                  
up      5 days ago      restic_check  
```

Bram Daams's avatar
Bram Daams committed
72
73
74
75
76
77
78
## Configuration
Create a configuration file `/etc/sch.conf` that looks like:
``` ini
[hc]
healthchecks_api_url = https://hc.example.com/api/v1/
healthchecks_api_key = xxmysecretkeyxx
```
Bram Daams's avatar
Bram Daams committed
79
And fill in the API URL and the key obtained from the Healthchecks project
Bram Daams's avatar
Bram Daams committed
80
settings block labeled "API Access".
Bram Daams's avatar
Bram Daams committed
81

Bram Daams's avatar
Bram Daams committed
82
83
84
85
86
87
88
Optionally, specify the log level in the configuration file:
``` ini
[sch]
loglevel = DEBUG
```
Possible values for loglevel are explained [here](https://docs.python.org/3/library/logging.html#levels). The default log level is `ERROR`.

Bram Daams's avatar
Bram Daams committed
89
## Monitoring cron jobs
Bram Daams's avatar
Bram Daams committed
90
Just decorate your existing cron tabs by specifying the alternative `sch`:
Bram Daams's avatar
Bram Daams committed
91
92
93
94
```
SHELL=/usr/local/bin/sch
```
This line should be above the cron lines you want to have monitored by Healthchecks.
Bram Daams's avatar
Bram Daams committed
95

Bram Daams's avatar
Bram Daams committed
96
97
98
99
100
Only jobs with the environment variable `JOB_ID`, ie:
```
*/5 * * * * root JOB_ID=some_id /path/to/some_command
```
The value of `JOB_ID` should be unique for the host.
101

Bram Daams's avatar
Bram Daams committed
102
103
The combination of the `JOB_ID` environment variable and the `sch` shell is enough
to have the job checked in Healthchecks.
Bram Daams's avatar
Bram Daams committed
104

Bram Daams's avatar
Bram Daams committed
105
At each run of the job, `sch` will take care that the schedule, description and
Bram Daams's avatar
Bram Daams committed
106
107
other metadata is synchronized whenever there's a change in the cron job. Just
makes sure to not change the `JOB_ID` (or it will create a new check).
Bram Daams's avatar
Bram Daams committed
108
109
110
111
112
113
114

### Per job configurable options
Just like the `JOB_ID` environment described in the previous paragraph. There are
other job specific environment variables that can be used to configure the behavior
of the cron job or the associated Healthchecks check. These are described in the
table below:

Bram Daams's avatar
Bram Daams committed
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
| Environment variable | Example value | Description | Associated Healthchecks check setting |
| :--------------------|:--------------| ------------|---------------------------------------|
| `JOB_ID*`            | `backup`      | Required for `sch` to interact with the Healthchecks API | check name, tags |
| `JOB_TAGS`           | `foo,bar`     | Specify tag names separated by a comma | tags |
| `JOB_GRACE`          | `5m`          | Grace time specified in seconds or use the time interval format described below. The grace time will be set to 1.2 times the execution time + `JOB_RNDWAIT` + 30 seconds. As per the Healthchecks API, the minimal grace time is 1 minute and the maximum grace time is 30 days. | grace time |
| `JOB_RNDWAIT`        | `1m `         | Max. wait time in seconds or use the time interval format described below. Use this setting to introduce a random delay. `sch` will wait a random time between 0 and `JOB_RNDWAIT` before executing the job's command. | grace time |

#### Interval format
If no suffixes are used, seconds are assumed.
You can make use of the following suffixes to specify an interval:

| Suffix | Interval |
|--------|----------|
| s      | seconds  |
| m      | minutes  |
| h      | hours    |
| D      | days     |
| W      | weeks    |
| M      | months   |
| Y      | years    |

Although days and weeks are accepted, you might want to limit the interval to several minutes ;-)

Examples:

| Interval | Duration     |
|----------|-------------:|
| `5m`     |  300 seconds |
| `120`    |  120 seconds |
| `1h30m`  | 5400 seconds | 
Bram Daams's avatar
Bram Daams committed
145

Bram Daams's avatar
Bram Daams committed
146
### Other meta data
Bram Daams's avatar
Bram Daams committed
147
- the cron lines' **comment** is used for the description of the check. The comment line just above a cron line or the inline comment is used
Bram Daams's avatar
Bram Daams committed
148
- `$USER`: the current user running the cron command is used to create a tag named `user=$USER`
Bram Daams's avatar
Bram Daams committed
149

Bram Daams's avatar
Bram Daams committed
150
### Some example cron jobs
Bram Daams's avatar
Bram Daams committed
151
152
153
154
155
156
An example of a cron file that touches most of the functionality would look like:
```
SHELL=/usr/local/bin/sch
# if this check fails, the host is probably offline
* * * * * root JOB_ID=true /bin/true
```
Bram Daams's avatar
Bram Daams committed
157
158

Although above cron job is useful, a more advanced configuration could look like:
Bram Daams's avatar
Bram Daams committed
159
160
161
```
SHELL=/usr/loca/bin/sch
# super important backup, if this one fails: fix with top priority!
Bram Daams's avatar
Bram Daams committed
162
10 8-20/2 * * mon-fri  backup  JOB_ID=db-backups JOB_TAGS=db,backup,my_project JOB_RNDWAIT=2m JOB_GRACE=5m /usr/local/bin/run-db-backups
Bram Daams's avatar
Bram Daams committed
163
```
Bram Daams's avatar
Bram Daams committed
164
Resulting in the following check:
Bram Daams's avatar
Bram Daams committed
165
![screenshot of a more advanced check](https://gitlab.science.ru.nl/bram/sch/-/raw/master/doc/hc-screenshot-advanced.png)
Bram Daams's avatar
Bram Daams committed
166

Bram Daams's avatar
Bram Daams committed
167
![screenshot of a more advanced check with description](https://gitlab.science.ru.nl/bram/sch/-/raw/master/doc/hc-screenshot-advanced-description.png)
Bram Daams's avatar
Bram Daams committed
168

Bram Daams's avatar
Bram Daams committed
169
170
171
172
173
174
175
### Job execution
`sch` takes over the role of the shell. Jobs not containing the `JOB_ID` environment variable are directly executed with `os.system`.
For `sch` managed jobs:
- `sch` will start with pinging `/start` endpoint of the check
- os.sytem executes the command
- depending on the exit code, it will ping for success or ping the `/fail` end point on failure

Bram Daams's avatar
Bram Daams committed
176
### References
Bram Daams's avatar
Bram Daams committed
177
* python-crontab <https://pypi.org/project/python-crontab/>
Bram Daams's avatar
Bram Daams committed
178
* crab <https://github.com/grahambell/crab>
Bram Daams's avatar
Bram Daams committed
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201

## Notes
### fully qualified domain name
`sch` uses the FQDN to identify the hosts it's running on. You can check the FQDN with:
``` console
$ hostname --fqdn
host.example.com
```

However, on some systems that don't know the domain part, it just returns the
(short) hostname instead:
``` console
$ hostname --fqdn
host
```

If this is the case, you can fix that by editing the `/etc/hosts` file so look
like this:
```
127.0.0.1	localhost
127.0.1.1	host.example.com host
```

Bram Daams's avatar
comma    
Bram Daams committed
202
Afterwards, `hostname --fqdn` should return the FQDN. Beware that `sch` will
Bram Daams's avatar
Bram Daams committed
203
create new checks when the FQDN changes.