README.md 5.44 KB
Newer Older
Bram Daams's avatar
Bram Daams committed
1
2
# SmartCronHelper
A cron shell wrapper for registering and updating cron jobs automatically in
Bram Daams's avatar
Bram Daams committed
3
[healthchecks](https://healthchecks.io) or your [own hosted copy of Healthchecks](https://github.com/healthchecks/healthchecks).
Bram Daams's avatar
Bram Daams committed
4

Bram Daams's avatar
Bram Daams committed
5
> warning: this software package should be considered "alpha"
Bram Daams's avatar
Bram Daams committed
6

Bram Daams's avatar
Bram Daams committed
7
8

## Installation
Bram Daams's avatar
Bram Daams committed
9
Install sch system wide with pip
Bram Daams's avatar
Bram Daams committed
10
``` console
Bram Daams's avatar
Bram Daams committed
11
$ sudo pip3 install sch
Bram Daams's avatar
Bram Daams committed
12
13
```

Bram Daams's avatar
Bram Daams committed
14
A `sch` cli should now be availble:
Bram Daams's avatar
Bram Daams committed
15
``` console
Bram Daams's avatar
Bram Daams committed
16
17
$ which sch
/usr/local/bin/sch
Bram Daams's avatar
Bram Daams committed
18
```
Bram Daams's avatar
Bram Daams committed
19
20
21
22
23
24
25
26
27
28
29
30
31
32

`sch --version` should return something like:
``` console
sch, version 0.2.1
```

## Configuration
Create a configuration file `/etc/sch.conf` that looks like:
``` ini
[hc]
healthchecks_api_url = https://hc.example.com/api/v1/
healthchecks_api_key = xxmysecretkeyxx
```

Bram Daams's avatar
Bram Daams committed
33
And fill in the API URL and the key obtained from the Healthchecks project
Bram Daams's avatar
Bram Daams committed
34
settings block labeled "API Access".
Bram Daams's avatar
Bram Daams committed
35

Bram Daams's avatar
Bram Daams committed
36
## Monitoring cron jobs
Bram Daams's avatar
Bram Daams committed
37
Just decorate your existing cron tabs by specifying the alternative `sch`:
Bram Daams's avatar
Bram Daams committed
38
39
40
41
```
SHELL=/usr/local/bin/sch
```
This line should be above the cron lines you want to have monitored by Healthchecks.
Bram Daams's avatar
Bram Daams committed
42

Bram Daams's avatar
Bram Daams committed
43
44
45
46
47
Only jobs with the environment variable `JOB_ID`, ie:
```
*/5 * * * * root JOB_ID=some_id /path/to/some_command
```
The value of `JOB_ID` should be unique for the host.
48

Bram Daams's avatar
Bram Daams committed
49
50
The combination of the `JOB_ID` environment variable and the `sch` shell is enough
to have the job checked in Healthchecks.
Bram Daams's avatar
Bram Daams committed
51

Bram Daams's avatar
Bram Daams committed
52
53
54
55
56
57
At each run of the job, `sch` will take care that the schedule, description and 
other metadata is synchronized whenever there's a change in the cron job. Just
makes sure to not change the `JOB_ID` (or it will create a new check).
 
### Other meta data
The following data is used to configure a corresponding Healthchecks check:
Bram Daams's avatar
Bram Daams committed
58
- `JOB_ID`: the environment variable is used for the name of the check and a tag named `job_id={value of JOB_ID}`
Bram Daams's avatar
Bram Daams committed
59
- the cron lines' **comment** is used for the description of the check. The comment line just above a cron line or the inline comment is used
Bram Daams's avatar
Bram Daams committed
60
- `JOB_TAGS`: use this environment variable in a job to specify tag names separated by a comma to specify additional tags
Bram Daams's avatar
Bram Daams committed
61
- `$USER`: the current user running the cron command is used to create a tag named `user=$USER`
Bram Daams's avatar
Bram Daams committed
62
- the jobs **schedule** and the hosts **timezone** is used to set the checks schedule
63
- `JOB_GRACE`: the value of this environment variable is used to set the grace time in seconds for the check. See JOB_GRACE for valid interval formats.
Bram Daams's avatar
Bram Daams committed
64
- when registering a new check and JOB_GRACE is not set, the **execution time** of the command is used to set an initial grace time. The grace time will be set to 1.2 times the execution time + 30 seconds. As per the Healthchecks API, the minimal grace time is 1 minute and the maximum grace time is 30 days.
Bram Daams's avatar
Bram Daams committed
65

Bram Daams's avatar
Bram Daams committed
66
67
68
69
70
71
An example of a cron file that touches most of the functionality would look like:
```
SHELL=/usr/local/bin/sch
# if this check fails, the host is probably offline
* * * * * root JOB_ID=true /bin/true
```
Bram Daams's avatar
Bram Daams committed
72
Although above cron job is useful, a more realistic could look like:
Bram Daams's avatar
Bram Daams committed
73
74
75
```
SHELL=/usr/loca/bin/sch
# super important backup, if this one fails: fix with top priority!
Bram Daams's avatar
Bram Daams committed
76
10 8-20/2 * mon-fri  backup  JOB_ID=db-backups JOB_TAGS=db,backup,my_project JOB_GRACE=5m /usr/local/bin/run-db-backups
Bram Daams's avatar
Bram Daams committed
77
```
Bram Daams's avatar
Bram Daams committed
78

79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
#### JOB_GRACE interval format
If no suffixes are used, seconds are assumed.
You can make use of the following suffixes to specify an interval:

| Suffix | Interval |
|--------|----------|
| s      | seconds  |
| m      | minutes  |
| h      | hours    |
| D      | days     |
| W      | weeks    |
| M      | months   |
| Y      | years    |

Although days and weeks are accepted, you might want to limit the interval to several hours ;-)

Bram Daams's avatar
Bram Daams committed
95
Examples:
96

Bram Daams's avatar
Bram Daams committed
97
98
99
100
101
102
| environment variable | grace time |
|----------------------|------------|
| `JOB_GRACE=5m`       | 300s       |
| `JOB_GRACE=120`      | 120s       |
| `JOB_GRACE=1h30m`    | 5400s      |

103

Bram Daams's avatar
Bram Daams committed
104
105
106
107
108
109
110
### Job execution
`sch` takes over the role of the shell. Jobs not containing the `JOB_ID` environment variable are directly executed with `os.system`.
For `sch` managed jobs:
- `sch` will start with pinging `/start` endpoint of the check
- os.sytem executes the command
- depending on the exit code, it will ping for success or ping the `/fail` end point on failure

Bram Daams's avatar
Bram Daams committed
111
112
113
114
115
116
## Development environment
### Setup environment
``` console
$ python -m venv venv
$ . venv/bin/activate
$ pip install --editable .
Bram Daams's avatar
Bram Daams committed
117
```
Bram Daams's avatar
Bram Daams committed
118

Bram Daams's avatar
Bram Daams committed
119
### Testing
Bram Daams's avatar
Bram Daams committed
120
Create a file named `sch.conf` and edit the Healthchecks API URL and key:
Bram Daams's avatar
Bram Daams committed
121
122
123
``` console
cp sch.conf.example sch.conf
```
Bram Daams's avatar
Bram Daams committed
124
The configuration file looks like:
Bram Daams's avatar
Bram Daams committed
125
126
127
128
129
130
131
132
133
134
135
``` console
 $ cat sch.conf.example 
[hc]
healthchecks_api_url = https://hc.example.com/api/v1/
healthchecks_api_key = xxmysecretkeyxx
```
Create a test cron job:
``` console
$ sudo cp doc/testcrontab /etc/cron.d/test
$ ./testshell.sh
```
Bram Daams's avatar
Bram Daams committed
136

Bram Daams's avatar
Bram Daams committed
137
138
### Syntax check
Style Guide Enforcement:
Bram Daams's avatar
Bram Daams committed
139
140
141
142
143
``` console
$ pip install flake8
$ flake8 *py
```

Bram Daams's avatar
Bram Daams committed
144
### References
Bram Daams's avatar
Bram Daams committed
145
* python-crontab <https://pypi.org/project/python-crontab/>
Bram Daams's avatar
Bram Daams committed
146
* crab <https://github.com/grahambell/crab>
Bram Daams's avatar
Bram Daams committed
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169

## Notes
### fully qualified domain name
`sch` uses the FQDN to identify the hosts it's running on. You can check the FQDN with:
``` console
$ hostname --fqdn
host.example.com
```

However, on some systems that don't know the domain part, it just returns the
(short) hostname instead:
``` console
$ hostname --fqdn
host
```

If this is the case, you can fix that by editing the `/etc/hosts` file so look
like this:
```
127.0.0.1	localhost
127.0.1.1	host.example.com host
```

Bram Daams's avatar
comma    
Bram Daams committed
170
Afterwards, `hostname --fqdn` should return the FQDN. Beware that `sch` will
Bram Daams's avatar
Bram Daams committed
171
create new checks when the FQDN changes.