README.md 6.42 KB
Newer Older
Bram Daams's avatar
Bram Daams committed
1
2
[![pypy badge](https://img.shields.io/pypi/v/sch.svg)](https://pypi.python.org/pypi/)

Bram Daams's avatar
Bram Daams committed
3
# SmartCronHelper
Bram Daams's avatar
Bram Daams committed
4
5
![sch logo](https://gitlab.science.ru.nl/uploads/-/system/project/avatar/3732/sch3.png)

Bram Daams's avatar
Bram Daams committed
6
A cron shell wrapper for registering and updating cron jobs automatically in
Bram Daams's avatar
Bram Daams committed
7
[healthchecks](https://healthchecks.io) or your [own hosted copy of Healthchecks](https://github.com/healthchecks/healthchecks).
Bram Daams's avatar
Bram Daams committed
8

Bram Daams's avatar
Bram Daams committed
9
> warning: this software package should be considered "alpha"
Bram Daams's avatar
Bram Daams committed
10

Bram Daams's avatar
Bram Daams committed
11
12

## Installation
Bram Daams's avatar
Bram Daams committed
13
Install sch system wide with pip
Bram Daams's avatar
Bram Daams committed
14
``` console
Bram Daams's avatar
Bram Daams committed
15
$ sudo pip3 install sch
Bram Daams's avatar
Bram Daams committed
16
17
```

Bram Daams's avatar
Bram Daams committed
18
A `sch` cli should now be availble:
Bram Daams's avatar
Bram Daams committed
19
``` console
Bram Daams's avatar
Bram Daams committed
20
21
$ which sch
/usr/local/bin/sch
Bram Daams's avatar
Bram Daams committed
22
```
Bram Daams's avatar
Bram Daams committed
23
24
25

`sch --version` should return something like:
``` console
26
sch, version 0.6.0
Bram Daams's avatar
Bram Daams committed
27
```
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
## Command line usage
See the `--help` option for usage:
``` console
Usage: sch [OPTIONS] COMMAND [ARGS]...

  sch - A cron shell wrapper for registering and updating cron jobs
  automatically in Healthchecks. The Healthchecks project api_url and
  api_key should be configured in /etc/sch.conf.

Options:
  --version                 Show the version and exit.
  -c, --shell_command TEXT  Command to execute. This how Cron executes 'sch'
                            when it is set as SHELL.
  --help                    Show this message and exit.

Commands:
  list  List checks for the configured Healthchecks project.
```

### `list` command
``` console
Usage: sch list [OPTIONS]

  List checks for the configured Healthchecks project.

Options:
  -l, --localhost / -a, --all     List checks that originate from this host
                                  (default) or list all checks.
  -s, --status [up|down|grace|started|pause|new]
                                  Show only checks that have the specified
                                  status.
  --help                          Show this message and exit.
```
Bram Daams's avatar
Bram Daams committed
61

Bram Daams's avatar
Bram Daams committed
62
63
64
65
66
67
68
69
70
71
Example output
``` console
$ sch list 
Status  Last ping       Name                                    
------- --------------- ----------------------------------------
up      2 minutes ago   disk-check                              
up      4 hours ago     restic                                  
up      5 days ago      restic_check  
```

Bram Daams's avatar
Bram Daams committed
72
73
74
75
76
77
78
79
## Configuration
Create a configuration file `/etc/sch.conf` that looks like:
``` ini
[hc]
healthchecks_api_url = https://hc.example.com/api/v1/
healthchecks_api_key = xxmysecretkeyxx
```

Bram Daams's avatar
Bram Daams committed
80
And fill in the API URL and the key obtained from the Healthchecks project
Bram Daams's avatar
Bram Daams committed
81
settings block labeled "API Access".
Bram Daams's avatar
Bram Daams committed
82

Bram Daams's avatar
Bram Daams committed
83
## Monitoring cron jobs
Bram Daams's avatar
Bram Daams committed
84
Just decorate your existing cron tabs by specifying the alternative `sch`:
Bram Daams's avatar
Bram Daams committed
85
86
87
88
```
SHELL=/usr/local/bin/sch
```
This line should be above the cron lines you want to have monitored by Healthchecks.
Bram Daams's avatar
Bram Daams committed
89

Bram Daams's avatar
Bram Daams committed
90
91
92
93
94
Only jobs with the environment variable `JOB_ID`, ie:
```
*/5 * * * * root JOB_ID=some_id /path/to/some_command
```
The value of `JOB_ID` should be unique for the host.
95

Bram Daams's avatar
Bram Daams committed
96
97
The combination of the `JOB_ID` environment variable and the `sch` shell is enough
to have the job checked in Healthchecks.
Bram Daams's avatar
Bram Daams committed
98

Bram Daams's avatar
Bram Daams committed
99
100
101
102
103
104
At each run of the job, `sch` will take care that the schedule, description and 
other metadata is synchronized whenever there's a change in the cron job. Just
makes sure to not change the `JOB_ID` (or it will create a new check).
 
### Other meta data
The following data is used to configure a corresponding Healthchecks check:
Bram Daams's avatar
Bram Daams committed
105
- `JOB_ID`: the environment variable is used for the name of the check and a tag named `job_id={value of JOB_ID}`
Bram Daams's avatar
Bram Daams committed
106
- the cron lines' **comment** is used for the description of the check. The comment line just above a cron line or the inline comment is used
Bram Daams's avatar
Bram Daams committed
107
- `JOB_TAGS`: use this environment variable in a job to specify tag names separated by a comma to specify additional tags
Bram Daams's avatar
Bram Daams committed
108
- `$USER`: the current user running the cron command is used to create a tag named `user=$USER`
Bram Daams's avatar
Bram Daams committed
109
- the jobs **schedule** and the hosts **timezone** is used to set the checks schedule
110
- `JOB_GRACE`: the value of this environment variable is used to set the grace time in seconds for the check. See JOB_GRACE for valid interval formats.
Bram Daams's avatar
Bram Daams committed
111
- when registering a new check and JOB_GRACE is not set, the **execution time** of the command is used to set an initial grace time. The grace time will be set to 1.2 times the execution time + 30 seconds. As per the Healthchecks API, the minimal grace time is 1 minute and the maximum grace time is 30 days.
Bram Daams's avatar
Bram Daams committed
112

Bram Daams's avatar
Bram Daams committed
113
114
115
116
117
118
An example of a cron file that touches most of the functionality would look like:
```
SHELL=/usr/local/bin/sch
# if this check fails, the host is probably offline
* * * * * root JOB_ID=true /bin/true
```
Bram Daams's avatar
Bram Daams committed
119
Although above cron job is useful, a more realistic could look like:
Bram Daams's avatar
Bram Daams committed
120
121
122
```
SHELL=/usr/loca/bin/sch
# super important backup, if this one fails: fix with top priority!
Bram Daams's avatar
Bram Daams committed
123
10 8-20/2 * mon-fri  backup  JOB_ID=db-backups JOB_TAGS=db,backup,my_project JOB_GRACE=5m /usr/local/bin/run-db-backups
Bram Daams's avatar
Bram Daams committed
124
```
Bram Daams's avatar
Bram Daams committed
125

126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
#### JOB_GRACE interval format
If no suffixes are used, seconds are assumed.
You can make use of the following suffixes to specify an interval:

| Suffix | Interval |
|--------|----------|
| s      | seconds  |
| m      | minutes  |
| h      | hours    |
| D      | days     |
| W      | weeks    |
| M      | months   |
| Y      | years    |

Although days and weeks are accepted, you might want to limit the interval to several hours ;-)

Bram Daams's avatar
Bram Daams committed
142
Examples:
143

Bram Daams's avatar
Bram Daams committed
144
145
146
147
148
149
| environment variable | grace time |
|----------------------|------------|
| `JOB_GRACE=5m`       | 300s       |
| `JOB_GRACE=120`      | 120s       |
| `JOB_GRACE=1h30m`    | 5400s      |

150

Bram Daams's avatar
Bram Daams committed
151
152
153
154
155
156
157
### Job execution
`sch` takes over the role of the shell. Jobs not containing the `JOB_ID` environment variable are directly executed with `os.system`.
For `sch` managed jobs:
- `sch` will start with pinging `/start` endpoint of the check
- os.sytem executes the command
- depending on the exit code, it will ping for success or ping the `/fail` end point on failure

Bram Daams's avatar
Bram Daams committed
158
### References
Bram Daams's avatar
Bram Daams committed
159
* python-crontab <https://pypi.org/project/python-crontab/>
Bram Daams's avatar
Bram Daams committed
160
* crab <https://github.com/grahambell/crab>
Bram Daams's avatar
Bram Daams committed
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183

## Notes
### fully qualified domain name
`sch` uses the FQDN to identify the hosts it's running on. You can check the FQDN with:
``` console
$ hostname --fqdn
host.example.com
```

However, on some systems that don't know the domain part, it just returns the
(short) hostname instead:
``` console
$ hostname --fqdn
host
```

If this is the case, you can fix that by editing the `/etc/hosts` file so look
like this:
```
127.0.0.1	localhost
127.0.1.1	host.example.com host
```

Bram Daams's avatar
comma    
Bram Daams committed
184
Afterwards, `hostname --fqdn` should return the FQDN. Beware that `sch` will
Bram Daams's avatar
Bram Daams committed
185
create new checks when the FQDN changes.