README.md 5.99 KB
Newer Older
Bram Daams's avatar
Bram Daams committed
1
2
[![pypy badge](https://img.shields.io/pypi/v/sch.svg)](https://pypi.python.org/pypi/)

Bram Daams's avatar
Bram Daams committed
3
4
# SmartCronHelper
A cron shell wrapper for registering and updating cron jobs automatically in
Bram Daams's avatar
Bram Daams committed
5
[healthchecks](https://healthchecks.io) or your [own hosted copy of Healthchecks](https://github.com/healthchecks/healthchecks).
Bram Daams's avatar
Bram Daams committed
6

Bram Daams's avatar
Bram Daams committed
7
> warning: this software package should be considered "alpha"
Bram Daams's avatar
Bram Daams committed
8

Bram Daams's avatar
Bram Daams committed
9
10

## Installation
Bram Daams's avatar
Bram Daams committed
11
Install sch system wide with pip
Bram Daams's avatar
Bram Daams committed
12
``` console
Bram Daams's avatar
Bram Daams committed
13
$ sudo pip3 install sch
Bram Daams's avatar
Bram Daams committed
14
15
```

Bram Daams's avatar
Bram Daams committed
16
A `sch` cli should now be availble:
Bram Daams's avatar
Bram Daams committed
17
``` console
Bram Daams's avatar
Bram Daams committed
18
19
$ which sch
/usr/local/bin/sch
Bram Daams's avatar
Bram Daams committed
20
```
Bram Daams's avatar
Bram Daams committed
21
22
23
24
25

`sch --version` should return something like:
``` console
sch, version 0.2.1
```
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
## Command line usage
See the `--help` option for usage:
``` console
Usage: sch [OPTIONS] COMMAND [ARGS]...

  sch - A cron shell wrapper for registering and updating cron jobs
  automatically in Healthchecks. The Healthchecks project api_url and
  api_key should be configured in /etc/sch.conf.

Options:
  --version                 Show the version and exit.
  -c, --shell_command TEXT  Command to execute. This how Cron executes 'sch'
                            when it is set as SHELL.
  --help                    Show this message and exit.

Commands:
  list  List checks for the configured Healthchecks project.
```

### `list` command
``` console
Usage: sch list [OPTIONS]

  List checks for the configured Healthchecks project.

Options:
  -l, --localhost / -a, --all     List checks that originate from this host
                                  (default) or list all checks.
  -s, --status [up|down|grace|started|pause|new]
                                  Show only checks that have the specified
                                  status.
  --help                          Show this message and exit.
```
Bram Daams's avatar
Bram Daams committed
59
60
61
62
63
64
65
66
67

## Configuration
Create a configuration file `/etc/sch.conf` that looks like:
``` ini
[hc]
healthchecks_api_url = https://hc.example.com/api/v1/
healthchecks_api_key = xxmysecretkeyxx
```

Bram Daams's avatar
Bram Daams committed
68
And fill in the API URL and the key obtained from the Healthchecks project
Bram Daams's avatar
Bram Daams committed
69
settings block labeled "API Access".
Bram Daams's avatar
Bram Daams committed
70

Bram Daams's avatar
Bram Daams committed
71
## Monitoring cron jobs
Bram Daams's avatar
Bram Daams committed
72
Just decorate your existing cron tabs by specifying the alternative `sch`:
Bram Daams's avatar
Bram Daams committed
73
74
75
76
```
SHELL=/usr/local/bin/sch
```
This line should be above the cron lines you want to have monitored by Healthchecks.
Bram Daams's avatar
Bram Daams committed
77

Bram Daams's avatar
Bram Daams committed
78
79
80
81
82
Only jobs with the environment variable `JOB_ID`, ie:
```
*/5 * * * * root JOB_ID=some_id /path/to/some_command
```
The value of `JOB_ID` should be unique for the host.
83

Bram Daams's avatar
Bram Daams committed
84
85
The combination of the `JOB_ID` environment variable and the `sch` shell is enough
to have the job checked in Healthchecks.
Bram Daams's avatar
Bram Daams committed
86

Bram Daams's avatar
Bram Daams committed
87
88
89
90
91
92
At each run of the job, `sch` will take care that the schedule, description and 
other metadata is synchronized whenever there's a change in the cron job. Just
makes sure to not change the `JOB_ID` (or it will create a new check).
 
### Other meta data
The following data is used to configure a corresponding Healthchecks check:
Bram Daams's avatar
Bram Daams committed
93
- `JOB_ID`: the environment variable is used for the name of the check and a tag named `job_id={value of JOB_ID}`
Bram Daams's avatar
Bram Daams committed
94
- the cron lines' **comment** is used for the description of the check. The comment line just above a cron line or the inline comment is used
Bram Daams's avatar
Bram Daams committed
95
- `JOB_TAGS`: use this environment variable in a job to specify tag names separated by a comma to specify additional tags
Bram Daams's avatar
Bram Daams committed
96
- `$USER`: the current user running the cron command is used to create a tag named `user=$USER`
Bram Daams's avatar
Bram Daams committed
97
- the jobs **schedule** and the hosts **timezone** is used to set the checks schedule
98
- `JOB_GRACE`: the value of this environment variable is used to set the grace time in seconds for the check. See JOB_GRACE for valid interval formats.
Bram Daams's avatar
Bram Daams committed
99
- when registering a new check and JOB_GRACE is not set, the **execution time** of the command is used to set an initial grace time. The grace time will be set to 1.2 times the execution time + 30 seconds. As per the Healthchecks API, the minimal grace time is 1 minute and the maximum grace time is 30 days.
Bram Daams's avatar
Bram Daams committed
100

Bram Daams's avatar
Bram Daams committed
101
102
103
104
105
106
An example of a cron file that touches most of the functionality would look like:
```
SHELL=/usr/local/bin/sch
# if this check fails, the host is probably offline
* * * * * root JOB_ID=true /bin/true
```
Bram Daams's avatar
Bram Daams committed
107
Although above cron job is useful, a more realistic could look like:
Bram Daams's avatar
Bram Daams committed
108
109
110
```
SHELL=/usr/loca/bin/sch
# super important backup, if this one fails: fix with top priority!
Bram Daams's avatar
Bram Daams committed
111
10 8-20/2 * mon-fri  backup  JOB_ID=db-backups JOB_TAGS=db,backup,my_project JOB_GRACE=5m /usr/local/bin/run-db-backups
Bram Daams's avatar
Bram Daams committed
112
```
Bram Daams's avatar
Bram Daams committed
113

114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
#### JOB_GRACE interval format
If no suffixes are used, seconds are assumed.
You can make use of the following suffixes to specify an interval:

| Suffix | Interval |
|--------|----------|
| s      | seconds  |
| m      | minutes  |
| h      | hours    |
| D      | days     |
| W      | weeks    |
| M      | months   |
| Y      | years    |

Although days and weeks are accepted, you might want to limit the interval to several hours ;-)

Bram Daams's avatar
Bram Daams committed
130
Examples:
131

Bram Daams's avatar
Bram Daams committed
132
133
134
135
136
137
| environment variable | grace time |
|----------------------|------------|
| `JOB_GRACE=5m`       | 300s       |
| `JOB_GRACE=120`      | 120s       |
| `JOB_GRACE=1h30m`    | 5400s      |

138

Bram Daams's avatar
Bram Daams committed
139
140
141
142
143
144
145
### Job execution
`sch` takes over the role of the shell. Jobs not containing the `JOB_ID` environment variable are directly executed with `os.system`.
For `sch` managed jobs:
- `sch` will start with pinging `/start` endpoint of the check
- os.sytem executes the command
- depending on the exit code, it will ping for success or ping the `/fail` end point on failure

Bram Daams's avatar
Bram Daams committed
146
### References
Bram Daams's avatar
Bram Daams committed
147
* python-crontab <https://pypi.org/project/python-crontab/>
Bram Daams's avatar
Bram Daams committed
148
* crab <https://github.com/grahambell/crab>
Bram Daams's avatar
Bram Daams committed
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171

## Notes
### fully qualified domain name
`sch` uses the FQDN to identify the hosts it's running on. You can check the FQDN with:
``` console
$ hostname --fqdn
host.example.com
```

However, on some systems that don't know the domain part, it just returns the
(short) hostname instead:
``` console
$ hostname --fqdn
host
```

If this is the case, you can fix that by editing the `/etc/hosts` file so look
like this:
```
127.0.0.1	localhost
127.0.1.1	host.example.com host
```

Bram Daams's avatar
comma    
Bram Daams committed
172
Afterwards, `hostname --fqdn` should return the FQDN. Beware that `sch` will
Bram Daams's avatar
Bram Daams committed
173
create new checks when the FQDN changes.