README.md 5.91 KB
Newer Older
Bram Daams's avatar
Bram Daams committed
1
2
# SmartCronHelper
A cron shell wrapper for registering and updating cron jobs automatically in
Bram Daams's avatar
Bram Daams committed
3
[healthchecks](https://healthchecks.io) or your [own hosted copy of Healthchecks](https://github.com/healthchecks/healthchecks).
Bram Daams's avatar
Bram Daams committed
4

Bram Daams's avatar
Bram Daams committed
5
> warning: this software package should be considered "alpha"
Bram Daams's avatar
Bram Daams committed
6

Bram Daams's avatar
Bram Daams committed
7
8

## Installation
Bram Daams's avatar
Bram Daams committed
9
Install sch system wide with pip
Bram Daams's avatar
Bram Daams committed
10
``` console
Bram Daams's avatar
Bram Daams committed
11
$ sudo pip3 install sch
Bram Daams's avatar
Bram Daams committed
12
13
```

Bram Daams's avatar
Bram Daams committed
14
A `sch` cli should now be availble:
Bram Daams's avatar
Bram Daams committed
15
``` console
Bram Daams's avatar
Bram Daams committed
16
17
$ which sch
/usr/local/bin/sch
Bram Daams's avatar
Bram Daams committed
18
```
Bram Daams's avatar
Bram Daams committed
19
20
21
22
23

`sch --version` should return something like:
``` console
sch, version 0.2.1
```
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
## Command line usage
See the `--help` option for usage:
``` console
Usage: sch [OPTIONS] COMMAND [ARGS]...

  sch - A cron shell wrapper for registering and updating cron jobs
  automatically in Healthchecks. The Healthchecks project api_url and
  api_key should be configured in /etc/sch.conf.

Options:
  --version                 Show the version and exit.
  -c, --shell_command TEXT  Command to execute. This how Cron executes 'sch'
                            when it is set as SHELL.
  --help                    Show this message and exit.

Commands:
  list  List checks for the configured Healthchecks project.
```

### `list` command
``` console
Usage: sch list [OPTIONS]

  List checks for the configured Healthchecks project.

Options:
  -l, --localhost / -a, --all     List checks that originate from this host
                                  (default) or list all checks.
  -s, --status [up|down|grace|started|pause|new]
                                  Show only checks that have the specified
                                  status.
  --help                          Show this message and exit.
```
Bram Daams's avatar
Bram Daams committed
57
58
59
60
61
62
63
64
65

## Configuration
Create a configuration file `/etc/sch.conf` that looks like:
``` ini
[hc]
healthchecks_api_url = https://hc.example.com/api/v1/
healthchecks_api_key = xxmysecretkeyxx
```

Bram Daams's avatar
Bram Daams committed
66
And fill in the API URL and the key obtained from the Healthchecks project
Bram Daams's avatar
Bram Daams committed
67
settings block labeled "API Access".
Bram Daams's avatar
Bram Daams committed
68

Bram Daams's avatar
Bram Daams committed
69
## Monitoring cron jobs
Bram Daams's avatar
Bram Daams committed
70
Just decorate your existing cron tabs by specifying the alternative `sch`:
Bram Daams's avatar
Bram Daams committed
71
72
73
74
```
SHELL=/usr/local/bin/sch
```
This line should be above the cron lines you want to have monitored by Healthchecks.
Bram Daams's avatar
Bram Daams committed
75

Bram Daams's avatar
Bram Daams committed
76
77
78
79
80
Only jobs with the environment variable `JOB_ID`, ie:
```
*/5 * * * * root JOB_ID=some_id /path/to/some_command
```
The value of `JOB_ID` should be unique for the host.
81

Bram Daams's avatar
Bram Daams committed
82
83
The combination of the `JOB_ID` environment variable and the `sch` shell is enough
to have the job checked in Healthchecks.
Bram Daams's avatar
Bram Daams committed
84

Bram Daams's avatar
Bram Daams committed
85
86
87
88
89
90
At each run of the job, `sch` will take care that the schedule, description and 
other metadata is synchronized whenever there's a change in the cron job. Just
makes sure to not change the `JOB_ID` (or it will create a new check).
 
### Other meta data
The following data is used to configure a corresponding Healthchecks check:
Bram Daams's avatar
Bram Daams committed
91
- `JOB_ID`: the environment variable is used for the name of the check and a tag named `job_id={value of JOB_ID}`
Bram Daams's avatar
Bram Daams committed
92
- the cron lines' **comment** is used for the description of the check. The comment line just above a cron line or the inline comment is used
Bram Daams's avatar
Bram Daams committed
93
- `JOB_TAGS`: use this environment variable in a job to specify tag names separated by a comma to specify additional tags
Bram Daams's avatar
Bram Daams committed
94
- `$USER`: the current user running the cron command is used to create a tag named `user=$USER`
Bram Daams's avatar
Bram Daams committed
95
- the jobs **schedule** and the hosts **timezone** is used to set the checks schedule
96
- `JOB_GRACE`: the value of this environment variable is used to set the grace time in seconds for the check. See JOB_GRACE for valid interval formats.
Bram Daams's avatar
Bram Daams committed
97
- when registering a new check and JOB_GRACE is not set, the **execution time** of the command is used to set an initial grace time. The grace time will be set to 1.2 times the execution time + 30 seconds. As per the Healthchecks API, the minimal grace time is 1 minute and the maximum grace time is 30 days.
Bram Daams's avatar
Bram Daams committed
98

Bram Daams's avatar
Bram Daams committed
99
100
101
102
103
104
An example of a cron file that touches most of the functionality would look like:
```
SHELL=/usr/local/bin/sch
# if this check fails, the host is probably offline
* * * * * root JOB_ID=true /bin/true
```
Bram Daams's avatar
Bram Daams committed
105
Although above cron job is useful, a more realistic could look like:
Bram Daams's avatar
Bram Daams committed
106
107
108
```
SHELL=/usr/loca/bin/sch
# super important backup, if this one fails: fix with top priority!
Bram Daams's avatar
Bram Daams committed
109
10 8-20/2 * mon-fri  backup  JOB_ID=db-backups JOB_TAGS=db,backup,my_project JOB_GRACE=5m /usr/local/bin/run-db-backups
Bram Daams's avatar
Bram Daams committed
110
```
Bram Daams's avatar
Bram Daams committed
111

112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
#### JOB_GRACE interval format
If no suffixes are used, seconds are assumed.
You can make use of the following suffixes to specify an interval:

| Suffix | Interval |
|--------|----------|
| s      | seconds  |
| m      | minutes  |
| h      | hours    |
| D      | days     |
| W      | weeks    |
| M      | months   |
| Y      | years    |

Although days and weeks are accepted, you might want to limit the interval to several hours ;-)

Bram Daams's avatar
Bram Daams committed
128
Examples:
129

Bram Daams's avatar
Bram Daams committed
130
131
132
133
134
135
| environment variable | grace time |
|----------------------|------------|
| `JOB_GRACE=5m`       | 300s       |
| `JOB_GRACE=120`      | 120s       |
| `JOB_GRACE=1h30m`    | 5400s      |

136

Bram Daams's avatar
Bram Daams committed
137
138
139
140
141
142
143
### Job execution
`sch` takes over the role of the shell. Jobs not containing the `JOB_ID` environment variable are directly executed with `os.system`.
For `sch` managed jobs:
- `sch` will start with pinging `/start` endpoint of the check
- os.sytem executes the command
- depending on the exit code, it will ping for success or ping the `/fail` end point on failure

Bram Daams's avatar
Bram Daams committed
144
### References
Bram Daams's avatar
Bram Daams committed
145
* python-crontab <https://pypi.org/project/python-crontab/>
Bram Daams's avatar
Bram Daams committed
146
* crab <https://github.com/grahambell/crab>
Bram Daams's avatar
Bram Daams committed
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169

## Notes
### fully qualified domain name
`sch` uses the FQDN to identify the hosts it's running on. You can check the FQDN with:
``` console
$ hostname --fqdn
host.example.com
```

However, on some systems that don't know the domain part, it just returns the
(short) hostname instead:
``` console
$ hostname --fqdn
host
```

If this is the case, you can fix that by editing the `/etc/hosts` file so look
like this:
```
127.0.0.1	localhost
127.0.1.1	host.example.com host
```

Bram Daams's avatar
comma    
Bram Daams committed
170
Afterwards, `hostname --fqdn` should return the FQDN. Beware that `sch` will
Bram Daams's avatar
Bram Daams committed
171
create new checks when the FQDN changes.