Verified Commit 8a4cf927 authored by Hans-Nikolai Viessmann's avatar Hans-Nikolai Viessmann
Browse files

add scripts, upload readme

parent 23948e65
---------------------------------------------------------------------------
SAC - Single Assignment C
---------------------------------------------------------------------------
SAC COPYRIGHT NOTICE, LICENSE, AND DISCLAIMER
(c) Copyright 1994 - 2020 by
SAC Development Team
web: http://www.sac-home.org
email: info@sac-home.org
---------------------------------------------------------------------------
The SAC compiler, the SAC standard library, and all accompanying
software and documentation (in the following named this software)
is developed by the SAC Development Team (in the following named
the developer) which reserves all rights on this software.
Permission to use this software is hereby granted free of charge
for any non-profit purpose in a non-commercial environment, i.e. for
educational or research purposes in a non-profit institute or for
personal, non-commercial use. For this kind of use it is allowed to
copy or redistribute this software under the condition that the
complete distribution for a certain platform is copied or
redistributed and this copyright notice, license agreement, and
warranty disclaimer appears in each copy. ANY use of this software with
a commercial purpose or in a commercial environment is not granted by
this license.
The developer disclaims all warranties with regard to this software,
including all implied warranties of merchantability and fitness. In no
event shall the developer be liable for any special, indirect or
consequential damages or any damages whatsoever resulting from loss of
use, data, or profits, whether in an action of contract, negligence, or
other tortuous action, arising out of or in connection with the use or
performance of this software. The entire risk as to the quality and
performance of this software is with you. Should this software prove
defective, you assume the cost of all servicing, repair, or correction.
---------------------------------------------------------------------------
......@@ -3,72 +3,94 @@ SaC Packages
In this repository we store packages associated with the SaC projects using Git LFS.
We achieve two advantages from this,
We currently creates packages for:
1. we are able to associate each package release/build with a new Git commit, which means
we can build a history of each release.
2. easier to access/find a given release. Our current listing via https://sac-home.org is
not very easy to look through, and has caused users to be confused. Unfortunately we do
create a few different releases for many different platforms and so per build we have 10+
packages. These are listed with previously created packages, thus making it difficult to
gain an overview here.
3. We can mark releases as having problems/known-issues, so that users are better informed
about what release is more likely to work for them.
- RHEL 6, RHEL 7
- Ubuntu 16.04, Ubuntu 18.04
- Linux generic (based on ArchLinux)
- MacOS 10.14+
To get the latest packages, checkout the [release page](https://gitlab.science.ru.nl/sac-group/sac-packages/-/releases).
## Things to be aware of
### Package variants
We provide two types of packages variants, _basic_ and _full_.
Basic package
- package excludes special features, like the GPU backend or advanced multi-threading scheduling.
- makes use of minimal external dependencies: `gcc`, `libc`, `uuid-lib`.
Full package
- package includes GPU backend and advanced multi-threading scheduling.
- more external dependencies: `gcc`, `libc`, `uuid-lib`, `hwloc`, `cuda`[^1]
Process
-------
**We suggest that you use the _basic_ package variant, unless you are sure you
need CUDA.**
As we are using containers to build our packages we do the following, there are a few ways
that we can get the built packages into this LFS repository:
### Externally-maintained packages
* we commit them directly (possible race-condition as jobs may push at same time):
1. build packages in pipeline
2. in cloned packages repo, place packages in correct directory, and push
3. when pipeline completes, signal this repo to run pipeline
4. this pipeline will look at git history since last tag, get all new files, generate
URLs, and create a new tag and associate a release message with the new files.
* we capture the assets via GitLab Pipeline/Job API:
1. build packages in pipeline
2. when pipeline completes, signal this repo to run pipeline
3. this pipeline will call API to get _latest_ completed jobs from `sac2c` repo, and
retrieve all the build artifacts.
4. We then generate URLs, and create a new tag and associate a release message with the
new files.
We also have some user-contributed packages — as these are packages outwith the
project, we can not provide any support on these. Please communicate with the
package maintainers directly.
The latter option seems to be much simplier, and also avoids race-conditions.
| OS | External Link |
| --- | --- |
| ArchLinux | via [sac-compiler-weekly](https://aur.archlinux.org/packages/sac-compiler-weekly) and [sac-stdlib-weekly](https://aur.archlinux.org/packages/sac-stdlib-weekly)
The artifacts API uses the following `GET` URL: `GET /projects/:id/jobs/artifacts/:ref_name/download?job=name`,
where `id` is the project ID, `ref_name` is the GIT branch name, and `name` is the name of
the job used to build the packages. This API call will check for the _latest_ successfully
completed pipeline, and retrieved an archive containing all the build artifacts for that job.
To be clear, we need to call this for *all* job names in order to get all packages.
## Installation
Requirements
------------
### Linux generic package
The following changes are needed for both the existing package building project
(at https://www.macs.hw.ac.uk/gitlab/sac-group/build-sac-pkgs) and this repository.
The contents of this package can be installed anywhere on your system (this is
particularly useful if you do not have root permission!). Upon open the archive,
you will find a README file and an install script (`install.sh`). Please read
the README on how to use the install script.
- [x] the build project no longer should try to SSH into MACS server - this means we can remove
some checks about version number there. Instead we need now call the GitLab API to check
if the release already exists (in the case where we do weeklies).
## Further Information
Model for this is: `https://gitlab.science.ru.nl/api/v4/projects/3930/releases/v1.3.3-359-4`
We need to provide an _access token_ in order to make this call.
- once the build pipeline completes succesfully, we need to signal the package repo that
this has happened. The easiest way to do this is to add a final stage to the build pipeline
which calls the trigger API of the package pipeline.
- The package pipeline needs to know what jobs to get the artifacts from, we can do this by pass
the job names as a variable in the hook, something like `-F "variables[RUN_NIGHTLY_BUILD]=true"`
### Repository Directory Scheme
Access LFS packages via HTTP(S)
-------------------------------
Packages are stored under `packages`, using the following directory structure scheme:
As far as I know, there is no GitLab API to retrieve this value, however the LFS objects are
treated as standard assets within the GitLab WebUI, meaning there is a consistent semantic URL
for each object: `https://gitlab.science.ru.nl/sac-group/sac-packages/raw/master/<path in repo>`
```
packages/<release type>/<platform>/<version>/<variant>/...pkgs...
```
Given this, it is straightforward to associate a SaC release with a GitLab tracked release page.
release type
: we have two release types, _release_ and _weekly_. The first indicates a stable
package that could be using in production solution, and the latter is unstable. In practice
we don't release many _release_ packages as the compiler is constantly in flux. That being
said we try to ensure that the _weekly_ package is functional.
platform
: we create packages for different platforms
version
: We use semantic versioning, but additionally store the commit count (since the
last version change) and the release number. Using `1.3.2-256-1` as an example,
`1.3.2` is the version number which has `256` new commits on top and has been
packages/release only once (e.g. `1`).
variant
: We provide two types of packages variants, *basic* and *full*.
## FAQ
### Why store the package here?
1. we are able to associate each package release/build with a new Git commit, which means
we can build a history of each release.
2. easier to access/find a given release. Our previous listing via https://sac-home.org was
not very easy to look through, and did cause users to be confused. Unfortunately we do
create a few different releases for many different platforms and so per build we have 10+
packages. These are listed with previously created packages, thus making it difficult to
gain an overview there.
3. We can mark releases as having problems/known-issues, so that users are better informed
about what release is more likely to work for them.
## Footnotes
[^1]: CUDA must be installed as per the recommended guidelines of the distribution. Our packages
are built using docker containers provided by NVIDIA, look there for their installation procedure.
SaC Packages
============
In this repository we store packages associated with the SaC projects using Git LFS.
We achieve two advantages from this,
1. we are able to associate each package release/build with a new Git commit, which means
we can build a history of each release.
2. easier to access/find a given release. Our current listing via https://sac-home.org is
not very easy to look through, and has caused users to be confused. Unfortunately we do
create a few different releases for many different platforms and so per build we have 10+
packages. These are listed with previously created packages, thus making it difficult to
gain an overview here.
3. We can mark releases as having problems/known-issues, so that users are better informed
about what release is more likely to work for them.
Process
-------
As we are using containers to build our packages we do the following, there are a few ways
that we can get the built packages into this LFS repository:
* we commit them directly (possible race-condition as jobs may push at same time):
1. build packages in pipeline
2. in cloned packages repo, place packages in correct directory, and push
3. when pipeline completes, signal this repo to run pipeline
4. this pipeline will look at git history since last tag, get all new files, generate
URLs, and create a new tag and associate a release message with the new files.
* we capture the assets via GitLab Pipeline/Job API:
1. build packages in pipeline
2. when pipeline completes, signal this repo to run pipeline
3. this pipeline will call API to get _latest_ completed jobs from `sac2c` repo, and
retrieve all the build artifacts.
4. We then generate URLs, and create a new tag and associate a release message with the
new files.
The latter option seems to be much simplier, and also avoids race-conditions.
The artifacts API uses the following `GET` URL: `GET /projects/:id/jobs/artifacts/:ref_name/download?job=name`,
where `id` is the project ID, `ref_name` is the GIT branch name, and `name` is the name of
the job used to build the packages. This API call will check for the _latest_ successfully
completed pipeline, and retrieved an archive containing all the build artifacts for that job.
To be clear, we need to call this for *all* job names in order to get all packages.
Requirements
------------
The following changes are needed for both the existing package building project
(at https://www.macs.hw.ac.uk/gitlab/sac-group/build-sac-pkgs) and this repository.
- [x] the build project no longer should try to SSH into MACS server - this means we can remove
some checks about version number there. Instead we need now call the GitLab API to check
if the release already exists (in the case where we do weeklies).
Model for this is: `https://gitlab.science.ru.nl/api/v4/projects/3930/releases/v1.3.3-359-4`
We need to provide an _access token_ in order to make this call.
- once the build pipeline completes succesfully, we need to signal the package repo that
this has happened. The easiest way to do this is to add a final stage to the build pipeline
which calls the trigger API of the package pipeline.
- The package pipeline needs to know what jobs to get the artifacts from, we can do this by pass
the job names as a variable in the hook, something like `-F "variables[RUN_NIGHTLY_BUILD]=true"`
Access LFS packages via HTTP(S)
-------------------------------
As far as I know, there is no GitLab API to retrieve this value, however the LFS objects are
treated as standard assets within the GitLab WebUI, meaning there is a consistent semantic URL
for each object: `https://gitlab.science.ru.nl/sac-group/sac-packages/raw/master/<path in repo>`
Given this, it is straightforward to associate a SaC release with a GitLab tracked release page.
#!/usr/bin/env bash
# this script gathers all the build artifacts from a specified Gitlab CI
# pipeline. It does this by parsing *all* jobs and retrieving the artifact
# archive for each job. It then unpacks these, assuming that the archives
# have a proper directory-structure, e.g.
#
# <schedule or release>/<distro>/<version>/<variant>/... files ...
#
# Note: makes use of external tools `curl`, `jq`, `unzip`
declare -g APIKEY GLURL PRID PIPEID VERB=false
usage() { echo "Usage: $0 [-v...] -a <api key> -u <gitlab url> -p <project id> -c <pipeline id>" >&2; exit 0; }
msginfo() { ${VERB} && echo "-- $*" >&2; }
msgerror() { echo "-! $*" >&2; }
hasartifact () {
declare -i ret
msginfo "parsing JSON for job ${1}"
if ! curl -s --header "PRIVATE-TOKEN: ${APIKEY}" "${GLURL}/api/v4/projects/${PRID}/jobs/${1}" | \
jq -e '[.artifacts[].file_type == "archive"] | any' >/dev/null; then
ret=1
else
ret=0
fi
return $ret
}
# we need to get the project ID and process the assets
while getopts "a:u:p:c:v" o; do
case "${o}" in
a)
APIKEY=${OPTARG}
;;
u)
GLURL=${OPTARG}
;;
p)
PRID=${OPTARG}
;;
c)
PIPEID=${OPTARG}
;;
v)
VERB=true
;;
*)
usage
;;
esac
done
shift $((OPTIND-1))
if [ -z "${APIKEY}" ] || [ -z "${GLURL}" ] || [ -z "${PRID}" ] || [ -z "${PIPEID}" ]; then
usage
fi
# get list of job ids and retrieve artifact archives
while IFS=' ' read -r jobid jobname; do
echo "Retrieving artifacts for \`$jobname' ($jobid)"
if hasartifact "$jobid"; then
if ! curl --output "artifact-${jobname}.zip" --header "PRIVATE-TOKEN: ${APIKEY}" \
"${GLURL}/api/v4/projects/${PRID}/jobs/${jobid}/artifacts"; then
msgerror "Unable to retireve artifact, curl exit code \`$?'"
exit 1
fi
else
msgerror "Job \`$jobname' has no artifacts!"
#exit 2
fi
done <<< "$(curl -s --header "PRIVATE-TOKEN: ${APIKEY}" "${GLURL}/api/v4/projects/${PRID}/pipelines/${PIPEID}/jobs" | \
jq -r '.[] | ((.id | tostring) + " " + .name)')"
for archive in artifact-*.zip; do
msginfo "unpacking $archive..."
if ! unzip "$archive" -d packages/; then
msgerror "Unable to unzip $archive, exit code $?"
exit 3
fi
rm "$archive"
done
exit 0
#!/usr/bin/env bash
# this script generated the _template_ for the release notes with all the
# links for the packages provided. This assumes that our packages are well
# named!
#
# this script makes use of `find`.
declare -g -r GURL='https://gitlab.science.ru.nl/sac-group/sac-packages/-/raw/master'
declare -g REL WLY=false MK=true
usage() { echo "Usage: $0 [-w] [-n] <version>" >&2; exit 0; }
# we need to get the project ID and process the assets
while getopts "wn" o; do
case "${o}" in
w)
WLY=true
;;
n)
MK=false
;;
*)
usage
;;
esac
done
shift $((OPTIND-1))
declare -g -r VERSION="$1"
[ $WLY = true ] && REL="weekly" || REL="release"
if [ -z "${VERSION}" ]; then
usage
fi
if [ $MK = false ]; then
cat << EOF
^ System ^ Basic ^ Full ^
| Linux (generic) | $(find . -type f -path "*/${REL}/Linux/${VERSION}/basic/*" -printf "[[${GURL}/%P|%f]]//" | rev | cut -c 3- | rev) | $(find . -type f -path "*/${REL}/Linux/${VERSION}/full/*" -printf "[[${GURL}/%P|%f]]//" | rev | cut -c 3- | rev) |
| RHEL 7 | $(find . -type f -path "*/${REL}/RHEL7/${VERSION}/basic/*" -printf "[[${GURL}/%P|%f]]//" | rev | cut -c 3- | rev) | $(find . -type f -path "*/${REL}/RHEL7/${VERSION}/full/*" -printf "[[${GURL}/%P|%f]]//" | rev | cut -c 3- | rev) |
| RHEL 6 | $(find . -type f -path "*/${REL}/RHEL6/${VERSION}/basic/*" -printf "[[${GURL}/%P|%f]]//" | rev | cut -c 3- | rev) | $(find . -type f -path "*/${REL}/RHEL6/${VERSION}/full/*" -printf "[[${GURL}/%P|%f]]//" | rev | cut -c 3- | rev) |
| Ubuntu 18.04 | $(find . -type f -path "*/${REL}/Ubl18/${VERSION}/basic/*" -printf "[[${GURL}/%P|%f]]//" | rev | cut -c 3- | rev) | $(find . -type f -path "*/${REL}/Ubl18/${VERSION}/full/*" -printf "[[${GURL}/%P|%f]]//" | rev | cut -c 3- | rev) |
| Ubuntu 16.04 | $(find . -type f -path "*/${REL}/Ubl16/${VERSION}/basic/*" -printf "[[${GURL}/%P|%f]]//" | rev | cut -c 3- | rev) | $(find . -type f -path "*/${REL}/Ubl16/${VERSION}/full/*" -printf "[[${GURL}/%P|%f]]//" | rev | cut -c 3- | rev) |
| MacOS | $(find . -type f -path "*/${REL}/MacOS/${VERSION}/basic/*" -printf "[[${GURL}/%P|%f]]//" | rev | cut -c 3- | rev) | N/A |
EOF
else
cat << EOF
About
===
_pending_$([ $WLY = true ] && printf "\n\n_This is a weekly release, which is based on the latest upstream changes. It may be unstable._")
Changelog
===
_pending_
Package links[^1]
===
| System | Basic | Full |
| ------: | ------ | ----- |
| Linux (generic) | $(find . -type f -path "*/${REL}/Linux/${VERSION}/basic/*" -printf "[%f](${GURL}/%P)<br>" | rev | cut -c 5- | rev) | $(find . -type f -path "*/${REL}/Linux/${VERSION}/full/*" -printf "[%f](${GURL}/%P)<br>" | rev | cut -c 5- | rev) |
| RHEL 7 | $(find . -type f -path "*/${REL}/RHEL7/${VERSION}/basic/*" -printf "[%f](${GURL}/%P)<br>" | rev | cut -c 5- | rev) | $(find . -type f -path "*/${REL}/RHEL7/${VERSION}/full/*" -printf "[%f](${GURL}/%P)<br>" | rev | cut -c 5- | rev) |
| RHEL 6 | $(find . -type f -path "*/${REL}/RHEL6/${VERSION}/basic/*" -printf "[%f](${GURL}/%P)<br>" | rev | cut -c 5- | rev) | $(find . -type f -path "*/${REL}/RHEL6/${VERSION}/full/*" -printf "[%f](${GURL}/%P)<br>" | rev | cut -c 5- | rev) |
| Ubuntu 18.04 | $(find . -type f -path "*/${REL}/Ubl18/${VERSION}/basic/*" -printf "[%f](${GURL}/%P)<br>" | rev | cut -c 5- | rev) | $(find . -type f -path "*/${REL}/Ubl18/${VERSION}/full/*" -printf "[%f](${GURL}/%P)<br>" | rev | cut -c 5- | rev) |
| Ubuntu 16.04 | $(find . -type f -path "*/${REL}/Ubl16/${VERSION}/basic/*" -printf "[%f](${GURL}/%P)<br>" | rev | cut -c 5- | rev) | $(find . -type f -path "*/${REL}/Ubl16/${VERSION}/full/*" -printf "[%f](${GURL}/%P)<br>" | rev | cut -c 5- | rev) |
| MacOS | $(find . -type f -path "*/${REL}/MacOS/${VERSION}/basic/*" -printf "[%f](${GURL}/%P)<br>" | rev | cut -c 5- | rev) | N/A |
### Notes
[^1]: you can also view these packages within the repository and access them via Git-LFS
EOF
fi
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment