Use a centralised file for library information

.dockerignore

+db/storage

CONTRIBUTING.md

@@ -53,26 +53,34 @@ We will take immediate action.
 - Add yourself to the authors list in `README.md`
 
 ## 3. I just want to add library X
-If the library is not well known it is advised to first open an issue to see
+If the library is not well-known it is advised to first open an issue to see
 whether it is suitable to be indexed by cloogle.
 
-To add a library you have to add it in to the following places:
-
-- `backend/builddb.icl`
-
-	Add your library to the `zero` instance of `CLI` to get it indexed.
-- `backend/Dockerfile`
-
-	In this file you have to add the download of the library. E.g. add a line
-	to get the files to the docker (e.g. with subversion, git, mercurial etc.)
-	to `/opt/clean/lib`.
-- `frontend/Dockerfile`
-
-	In this file do the same as in the backend to make the files through the
-	web frontend.
-- `frontend/index.html`
-
-	Add your library to the checkboxes in the miscellaneous column.
+To add a library you have to modify [`libs.json`](/libs.json). This file is a
+JSON record with three collections of Clean libraries. Usually, you should add
+the new library to the `Miscellaneous` collection. The newly added item may
+contain the following fields:
+
+- `name` (**required**): a human-readable name.
+- `fetch_url` (**required**): choose one of:
+  - `["Git", "<URL>"]` where `<URL>` points to a public git repository;
+  - `["SVN", "<URL>"]` where `<URL>` points to a public Subversion repository;
+  - `["CleanDistribution", "<NAME>"]`, if the library is distributed in Clean
+    nightlies as `<NAME>`.
+- `path`: the path from the root of the repository to the files that should be
+  indexed.
+- `info_url`: a URL to an informative page about the library.
+- `pattern_exclude`: a pattern (see below) for files to exclude.
+- `pattern_app`: a pattern (see below) for modules that should be marked as
+  apps rather than libraries.
+- `pattern_core`: a pattern (see below) for modules that are part of the core
+  of the library.
+
+Patterns are JSON lists of simple patterns. A simple pattern is one of:
+
+- `["PWildcard"]`, to match everything;
+- `["PStartsWith",<S>]`, to match paths starting with `<S>`;
+- `["PNot",<P>]`, to negate the simple pattern `<P>`.
 
 ## 4. I just want to add a new user agent to the statistics
 If you have created a new Cloogle client, please give it a specific user agent

Cloogle @ 96f1ea60

-Subproject commit f700b32676d72bd559de72cdf67a35bd642b759a
+Subproject commit 96f1ea6060f46f72c37fe17f83e2c447a5f96a5f

backend/Dockerfile

 FROM camilstaps/clean:nightly
 
-COPY . /usr/src/cloogle
+COPY backend /usr/src/cloogle
+COPY libs.json /usr/src/cloogle/libs.json
+COPY util/fetch_libs.sh /usr/src/cloogle/fetch_libs.sh
 WORKDIR /usr/src/cloogle
 
 RUN install_clean.sh bundle-complete 2018-03-27 \
-	&& PACKAGES="patch" \
+	&& PACKAGES="patch jq" \
 	&& apt-get update -qq\
 	&& apt-get install -qq $PACKAGES --no-install-recommends\
 	&& make distclean CloogleServer builddb\
-	&& rm -r /opt/clean/lib/*\
-	&& svn checkout https://svn.cs.ru.nl/repos/SoccerFun/src /opt/clean/lib/SoccerFun \
-	&& svn checkout https://svn.cs.ru.nl/repos/clean-libraries/trunk/Libraries/ObjectIO/ObjectIO /opt/clean/lib/ObjectIO \
-	&& svn checkout https://svn.cs.ru.nl/repos/clean-libraries/trunk/Libraries/MersenneTwister /opt/clean/lib/MersenneTwister \
-	&& svn checkout https://svn.cs.ru.nl/repos/clean-compiler/trunk /opt/clean/lib/clean-compiler \
-	&& svn checkout https://svn.cs.ru.nl/repos/clean-ide/trunk /opt/clean/lib/clean-ide \
-	&& git clone https://gitlab.science.ru.nl/mlubbers/CleanSerial /opt/clean/lib/CleanSerial \
-	&& git clone https://github.com/clean-cloogle/Cloogle /opt/clean/lib/Cloogle \
-	&& git clone https://github.com/clean-cloogle/libcloogle /opt/clean/lib/libcloogle \
-	&& git clone https://github.com/clean-cloogle/CleanTypeUnifier /opt/clean/lib/CleanTypeUnifier \
-	&& git clone https://github.com/clean-cloogle/CleanPrettyPrint /opt/clean/lib/CleanPrettyPrint \
-	&& git clone https://github.com/camilstaps/CleanInotify /opt/clean/lib/CleanInotify \
-	&& git clone https://github.com/camilstaps/CleanSnappy /opt/clean/lib/CleanSnappy \
-	&& git clone https://github.com/clean-cloogle/clean-irc /opt/clean/lib/clean-irc \
-	&& git clone https://github.com/dopefishh/clean-selectloop /tmp/clean-selectloop \
-		&& mv /tmp/clean-selectloop/libraries /opt/clean/lib/clean-selectloop \
-	&& curl -sSL ftp://ftp.cs.ru.nl/pub/Clean/builds/linux-x64/clean-bundle-complete-linux-x64-latest.tgz |\
-	 	tar -xz -C /opt/clean --strip-components=1\
-	&& rm -r /opt/clean/lib/StdEnv\
-	&& git clone https://github.com/clean-cloogle/StdEnv-doc /opt/clean/lib/StdEnv\
+	&& ./fetch_libs.sh /opt/clean/lib\
 	&& make types.json\
 	&& rm -rf \
 		Cloogle \

backend/builddb.icl

@@ -9,75 +9,51 @@ import StdMisc
 import StdString
 import StdTuple
 
+import Control.Monad => qualified join
 import Data.Either
+import Data.Error
 from Data.Func import $, hyperstrict, mapSt
 import Data.Functor
 import Data.List
 import Data.Maybe
 import System.CommandLine
-from Text import class Text(concat,startsWith), instance Text String
+import System.File
+from Text import class Text(join,startsWith), instance Text String
+import Text.GenJSON
 
 import CloogleDB
 import Type
 from CloogleDBFactory import :: TemporaryDB, newTemporaryDB, finaliseDB,
-	findModules, indexModule, constructor_functions, record_functions
+	findModules, indexModule, constructor_functions, record_functions,
+	:: IndexItem, :: SourceURL, :: PathPattern
 
 import Builtins
 
-:: CLI = { help    :: Bool
-         , version :: Bool
-         , root    :: String
-         , libs    :: [(String, String ModuleEntry -> ModuleEntry)]
-         , exclude :: [String]
-         }
+:: CLI =
+	{ help      :: !Bool
+	, root      :: !String
+	, libs_file :: !String
+	}
 
-instance zero CLI where
-	zero = { version = False
-	       , help    = False
-	       , root    = "/opt/clean/lib/"
-	       , libs    = [ ("ArgEnv",           const id)
-	                   , ("CleanInotify",     const id)
-	                   , ("CleanPrettyPrint", const id)
-	                   , ("CleanSerial",      const id)
-	                   , ("CleanSnappy",      const id)
-	                   , ("CleanTypeUnifier", const id)
-	                   , ("Cloogle",          const id)
-	                   , ("Directory",        const id)
-	                   , ("Dynamics",         const id)
-	                   , ("Gast",             const id)
-	                   , ("GraphCopy",        const id)
-	                   , ("MersenneTwister",  const id)
-	                   , ("ObjectIO",         \s me -> {me & me_is_core=not (startsWith "Std" s)})
-	                   , ("Platform",         const id)
-	                   , ("Sapl",             const id)
-	                   , ("SoccerFun",        const \me -> {me & me_is_app=True})
-	                   , ("StdEnv",           const id)
-	                   , ("StdLib",           const id)
-	                   , ("TCPIP",            const id)
-	                   , ("iTasks",           const id)
-	                   , ("clean-compiler",   const \me -> {me & me_is_app=True})
-	                   , ("clean-ide",        const \me -> {me & me_is_app=True})
-	                   , ("clean-irc",        const id)
-	                   , ("clean-selectloop", const id)
-	                   , ("libcloogle",       const id)
-	                   ]
-	       , exclude = [ "StdEnv/_startup"
-	                   , "StdEnv/_system"
-	                   , "Platform/Deprecated"
-	                   , "Platform/Data/Graph/Inductive/Query"
-	                   , "SoccerFun/RefereeCoach_"
-	                   , "SoccerFun/Team_"
-	                   ]
-	       }
+derive JSONDecode IndexItem, SourceURL, PathPattern
+
+instance zero CLI
+where
+	zero =
+		{ help      = False
+		, root      = "/opt/clean/lib/"
+		, libs_file = "libs.json"
+		}
 
 
-VERSION :== "Cloogle's builddb version 0.1\n"
-USAGE :== concat [
-	VERSION, "\n",
-	"Usage: ./builddb [opts] > types.json\n\n",
-	"\t-h, --help Show this help\n",
-	"\t-r PATH    Change the library root to PATH\n",
-	"\t-l PATH    Add PATH to the librarypaths relative to the root\n"]
+USAGE :== join "\n"
+	[ "Cloogle builddb\n"
+	, "Usage: ./builddb [opts] > types.json\n"
+	, "Options:"
+	, "  --help    Show this help"
+	, "  -r PATH   Change the library root to PATH (default: /opt/clean/lib)"
+	, "  -l PATH   Use PATH for a list of libraries to index (default: libs.json)"
+	, ""]
 
 Start :: *World -> *World
 Start w
@@ -87,8 +63,17 @@ Start w
 	(Left e) = fclose (f <<< e) w
 	(Right cli)
 	| cli.help    = fclose (f <<< USAGE) w
-	| cli.version = fclose (f <<< VERSION) w
-	# (modss, w)  = mapSt (flip (uncurry $ findModules cli.exclude cli.root) "") cli.libs w
+	# (libsf, w)  = readFile cli.libs_file w
+	# libsjson    = fromString $ fromOk libsf
+	# libs        = case libsjson of
+		JSONObject groups -> sequence $ [fromJSON i \\ (_,JSONArray g) <- groups, i <- g]
+		_                 -> Nothing
+	| isError libsf || isNothing libs
+		# err = stderr <<< "Could not read " <<< cli.libs_file <<< "\n"
+		# (_,w) = fclose err w
+		= fclose f w
+	# libs        = fromJust libs
+	# (modss, w)  = mapSt (flip (findModules cli.root) "") libs w
 	# mods        = flatten modss
 	#! (db, w)    = loop cli.root mods newTemporaryDB w
 	#! db         = finaliseDB builtins db
@@ -103,13 +88,16 @@ Start w
 | not ok = abort "Couldn't close stdio"
 = w
 where
-	loop :: String [(String,String,String ModuleEntry -> ModuleEntry)] !TemporaryDB !*World -> *(!TemporaryDB, !*World)
+	loop :: String [ModuleEntry] !TemporaryDB !*World -> *(!TemporaryDB, !*World)
 	loop _ [] db w = (db,w)
-	loop root [(lib,mod,modf):list] db w
-	#! w = snd (fclose (stderr <<< lib <<< ": " <<< mod <<< "\n") w)
-	#! (db, w) = indexModule False root mod lib modf db w
+	loop root [mod:list] db w
+	#! (_, w) = fclose (stderr <<< lib <<< ": " <<< modname <<< "\n") w
+	#! (db, w) = indexModule False root mod db w
 	#! db = hyperstrict db
 	= loop root list db w
+	where
+		lib = fromJust (getLibrary mod.me_loc)
+		modname = getName mod.me_loc
 
 	builtins =
 		map FunctionEntry builtin_functions ++
@@ -123,11 +111,10 @@ where
 	parseCLI [] = Right zero
 	parseCLI [x:a] = case (x,a) of
 		("--help", xs) = (\c->{c & help=True}) <$> parseCLI xs
-		("--version", xs) = (\c->{c & version=True}) <$> parseCLI xs
-		("-l", []) = Left "'-l' requires an argument"
 		("-r", []) = Left "'-r' requires an argument"
 		("-r", [x:xs]) = (\c->{c & root=x}) <$> parseCLI xs
-		("-l", [x:xs]) = (\c->{c & libs=[(x,const id):c.libs]}) <$> parseCLI xs
+		("-l", []) = Left "'-l' requires an argument"
+		("-l", [x:xs]) = (\c->{c & libs_file=x}) <$> parseCLI xs
 		(x, _) = Left $ "Unknown option '" +++ x +++ "'"
 
 	printStats :: !*CloogleDB !*File -> *(*CloogleDB, *File)

docker-compose.yml

 version: '2'
 services:
   backend:
-    build: backend
+    build:
+      context: .
+      dockerfile: backend/Dockerfile
     ports:
       - "31215:31215"
     volumes:
@@ -12,7 +14,9 @@ services:
       - IPC_LOCK
 
   frontend:
-    build: frontend
+    build:
+      context: .
+      dockerfile: frontend/Dockerfile
     depends_on:
       - backend
     ports:

frontend/Dockerfile

@@ -7,9 +7,9 @@ RUN docker-php-source extract \
 	&& docker-php-ext-install sockets mysqli \
 	&& docker-php-source delete
 
-COPY logo.svg /var/www/logo.svg
-COPY birthday.patch /var/www/birthday.patch
-COPY birthday.sh /var/www/birthday.sh
+COPY frontend/logo.svg /var/www/logo.svg
+COPY frontend/birthday.patch /var/www/birthday.patch
+COPY frontend/birthday.sh /var/www/birthday.sh
 RUN cd /var/www \
 	&& apt-get update -qq \
 	&& apt-get install -qq imagemagick librsvg2-bin \
@@ -20,37 +20,25 @@ RUN cd /var/www \
 	&& apt-get autoremove -qq
 
 # Clean libraries, for /src
-RUN apt-get update -qq && apt-get install -qq subversion ca-certificates git
-
-RUN mkdir -p /opt/clean && cd /opt/clean && \
-	curl -sSL ftp://ftp.cs.ru.nl/pub/Clean/builds/linux-x64/clean-bundle-complete-linux-x64-latest.tgz \
-		| tar -xz --strip-components=1
-
-RUN svn checkout https://svn.cs.ru.nl/repos/SoccerFun/src /opt/clean/lib/SoccerFun \
-	&& svn checkout https://svn.cs.ru.nl/repos/clean-libraries/trunk/Libraries/MersenneTwister /opt/clean/lib/MersenneTwister \
-	&& svn checkout https://svn.cs.ru.nl/repos/clean-libraries/trunk/Libraries/ObjectIO/ObjectIO /opt/clean/lib/ObjectIO \
-	&& svn checkout https://svn.cs.ru.nl/repos/clean-ide/trunk /opt/clean/lib/clean-ide \
-	&& svn checkout https://svn.cs.ru.nl/repos/clean-compiler/trunk /opt/clean/lib/clean-compiler \
-	&& git clone https://gitlab.science.ru.nl/mlubbers/CleanSerial /opt/clean/lib/CleanSerial \
-	&& git clone https://github.com/clean-cloogle/Cloogle /opt/clean/lib/Cloogle \
-	&& git clone https://github.com/clean-cloogle/libcloogle /opt/clean/lib/libcloogle \
-	&& git clone https://github.com/clean-cloogle/CleanTypeUnifier /opt/clean/lib/CleanTypeUnifier \
-	&& git clone https://github.com/clean-cloogle/CleanPrettyPrint /opt/clean/lib/CleanPrettyPrint \
-	&& git clone https://github.com/camilstaps/CleanInotify /opt/clean/lib/CleanInotify \
-	&& git clone https://github.com/camilstaps/CleanSnappy /opt/clean/lib/CleanSnappy \
-	&& git clone https://github.com/clean-cloogle/clean-irc /opt/clean/lib/clean-irc \
-	&& git clone https://github.com/dopefishh/clean-selectloop /tmp/clean-selectloop \
-		&& mv /tmp/clean-selectloop/libraries /opt/clean/lib/clean-selectloop \
-	&& rm -r /opt/clean/lib/StdEnv \
-	&& git clone https://github.com/clean-cloogle/StdEnv-doc /opt/clean/lib/StdEnv
+COPY util /tmp/util
+COPY libs.json /var/libs.json
+RUN apt-get update -qq && apt-get install -qq subversion ca-certificates git jq\
+	&& cd /tmp/util\
+	&& ln -s /var/libs.json\
+	&& ./fetch_libs.sh /opt/clean/lib\
+	&& apt-get remove -qq subversion ca-certificates git jq\
+	&& apt-get autoremove -qq
 
 # Pygments, for highlighting
 RUN apt-get update -qq \
 	&& apt-get install -qq python3.5 python3-pip \
 	&& pip3 install pygments pygments-lexer-clean
 
-COPY index_common_problems.py .
-RUN ./index_common_problems.py && rm index_common_problems.py
+COPY frontend/index_common_problems.py .
+RUN apt-get update -qq && apt-get install -qq ca-certificates git\
+	&& ./index_common_problems.py\
+	&& apt-get remove -qq ca-certificates git\
+	&& apt-get autoremove -qq
 
-COPY iconv.sh .
+COPY frontend/iconv.sh .
 RUN bash iconv.sh && rm iconv.sh

frontend/index.html → frontend/index.php

@@ -43,44 +43,33 @@
 				<label><input type="checkbox" id="include-core"/> Include library core modules</label><br/>
 				<label><input type="checkbox" id="include-apps"/> Include apps</label><br/>
 				<table>
-					<tr>
-						<th><a title="Toggle selection" href="javascript:toggleLibSelection('libs-clean-2.4')">Clean 2.4</a></th>
-						<th><a title="Toggle selection" href="javascript:toggleLibSelection('libs-official')">Official</a></th>
-						<th><a title="Toggle selection" href="javascript:toggleLibSelection('libs-misc')">Miscellaneous</a></th>
-					</tr>
-					<tr>
-						<td id="libs-clean-2.4">
-							<label><input type="checkbox" class="search-libs" checked="checked" value="ArgEnv"/> ArgEnv</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="Directory"/> Directory</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="Dynamics"/> Dynamics</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="Gast"/> Gast</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="MersenneTwister"/> MersenneTwister</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="StdEnv"/> StdEnv</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="StdLib"/> StdLib</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="TCPIP"/> TCPIP</label><br/>
-						</td>
-						<td id="libs-official">
-							<label><input type="checkbox" class="search-libs" checked="checked" value="GraphCopy"/> GraphCopy</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="iTasks"/> iTasks</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="ObjectIO"> ObjectIO</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="Platform"/> Platform</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="Sapl"/> Sapl</label><br/>
-						</td>
-						<td id="libs-misc">
-							<label><input type="checkbox" class="search-libs" checked="checked" value="clean-compiler"/> clean-compiler (<abbr title="This requires that 'include apps' is set.">app</abbr>)</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="clean-ide"/> clean-ide (<abbr title="This requires that 'include apps' is set.">app</abbr>)</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="clean-irc"/> clean-irc</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="clean-selectloop"/> clean-selectloop</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="CleanInotify"/> CleanInotify</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="CleanSerial"/> CleanSerial</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="CleanSnappy"/> CleanSnappy</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="CleanTypeUnifier"/> CleanTypeUnifier</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="CleanPrettyPrint"/> CleanPrettyPrint</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="Cloogle"/> Cloogle</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="libcloogle"/> libcloogle</label><br/>
-							<label><input type="checkbox" class="search-libs" checked="checked" value="SoccerFun"> SoccerFun (<abbr title="This requires that 'include apps' is set.">app</abbr>)</label><br/>
-						</td>
-					</tr>
+					<?php
+						function make_group_id($name) {
+							return 'libs-' . str_replace(' ', '-', $name);
+						}
+
+						$groups = json_decode(file_get_contents('/var/libs.json'), true);
+
+						echo '<tr>';
+						foreach (array_keys($groups) as $group) {
+							echo '<th><a title="Toggle selection" href="javascript:toggleLibSelection(\'' . make_group_id($group) . '\')">' . $group . '</a></th>';
+						}
+						echo '</tr>';
+
+						echo '<tr>';
+						foreach ($groups as $group => $libs) {
+							echo '<td id="' . make_group_id($group) . '">';
+							foreach ($libs as $lib) {
+								echo '<label><input type="checkbox" class="search-libs" checked="checked" value="' . $lib['name'] . '"/> ' . $lib['name'];
+								if (isset($lib['pattern_app']) && $lib['pattern_app'] == [['PWildcard']]) {
+									echo ' (<abbr title="Some modules in this library require that \'include apps\' is turned on.">app</abbr>)';
+								}
+								echo '</label><br/>';
+							}
+							echo '</td>';
+						}
+						echo '</tr>';
+					?>
 				</table>
 			</div>
 		</div>

frontend/src/lib.php

@@ -77,48 +77,17 @@ function makeBrowser($dir, $basemodule) {
 	echo '</div>';
 }
 
-$alllibs = [
-	'Clean 2.4' => [
-		'ArgEnv',
-		'Directory',
-		'Dynamics',
-		'Gast',
-		'Generics',
-		'MersenneTwister',
-		'StdEnv',
-		'StdLib',
-		'TCPIP',
-	],
-	'Official' => [
-		'GraphCopy',
-		'iTasks',
-		'ObjectIO',
-		'Platform',
-		'Sapl',
-	],
-	'Miscellaneous' => [
-		'clean-compiler',
-		'clean-ide',
-		'CleanInotify',
-		'CleanPrettyPrint',
-		'CleanSerial',
-		'CleanSnappy',
-		'CleanTypeUnifier',
-		'Cloogle',
-		'libcloogle',
-		'SoccerFun',
-	]
-];
+$groups = json_decode(file_get_contents('/var/libs.json'), true);
 
 echo '<div class="browser">';
-foreach ($alllibs as $group => $libs) {
+foreach ($groups as $group => $libs) {
 	echo '<h4 class="browser-header">' . $group . '</h4>';
 	foreach ($libs as $lib) {
-		echo '<div class="browser-item directory toggle-container" data-name="' . $lib . '">' .
+		echo '<div class="browser-item directory toggle-container" data-name="' . $lib['name'] . '">' .
 				'<span class="toggler">' .
 					'<span class="toggle-icon">&#x229e</span>' .
-					'<span class="title">' . $lib . '</span></span>';
-		makeBrowser(CLEANLIB . '/' . $lib, '');
+					'<span class="title">' . $lib['name'] . '</span></span>';
+		makeBrowser(CLEANLIB . '/' . $lib['name'], '');
 		echo '</div>';
 	}
 }

frontend/src/src.php

@@ -13,6 +13,6 @@ $hl_lines = isset($_REQUEST['line']) ? escapeshellarg($_REQUEST['line']) : '';
 $fname = CLEANHOME . '/lib/' . $mod . '.' . $iclordcl;
 $efname = escapeshellarg($fname);
 
-system("python3 cloogle_pygments.py $efname $hl_lines");
+system("python3.5 cloogle_pygments.py $efname $hl_lines");
 
 endif;

libs.json

+{
+	"Clean 2.4":
+	[
+		{
+			"name": "ArgEnv",
+			"fetch_url": ["CleanDistribution", "ArgEnv"]
+		},
+		{
+			"name": "Directory",
+			"fetch_url": ["CleanDistribution", "Directory"]
+		},
+		{
+			"name": "Dynamics",
+			"fetch_url": ["CleanDistribution", "Dynamics"]
+		},
+		{
+			"name": "Gast",
+			"fetch_url": ["CleanDistribution", "Gast"],
+			"info_url": "https://gitlab.science.ru.nl/clean-and-itasks/gast"
+		},
+		{
+			"name": "MersenneTwister",
+			"fetch_url": ["SVN", "https://svn.cs.ru.nl/repos/clean-libraries"],
+			"path": "trunk/Libraries/MersenneTwister"
+		},
+		{
+			"name": "StdEnv",
+			"fetch_url": ["Git", "https://github.com/clean-cloogle/StdEnv-doc"],
+			"pattern_exclude": [["PStartsWith","_startup"],["PStartsWith","_system"]]
+		},
+		{
+			"name": "StdLib",
+			"fetch_url": ["CleanDistribution", "StdLib"]
+		},
+		{
+			"name": "TCPIP",
+			"fetch_url": ["CleanDistribution", "TCPIP"]
+		}
+	],
+	"Official":
+	[
+		{
+			"name": "GraphCopy",
+			"fetch_url": ["CleanDistribution", "GraphCopy"]
+		},
+		{
+			"name": "iTasks",
+			"fetch_url": ["CleanDistribution", "iTasks"],
+			"info_url": "https://gitlab.science.ru.nl/clean-and-itasks/iTasks-SDK"
+		},
+		{
+			"name": "ObjectIO",
+			"fetch_url": ["SVN", "https://svn.cs.ru.nl/repos/clean-libraries"],
+			"path": "trunk/Libraries/ObjectIO/ObjectIO"
+		},
+		{
+			"name": "Platform",
+			"fetch_url": ["CleanDistribution", "Platform"],
+			"info_url": "https://gitlab.science.ru.nl/clean-and-itasks/clean-platform",
+			"pattern_exclude": [["PStartsWith","Deprecated"]]
+		},
+		{
+			"name": "Sapl",
+			"fetch_url": ["CleanDistribution", "Sapl"],
+			"info_url": "https://gitlab.science.ru.nl/clean-and-itasks/clean-sapl"
+		}
+	],
+	"Miscellaneous":
+	[
+		{
+			"name": "CleanInotify",
+			"fetch_url": ["Git", "https://github.com/camilstaps/CleanInotify"],
+			"info_url": "https://github.com/camilstaps/CleanInotify"
+		},
+		{
+			"name": "CleanPrettyPrint",
+			"fetch_url": ["Git", "https://github.com/clean-cloogle/CleanPrettyPrint"],
+			"info_url": "https://github.com/clean-cloogle/CleanPrettyPrint"
+		},
+		{
+			"name": "CleanSerial",
+			"fetch_url": ["Git", "https://gitlab.science.ru.nl/mlubbers/CleanSerial"],
+			"info_url": "https://gitlab.science.ru.nl/mlubbers/CleanSerial"
+		},
+		{
+			"name": "CleanSnappy",
+			"fetch_url": ["Git", "https://github.com/camilstaps/CleanSnappy"],
+			"info_url": "https://github.com/camilstaps/CleanSnappy"
+		},
+		{
+			"name": "CleanTypeUnifier",
+			"fetch_url": ["Git", "https://github.com/clean-cloogle/CleanTypeUnifier"],
+			"info_url": "https://github.com/clean-cloogle/CleanTypeUnifier"
+		},
+		{
+			"name": "clean-compiler",
+			"fetch_url": ["SVN", "https://svn.cs.ru.nl/repos/clean-compiler"],
+			"path": "trunk",
+			"info_url": "https://svn.cs.ru.nl/repos/clean-compiler/",
+			"pattern_app": [["PWildcard"]]
+		},
+		{
+			"name": "clean-ide",
+			"fetch_url": ["SVN", "https://svn.cs.ru.nl/repos/clean-ide"],
+			"path": "trunk",
+			"info_url": "https://svn.cs.ru.nl/repos/clean-ide/",
+			"pattern_app": [["PWildcard"]]
+		},
+		{
+			"name": "clean-irc",
+			"fetch_url": ["Git", "https://github.com/clean-cloogle/clean-irc"],
+			"info_url": "https://github.com/clean-cloogle/clean-irc"
+		},
+		{
+			"name": "clean-selectloop",
+			"fetch_url": ["Git", "https://github.com/dopefishh/clean-selectloop"],
+			"info_url": "https://github.com/dopefishh/clean-selectloop",
+			"path": "libraries"
+		},
+		{
+			"name": "Cloogle",
+			"fetch_url": ["Git", "https://github.com/clean-cloogle/Cloogle"],
+			"info_url": "https://github.com/clean-cloogle/Cloogle"
+		},
+		{
+			"name": "libcloogle",
+			"fetch_url": ["Git", "https://github.com/clean-cloogle/libcloogle"],
+			"info_url": "https://github.com/clean-cloogle/libcloogle"
+		},
+		{
+			"name": "SoccerFun",
+			"fetch_url": ["SVN", "https://svn.cs.ru.nl/repos/SoccerFun"],
+			"path": "src",
+			"pattern_app": [["PWildcard"]],
+			"pattern_exclude": [["PStartsWith","StdTeam/Team_"],["PStartsWith","StdReferee/RefereeCoach_"]]
+		}
+	]
+}

util/fetch_libs.sh

+#!/bin/bash
+
+DEST="$1"
+
+rm -rf "$DEST"
+mkdir -p "$DEST"
+# Fetch Clean distribution
+curl -sSL ftp://ftp.cs.ru.nl/pub/Clean/builds/linux-x64/clean-bundle-complete-linux-x64-latest.tgz |\
+	tar -xz --exclude=exe -C "$DEST" --strip-components=2 clean-bundle-complete/lib
+
+exec 5< <(jq '.[]' < libs.json | jq '.[]' | jq -cMr '.name,.fetch_url[0,1],.path'