Commit 561c5069 authored by benoit's avatar benoit
Browse files

update

parent b27fc252
This diff is collapsed.
Information for developpers
OVERVIEW
Although the sources of camlp5 are written in camlp5 syntaxes, the
distribution contain no 'magic' binary file for the first compilation.
This first compilation can be done thanks to a version of the core of
camlp5 written in pure ocaml syntax. Once this 'core' is compiled, all
sources in revised syntax with syntax extensions can be compiled.
COMPATIBILITY
This distribution of camlp5 is compatible with several versions of
ocaml. The problem is about the definition of ocaml syntax trees which
may change from a version of ocaml to another. Since ocaml does not
install the sources nor the compiled versions of its syntax tree, a
copy of the necessary source files, borrowed from the source of the
ocaml compiler has been done in the directory 'ocaml_stuff', in
subdirectories with the ocaml version number.
If the present distribution of camlp5 is not compatible with the
version of ocaml you have (the command 'configure' tells you), it
is possible to add it. For that, you need the sources of the ocaml
distribution you have. Once done, after a 'configure' telling you
that it is not compatible, do:
make steal OCAML_SRC=<path-to-ocaml-sources>
This creates a new directory in 'ocaml_stuff' with sources of the
syntax tree of your ocaml compiler.
If you want to check that the sources of the syntax tree of ocaml
are up-to-date (e.g. if this is the current ocaml developpement), do:
make compare_stolen OCAML_SRC=<path-to-ocaml-sources>
The compatibility is also done with the file 'lib/versdep.ml',
which is a module containing miscellaneous features depending to
the version of OCaml.
In the directory 'ocaml_src' which contains the pure OCaml sources
of the Camlp5 core (see chapter TREE STRUCTURE below), there are as
many versions of this files as versions of OCaml. They are named
'<version>.ml' in the directory 'lib/versdep'. If you are adding
a new version of OCaml, you need this file. As a first step, make a
copy from a close version:
cd ocaml_src/lib/versdep
cp <close_version>.ml <version>.ml
Then, you can redo "configure" and do "make core". If the file
'ocaml_src/lib/versdep.ml' has a compilation problems, fix them and
to 'make core' again.
Later, the same file 'lib/versdep.ml' in camlp5 syntax may have
similar compilation problem. This file is in one examplary, thanks to
IFDEF used here or there.
While compiling with some specific version of ocaml, this file is
compiled with 'OCAML_vers' defined where 'vers' is the version number
form the beginning to the first space or charcter '+' with all dots
converted into underscores. For example, if your ocaml version is
7.04.2+dev35, you can see in the compilation process of versdep.ml
that OCAML_7_04_2 is defined, and you can add statements defined
by the syntax extension 'pa_macro.cmo', for example IFDEF OCAML_7_04_2.
Add statements like that in 'lib/versdep.ml' to make it compile.
TREE STRUCTURE
The directory 'ocaml_src' contains images in pure ocaml syntax of the
directories odyl lib main and meta. This allow to create a core version
of camlp5 from only the ocaml compiler installed.
You can decompose the building of the camlp5 core into:
1. make library_cold
just makes the directory 'ocaml_src/lib' and copy the cmo and cmi
files into the directory 'boot'
2. make compile_cold
makes the other directories of ocaml_src
3. make promote_cold
copies the executables "camlp5", "camlp5r" and the syntax
extensions (cmo files) into the directory 'boot'
From that point, the core camlp5 is in directory 'boot'. The real
sources in the top directories odyl lib main and meta, which are
written in revised syntax with some syntax extensions (grammars,
quotations) can be compiled. To achieve their compilation, you can
do:
make core
Or to compile everything do:
make all
or just:
make
Notice that doing "make core" or "make all" from scratch (after a
make clean), automatically start by making the core files from their
pure ocaml versions.
FAST COMPILATION FROM SCRATCH
./configure
make clean core compare
make coreboot
make all opt opt.opt
TESTING CHANGES
1. do your changes
2. do:
make core compare
if it says that the bootstrap is ok, you can do:
make all
make opt
make opt.opt
otherwise, to make sure everything is ok, first do:
make coreboot
sometimes two bootstraps ('make coreboot' twice) are necessary,
in particular if you change things in the directory 'lib'. It is
even possible that three bootstraps are necessary.
If things goes wrong, it is possible to return to the previous
version by typing:
make restore clean_hot
then you can change what is necessary and continue by typing:
make core
and test the bootstrap again:
make coreboot
After several bootstraps (by 'make coreboot' or 'make bootstrap'),
many versions are pushed in the directory 'boot' (you can type
'find boot -type d -print' to see that). If your system correctly
bootstraps, you can clean that by typing:
make cleanboot
which keeps only two versions. (The command 'make clean' also removes
these stack of versions.)
BEFORE COMMITTING YOUR CHANGES
Make sure that the cold start with pure ocaml sources work. For that, do:
make compare_sources | less
This shows you the changes that would be done in the ocaml pure sources
of the directory ocaml_src.
To make the new versions, do:
make new_sources
make promote_sources
Notice that these pure ocaml sources are not supposed to be modified
by hand, but only created by the above commands. Besides, their sources,
although pretty printed, are sometimes not easy to read, particularly for
expanded grammars (of the statement 'EXTEND').
However, if these sources do not compile, due to changes in the ocaml
compiler, it is possible to edit them. In this case, similar changes
may have to be done in the normal sources in revised syntax.
After doing 'make new_sources' above, and before doing 'make
promote_sources' below, it is possible to do 'make untouch_sources'
which change the dates of the new created files with the dates of the
old files if they are not modified. This way, the "svn commit" will not
have to compare these files, which may have some importance if you
network is not fast.
The 'make new_sources' builds a directory named 'ocaml_src.new'.
If this directory still exists, due to a previous 'make new_sources',
the command fails. In this case, just delete it (rm -rf ocaml_src.new)
without problem: this directory is not part of the distribution, it is
just temporary.
The 'make clean_sources' deletes old versions of ocaml_src, keeping
only the last and the before last ones.
The command:
make bootstrap_sources
is a shortcut for:
make new_sources
make untouch_sources
make promote_sources
make clean_sources
If there are changes in the specific file 'lib/versdep.ml', do
also:
make compare_all_versdep
and possibly:
make bootstrap_all_versdep
because this file, in 'ocaml_src/lib/versdep' directory has different
versions according to the ocaml version.
After having rebuilt the pure ocaml sources, check they work by
rebuilding everything from scratch, starting with "configure".
IF YOU CHANGE THE MAIN PARSER
If you change the main parser 'meta/pa_r.ml', you may check that the
quotations expanders of syntax tree 'meta/q_MLast.ml' match the new
version. For that, do:
cd meta
make compare_q_MLast
If no differences are displayed, it means that 'q_MLast.ml' is ok,
relatively to 'pa_r.ml'.
Otherwise, if the displayed differences seem reasonable, update the
version by typing:
make bootstrap_q_MLast
Then returning to the top directory, do 'make core compare' and
possibly 'make coreboot' (one of several times) to check the
correctness of the file.
And don't forget, if you want to commit, to re-create the pure ocaml
sources like indicated above.
Internal, very small, undocumented, or invisible changes
********************************************************
Camlp5 Version 6.12:
--------------------
* [20 Jan 14] Removed W3C icon image (in documentation) that should
not be in Camlp5 distribution (violates Debian guidelines, in
particular).
Camlp5 Version 6.02.1:
----------------------
* [05 Dec 01] Fixed pb in 'make bootstrap_sources': failed if it was
not a CVS working directory.
Camlp5 Version 6.02:
--------------------
* [12 Nov 10] In 'make steal' (resp. 'make compare_stolen'), build
(resp. compare) now 'config.ml' with 'utils/config.mlp' instead of
'utils/config.ml' (this file not being existent any more after
a 'make clean' in the OCaml sources).
Camlp5 Version 6.01:
--------------------
Camlp5 Version 6.00:
--------------------
* [02 Sep 10] Added compatibility for OCaml versions 3.06, 3.05, 3.04,
3.03, 3.02, 3.01, 3.00, 2.99, 2.04, 2.03, 2.02, 2.01, 2.00 and 1.07
(but do not work on 64 bits architectures due to OCaml bug about
input_value/output_value of that time: these versions can be used
only on 32 bits architectures). Sometimes (but not always), some of
the very old versions of OCaml (1.07, 2.00, 2.01) do not work, typically
by raising Stack_overflow in the compiler or failing with segmentation
faults in camlp5.
* [28 Aug 10] Re-added module Ast2pt (file main/ast2pt.ml). Created file
lib/versdep.ml and directory ocaml_src/lib/versdep for ocaml versions
dependance.
* [24 Aug 10] Added Makefile.withnew in directory etc for compilations
that must be done with the new created camlp5.
* [23 Aug 10] Changed variable 'NAME' into 'camlp5' in Makefiles and scripts.
* [20 Aug 10] Added compatibility for OCaml version 3.07.
* [18 Aug 10] Removed option "-qmod" (internal usage) from q_MLast.cmo.
* [18 Aug 10] Changed sources to put '#load' instead of normalized comments.
* [12 Aug 10] Added option "-ignloaddir" in "pa_r.ml".
Camlp5 Version 5.14:
---------------------
* [28 Apr 10] Fixed bug in "make boostrap_all_ast2pt".
Camlp5 Version 5.13:
---------------------
* [02 Nov 09] Moved all files ocaml_src/main/ast2pt.ml_<version> into new
subdirectory ocaml_src/main/ast2p as "<version>.ml".
* [2009] Returned repository to CVS (was SVN).
Camlp5 Version 5.08:
---------------------
* [04 Jan 08] Converted "pr_extend.ml", "pr_extfun.ml" and "q_phony.ml" into
usage of "pprintf".
Camlp5 Version 5.07:
---------------------
* [21 Dec 07] Changed all calls to "make" into "$(MAKE)" in Makefiles.
(Some were missing.)
Camlp5 Version 5.06:
---------------------
Camlp5 Version 5.05:
---------------------
* [28 Nov 07] Small change in interface Fstream for backtracking parsers.
Camlp5 Version 5.04:
---------------------
* [26 Nov 07] Added chapter titles in header of documentation in latex.
* [22 Nov 07] Fixed small bug in "make install", displayed an error
while testing the presence of cmx files.
Camlp5 Version 5.03:
---------------------
* [29 Oct 07] Replaced the exception [Grammar.SkipItem] (designed
to make IFDEF in constructor declarations and pattern matching
cases work) by the function [Grammar.skip_item].
Camlp5 Version 5.01:
---------------------
* [06 Oct 2007] Added Prtools.vlistf, Prtools.plistf and Prtools.plistbf,
same as their version without 'f', i.e. resp. Prtools.vlist, Prtools.plist
and Prtools.plistb, but with the list being a list of functions returning
a string allowing to use list of values of heterogen types.
* [03 Oct 2007] Fixed bug in Prtools.plistb: if printing at least two
elements, with the first one fitting in the line, a space was missing.
E.g. using pr_r.cmo with "module M (N : P) (Q : R) = struct end".
* [30 Sep 2007] Cleaned up and simplified q_MLast.ml pa_extend.ml pa_r.ml
while keeping compatibility.
Camlp5 Version 4.07:
---------------------
* [30 Jul 2007] Removed type "Pcaml.err_ctx" and exception "Pcaml.Qerror"
from the interface, since only used in the implementation.
* [28 Jul 2007] Re-added option "-no_slist" of pr_extend.cmo which had
disappeared after rewritting the pretty printing system.
* [28 Jul 2007] Fixed name of symbol in case of use of FOLD0, FOLD0SEP, etc.
and changed "simple_expr" in these syntax into 'expr LEVEL "simple"'.
* Copyright (c) 2007-2017, INRIA (Institut National de Recherches en
* Informatique et Automatique). All rights reserved.
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of INRIA, nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY INRIA AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
* THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
* PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL INRIA AND
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
* USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
* OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
In order to allow users to slowly convert to a new syntax tree,
uncompatible with the previous one, Camlp5 can be compiled in
two modes:
- "transitional": syntax tree is compatible but the new syntax tree
quotations kit in user syntax q_ast.cmo is not available.
- "strict": the new syntax tree quotations kit in user syntax q_ast.cmo
is available but the syntax tree is not compatible with the previous
versions.
You must choose among these modes at configuration time.
See Camlp5 documentation.
# Makefile,v
include config/Makefile
DIRS=lib odyl main meta etc top ocpp man
FDIRS=lib odyl main meta
OPTDIRS=ocaml_stuff lib odyl main meta etc
OPTOPTDIRS=compile
SHELL=/bin/sh
COLD_FILES=ocaml_src/main/argl.ml ocaml_src/main/mLast.mli ocaml_src/main/pcaml.ml ocaml_src/main/pcaml.mli ocaml_src/main/quotation.ml ocaml_src/main/quotation.mli ocaml_src/main/reloc.ml ocaml_src/main/reloc.mli ocaml_src/lib/extfun.ml ocaml_src/lib/extfun.mli ocaml_src/lib/fstream.ml ocaml_src/lib/fstream.mli ocaml_src/lib/gramext.ml ocaml_src/lib/gramext.mli ocaml_src/lib/grammar.ml ocaml_src/lib/grammar.mli ocaml_src/lib/plexer.ml ocaml_src/lib/plexer.mli ocaml_src/lib/stdpp.ml ocaml_src/lib/stdpp.mli ocaml_src/lib/token.ml ocaml_src/lib/token.mli ocaml_src/lib/versdep.ml ocaml_src/meta/pa_extend.ml ocaml_src/meta/pa_extend_m.ml ocaml_src/meta/pa_macro.ml ocaml_src/meta/pa_macro_gram.ml ocaml_src/meta/pa_r.ml ocaml_src/meta/pa_rp.ml ocaml_src/meta/pr_dump.ml ocaml_src/meta/q_MLast.ml ocaml_src/odyl/odyl_main.ml ocaml_src/odyl/odyl_main.mli ocaml_src/odyl/odyl.ml
PR_O=pr_o.cmo
DIFF_OPT=-Bwiu
# For possible installation in a fake root directory
# by "make install DESTDIR=..."
DESTDIR=
all: world.opt
out: boot/$(CAMLP5N)$(EXE)
set -e; cd ocaml_stuff; $(MAKE); cd ..
set -e; for i in $(DIRS); do cd $$i; $(MAKE) all; cd ..; done
opt:
set -e; for i in $(OPTDIRS); do cd $$i; $(MAKE) opt; cd ..; done
opt.opt: opt
cd compile; $(MAKE) opt; cd ..
ocaml_src/lib/versdep.ml:
@echo "Please run 'configure' first"; exit 2
boot/$(CAMLP5N)$(EXE): $(COLD_FILES)
set -e; cd ocaml_stuff; $(MAKE); cd ..
$(MAKE) clean_cold
$(MAKE) library_cold
$(MAKE) compile_cold
$(MAKE) promote_cold
$(MAKE) clean_cold
$(MAKE) clean_hot
$(MAKE) library
clean_hot:
cd ocaml_stuff; $(MAKE) clean; cd ..
for i in $(DIRS) compile; do (cd $$i; $(MAKE) clean; cd ..); done
depend:
cd etc; $(MAKE) pr_depend.cmo; cd ..
cd ocaml_stuff; $(MAKE) depend; cd ..
for i in $(DIRS) compile; do (cd $$i; $(MAKE) depend; cd ..); done
install:
@if test -z "$(LIBDIR)"; then \
echo "*** Variable LIBDIR not set"; exit 1; fi
@if test -z "$(CAMLP5N)"; then \
echo "*** Variable CAMLP5N not set"; exit 1; fi
$(RM) -rf "$(DESTDIR)$(LIBDIR)/$(CAMLP5N)"
for i in $(DIRS) compile; do \
(cd $$i; $(MAKE) install DESTDIR=$(DESTDIR); cd ..); \
done
uninstall:
@if test -z "$(LIBDIR)"; then \
echo "*** Variable LIBDIR not set"; exit 1; fi
@if test -z "$(CAMLP5N)"; then \
echo "*** Variable CAMLP5N not set"; exit 1; fi
$(RM) -rf "$(DESTDIR)$(LIBDIR)/$(CAMLP5N)"
cd "$(DESTDIR)$(BINDIR)"; $(RM) -f *$(CAMLP5N)* odyl ocpp; cd ..
cd "$(DESTDIR)$(MANDIR)/man1"; $(RM) -f *$(CAMLP5N)* odyl ocpp
clean::
$(MAKE) clean_hot clean_cold
$(RM) -f boot/*.cm[oi] boot/$(CAMLP5N)*
$(RM) -rf boot/SAVED
cd test; $(MAKE) clean; cd ..
scratch: clean
always:
# Normal bootstrap
bootstrap:
$(MAKE) backup
$(MAKE) promote
$(MAKE) clean_hot
$(MAKE) out
$(MAKE) compare
backup:
mkdir boot.new
$(MAKE) mv_git FROM=boot TO=boot.new
mv boot boot.new/SAVED
mv boot.new boot
restore:
mv boot/SAVED boot.new
$(MAKE) mv_git FROM=boot TO=boot.new
$(RM) -rf boot
mv boot.new boot
promote:
for i in $(FDIRS); do (cd $$i; $(MAKE) promote; cd ..); done
compare:
@if (for i in $(FDIRS); do \
cd $$i; \
if $(MAKE) compare 2>/dev/null; then cd ..; \
else exit 1; fi; \
done); \
then echo "Fixpoint reached, bootstrap succeeded."; \
else echo "Fixpoint not reached, try one more bootstrapping cycle."; \
fi
compare_test:
@(for i in $(FDIRS); do \
cd $$i; \
if $(MAKE) compare 2>/dev/null; then cd ..; \
else exit 1; fi; \
done)
cleanboot:
$(RM) -rf boot/SAVED/SAVED
# Core and core bootstrap
coreboot:
$(MAKE) backup
$(MAKE) promote
$(MAKE) clean_hot
$(MAKE) core
$(MAKE) compare
core: boot/$(CAMLP5N)$(EXE)
cd ocaml_stuff; $(MAKE) all; cd ..
set -e; for i in $(FDIRS); do cd $$i; $(MAKE) all; cd ..; done
clean_core:
for i in $(FDIRS); do (cd $$i; $(MAKE) clean; cd ..); done
# Everything in one command
world:
$(MAKE) core
$(MAKE) compare_test || $(MAKE) coreboot
$(MAKE) out
world.opt:
$(MAKE) core
$(MAKE) compare_test || $(MAKE) coreboot
$(MAKE) out
$(MAKE) opt
$(MAKE) opt.opt
library:
set -e; cd ocaml_stuff; $(MAKE); cd ..
set -e; cd lib; $(MAKE) all; cd ..
set -e; cd lib; $(MAKE) promote; cd ..
# Cold start using pure Objective Caml sources
library_cold:
set -e; cd ocaml_src/lib; $(MAKE) all; cd ../..
set -e; cd ocaml_src/lib; $(MAKE) promote; cd ../..
compile_cold:
cd ocaml_src; set -e; \
for i in $(FDIRS); do \
cd $$i; $(MAKE) all; cd ..; \
done; cd ..
promote_cold:
for i in $(FDIRS); do \
(cd ocaml_src/$$i; $(MAKE) promote; cd ../..); \
done
clean_cold:
for i in $(FDIRS); do \
(cd ocaml_src/$$i; $(MAKE) clean; cd ../..); \
done
# Stealing some Ocaml compiler sources
steal:
cd ocaml_stuff; $(MAKE) steal; cd ..
compare_stolen:
cd ocaml_stuff; $(MAKE) compare_stolen; cd ..
# Bootstrap the sources
bootstrap_sources:
$(RM) -rf ocaml_src.new
mkdir ocaml_src.new
$(MAKE) new_sources
$(MAKE) untouch_sources
$(MAKE) promote_sources
$(MAKE) clean_sources
bootstrap_source:
$(RM) -rf ocaml_src.new
mkdir -p ocaml_src.new/$$DIR
$(MAKE) new_source DIR=$$DIR FILE=$$FILE
mv ocaml_src.new/$$DIR/$$FILE ocaml_src/$$DIR/$$FILE
rmdir ocaml_src.new/$$DIR
rmdir ocaml_src.new
new_sources: oprinter
@-for i in $(FDIRS); do \
mkdir ocaml_src.new/$$i; \
$(MAKE) $(NO_PR_DIR) new_source DIR=$$i FILE=Makefile; \
echo ============================================; \
echo ocaml_src.new/$$i/.depend; \
(cd ocaml_src.new/$$i; cp ../../$$i/.depend .); \
done
@-mkdir ocaml_src.new/lib/versdep
@-(cd ocaml_src/lib/versdep; \
cp *.ml ../../../ocaml_src.new/lib/versdep/.)
@-mkdir ocaml_src.new/lib/versdep/jocaml
@-(cd ocaml_src/lib/versdep/jocaml; \
cp *.ml ../../../../ocaml_src.new/lib/versdep/jocaml)
@-for i in $(FDIRS); do \
files="$$(cd $$i; ls *.ml*)"; \
for j in $$files; do \
if [ "$$j" != "odyl_config.ml" ]; then \
$(MAKE) $(NO_PR_DIR) new_source DIR=$$i FILE=$$j; \
fi; \
done; \
done
new_source:
@cd $$DIR; k=$$FILE; opt=""; \
if [ "$$k" = "versdep.ml" ]; then \
k=versdep$(VERSDIR)/$(OVERSION).ml; \
VERSDIR="$(OCAMLN)"; \
else \
VERSDIR=""; \
fi; \
if [ "$$k" = "odyl_main.ml" ]; then \
opt="$$opt -DBOOTSTRAPPING_SOURCES"; \
fi; \
echo ============================================; \