TestEvents.dcl 2.92 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11
definition module Testing.TestEvents
/**
 * This module provides types representing events occurring during a test run.
 * Each Clean testing framework should report such events on StdOut, as JSON
 * representation of the event types provided here. The test runners
 * (https://gitlab.science.ru.nl/clean-and-itasks/clean-test) process the
 * events further.
 */

from Text.JSON import generic JSONEncode, generic JSONDecode, :: JSONNode, :: Maybe

12 13 14 15 16 17 18 19 20
/**
 * Events that are emitted from tests.
 * Specialised JSONEncode/JSONDecode instances are used for this type, which
 * have to be adapted in case the type definition is changed!
 */
:: TestEvent
	= StartEvent StartEvent //* A test has started
	| EndEvent   EndEvent   //* A test has finished

21 22
/**
 * Event emitted when a test is started.
23 24
 * Specialised JSONEncode/JSONDecode instances are used for this type, which
 * have to be adapted in case the type definition is changed!
25
 */
26
:: StartEvent = { name    :: !String //* The test's name
27 28 29 30
                }
/**
 * Event emitted after a test has finished.
 */
31 32 33
:: EndEvent   = { name    :: !String       //* The test's name
                , event   :: !EndEventType //* The event's type, indicating success
                , message :: !String       //* Message providing an explanation for the result
34 35 36 37 38 39 40 41
                }

/**
 * Indicating the type an end event, indicating whether the test was
 * successful or not.
 * Specialised JSONEncode/JSONDecode instances are used for this type, which
 * have to be adapted in case the type definition is changed!
 */
42 43 44 45
:: EndEventType = Passed            //* The test passed
                | Failed FailReason //* The test failed
                | Skipped           //* The test was not executed, but should be executed and pass for future versions
                | Lost              //* The test crashed
46

Camil Staps's avatar
Camil Staps committed
47 48 49 50
/**
 * Reasons for failing a test.
 */
:: FailReason
51 52 53
	= FailedAssertions [FailedAssertion]    //* Assertions that caused the test to fail
	| CounterExamples [CounterExample]      //* Example values for which the test failed
	| FailedChildren [(String, FailReason)] //* Subtests failed; the tuples are of name and failing reason
Camil Staps's avatar
Camil Staps committed
54

Camil Staps's avatar
Camil Staps committed
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81
/**
 * A counter-example to a test.
 */
:: CounterExample =
	{ counterExample   :: !JSONNode          //* The value that disproves the property
	, failedAssertions :: ![FailedAssertion] //* The assertions that failed in testing the property for that value
	}

/**
 * A failed test assertion.
 */
:: FailedAssertion
	= ExpectedRelation JSONNode Relation JSONNode //* A relation test failed

/**
 * A relation between two values.
 */
:: Relation
	= Eq //* Equality
	| Ne //* Negated equality
	| Lt //* Lesser than
	| Le //* Lesser than or equal to
	| Gt //* Greater than
	| Ge //* Greater than or equal to

derive JSONEncode TestEvent, StartEvent, EndEvent, FailReason, CounterExample, FailedAssertion, Relation
derive JSONDecode TestEvent, StartEvent, EndEvent, FailReason, CounterExample, FailedAssertion, Relation