Test Runner & Logger Framework

A lightweight, deterministic Bash test execution and reporting framework for executing automated regression tests and generating JUnit‑compatible XML reports.

Overview

This framework provides a minimal, robust infrastructure for running shell‑based test cases and producing machine‑readable test reports.

It is designed for environments where:

• tests are plain executable scripts
• no external test harness is desired
• output must be compatible with CI systems such as Jenkins, GitLab CI, or Azure DevOps
• reproducibility and transparency are required

The framework consists of:

• runner – discovers and executes test cases
• logger – receives structured events and writes a JUnit XML report
• test case API – simple shell functions for reporting objectives, steps, assets, blobs, references, and verdicts

The design avoids hidden magic and keeps all data flow explicit.

Architectur Model

The framework separates responsibilities into thre strictly isolated roles:

• test cases describe behaviour and emit structured events
• the runner controls execution and determines verdicts
• the logger owns the report an guarantees XML consistency
• the logger is the single writer of the XML state
• prevents partial or corrupted reports

Execution Model

Test cases describe behaviour but never control the final result. The runner evaluates lifecycle outcomes and assigns the final verdict, ensuring deterministic and reproducible test results independent of individual testcase implementation.

Features

Deterministic Execution

• Strict ordering of test cases
• No parallelism, no race conditions
• unidirectional FIFO‑based communication to the logger

JUnit XML Output

• Fully valid JUnit XML
• Compatible with Jenkins “Test Result” plugin
• Supports objectives, steps, references, assets, and binary blobs

Blob Support

• Arbitrary text or Base64 data
• Stored inside CDATA sections
• Safe for further XML processing

Clear Separation of Responsibilities

• Test cases only emit actions
• The runner assigns verdicts
• The logger writes the XML report

Minimal Dependencies

• POSIX shell + Bash
• xmlstarlet
• sed

Directory Structure

project/
  tests/
    bin/
      runner
      logger
    testsuite/
      01-testset-ClI/
        t01-example.tcase
        t02-other.tcase
      02-testset-environment/
        t01-validCases.tcase
        t02-invalidCases.tcase

Test cases follow the naming pattern:

File matcing:

  tNN*-name.tcase

are considered testcase and will be executed in sorted order.

Writing Test Cases

A testcase is a normal shell script. It is invoked by the runner and inherits its environment, including the public API

Test cases:

• must not require invocation parameters
• are responsible for preparing the testee for the specific scenario
• should not leave the testee in an undefined state

Public API

The case must return a status word to reflect the verdict of the test run. The following status words are defined:

RC_PASSED : The Testee behaved exacly as expected in all aspects of this testcase.

RC_SKIPPED : The testcase was intenitionally not executed (e.g., missing prerequisites).

RC_ERROR : A technical error occurred, likely outside the responsiblity of the testee (e.g., missing files, syntax error, execution, unstable infrastructure). Such issues should be resolved before evaluating functional correctness.

RC_FAILED : The testee did not behave according to the specification (e.g., an assertian failed). This is considered a functional failure.

Logging API

The logger maintains the XML log file to guarantee consistency at all times. Test cases sends structured events through the FIFO pipe.

log_objective <text> : Must be called once at the beginning. Creates the <testcase> node for this test case. The objective entry is added withing the <system-out> node.

log_description <text> : Adds a <description> element under <system-out> for the current testcase. : Subsequent calls are rejected as LOD violation, produce log_error, and return RC_ERROR, while leaving the execution flow under caller control. : If not provided, the testcase remains valid. : The description text may span multiple lines and may contain arbitrary content including special characters, blank lines, or XML-like fragments. : Encoding (e.g. Base64 or XML-safe transformation) is handled internally by the Runner. The caller must NOT perform any encoding.

: Example (HereDoc usage):

      read -r -d '' text <<'EOF'
      multiline line block

      even more multiline block

      potentially the last multiline block
      EOF

      log_description "$text"

log_reference <text> : Optional, may be called multiple times. Adds a <reference> entry under <references>.

log_precondition <text> : Optional, may be called multiple times. Adds a <precondition> entry under <preconditions>.

log_postcondition <text> : Optional, may be called multiple times. Adds a <postcondition> entry under <postconditions>.

log_step <id> <text> : Optional, may be called multiple times. Creates a <step id="..."> entry directly under<system-out>`.

log_asset <name> <value> : Optional, may be called multiple times. Useful for eposing internal state or vaiable content. Creates an <assert name="..."> entry directly under <system-out>.

log_blob <name> <content> : Optional, may be called multiple times. Content is wrapped in a CDATA section, allowing arbitrary data. Creates a <blob name="..."> entry directly under<system-out>`.

Testee API

load_testee <script>

Loads the specified DOT script into the current shell environment together with all project‑level and framework‑level include scripts.

During this loading phase, the framework guarantees that its global shell options remain unchanged. To enforce this invariant, the framework temporarily disables all attempts by the DOT script to modify shell options:

• For the duration of the import, the shell builtin set is replaced by a no‑op function.
• Any invocation of set … inside the DOT script has no effect on the execution environment.
• After the import completes, the original builtin is fully restored.

This behavior is part of the public API contract. A DOT script must not alter global shell options, and any attempt to do so is deterministically neutralized by the framework.

Extended Return Code Handling (URC Runtime Semantics)

DOT scripts use symbolic return codes generated by the URC system. Each symbolic constant represents a 16‑bit Unique Return Code (URC). Since POSIX shells can only propagate 8‑bit exit values, the framework provides an extended return‑code mechanism that preserves the full 16‑bit URC for testing and debugging.

URC Propagatino Model

A DOT script must return symbolic URC constants such as:

return $RC_someMeaningfulSymbol_1

Each symbolic constant expands to a 16‑bit numeric value in development mode. The framework ensures that:

• the full 16‑bit URC is captured and stored for test evaluation,
• the 8‑bit POSIX return value (URC % 256) is propagated to the caller,
• the symbolic name remains stable across development and deployment.

Overloaded Return Semantics

During development and testing, the shell builtin is replaced by a framework‑provided function with the following semantics:

• It accepts a 16‑bit URC value from the DOT script.
• It stores the full 16‑bit URC in the framework’s state file if it differs from the previously recorded value modulo 256.
• It then performs a POSIX‑compliant return by invoking the builtin with .

This mechanism guarantees that:

• the test framework always observes the exact URC that originated the error path,
• nested returns do not overwrite the stored URC when they merely propagate the same 8‑bit value,
• the DOT script’s control flow remains compatible with POSIX shell semantics.

Short‑Circuit Requirement for DOT Scripts

Because the overloaded return function executes within the caller’s context, DOT scripts must ensure that control flow terminates correctly after invoking a symbolic return. The API therefore requires DOT scripts to use short‑circuit operators:

On error paths:

return $RC_errorCondition_1 || builtin return

On success paths:

return $RC_OK && builtin return

This guarantees that the caller’s function body does not continue execution after a return condition has been triggered.

Deployment Mapping

In deployment mode, symbolic URCs are mapped to the productive return codes used by the released tool. The framework replaces the development mapping with an embedded block of productive values. This transformation:

• removes the short‑circuit cleanup expressions (|| builtin return / && builtin return),
• replaces the development URC source file with an embedded list of productive return codes,
• preserves the symbolic names so that the DOT script remains unchanged.

Example (deployment mapping):

readonly RC_OK=0
readonly RC_mainUnknownOption_1=1
readonly RC_mainNoPlaylist_1=1
...

The deployment mapping is reversible. Restoring the development environment reinstates the symbolic include file and the unique 16‑bit URC values.

API Guarantees

The framework guarantees that:

• all symbolic URCs are available when a DOT script is loaded,
• the correct mapping (development or deployment) is active,
• the full 16‑bit URC is always captured during development,
• the POSIX‑compliant 8‑bit return value is always propagated to the caller,
• DOT scripts can rely on deterministic, collision‑free error signatures.

DOT scripts must:

• use only symbolic URC constants,
• never return numeric literals,
• implement short‑circuit return patterns as described above,
• treat symbolic URCs as part of the stable public API.

Accessing the last 16-bit URC

lastExcCode : Returns the most recently recorded 16‑bit Unique Return Code (URC) from the framework state. The value corresponds to the last symbolic return executed by the DOT script, regardless of the 8‑bit POSIX return value propagated through the call stack.

The function reads the framework’s state object and extracts the field lastExcCode. If no URC has been recorded yet, a sentinel value is returned.

Example implementation (for reference only):

lastExcCode() {
    jq -r '.lastExcCode//9999' <"${__tc_state}"
}

Purpose : This function allows test cases to assert against the full 16‑bit URC, not the truncated 8‑bit POSIX return value. It enables precise validation of execution paths, error conditions, and coverage of symbolic return codes.

Usage in Assertions : A test case may evaluate the last URC using any assertion mechanism. For example:

assertEqual \
    --expected="$RC_meaningFullError_1" \
    --received="$(lastExcCode)" \
    --msg="Houston, we have a Problem" \
    || builtin return

The assertion framework typically returns a symbolic failure code such as RC_FAILED. Test authors may choose to:

• immediately terminate the test case using short‑circuit semantics (|| builtin return), or
• accumulate multiple assertion results and return a failure only after all checks have been executed.

Both patterns are supported and consistent with the API contract.

Interaction with the Overloaded Return Mechanism

The value returned by lastExcCode() always reflects the full 16‑bit URC captured by the overloaded return() function. Nested returns that merely propagate the same 8‑bit value do not overwrite the stored URC, ensuring that the original error source remains identifiable.

Logger Write Modes

The logger supports three write modes that control how often the XML report is written to disk. All modes produce valid JUnit XML; they differ only in granularity and performance characteristics. Write modes affect persistance frequency only, never test semantics.

writeMode=full

The logger writes the XML file after every received action (objective, description, reference, step, asset, blob, verdict). This mode provides the most detailed view of the test execution.

Characteristics:

• maximum transparency
• many write operations
• ideal for debugging individual test cases
• intermediate states are always visible

Typical use cases:

• active debugging
• detailed inspection of complex test logic
• reproducing subtle or timing‑sensitive issues

writeMode=verdict

The logger writes the XML file only when a test case receives its verdict. Intermediate actions within a test case are not persisted.

Characteristics:

• fewer writes than full
• still provides orientation about which test case completed last
• useful for narrowing down failures in larger suites

Typical use cases:

• diagnosing failing or unstable test suites
• identifying the point of failure before switching to full
• triage workflows

writeMode=final

The logger writes the XML file once, after all test cases have finished.

Characteristics:

• minimal I/O
• fastest mode
• produces only the final state of the entire suite
• no intermediate states

Typical use cases:

• CI pipelines (unit tests, check‑ins)
• daily/weekly/regression runs
• stable environments where only the final result matters

Recommendation Table

|---------------------+-------------+------------------------------------|
| Purpose / Scenario  | Recommended | Rational                           |
|                     | Mode        |                                    |
|---------------------+-------------+------------------------------------|
| Debugging a single  | full        | complete visibility of all actions |
| test case           |             |                                    |
|---------------------+-------------+------------------------------------|
| Investigating a     | verdict     | shows the last successfully        |
| failing suite       |             | completed test case                |
|---------------------+-------------+------------------------------------|
| CI / unit tests     | verdict     | balanced overhead, debatable       |
|                     |             | test coverage                      |
|---------------------+-------------+------------------------------------|
| CI regression runs  | final       | minimal overhead, stable environ-  |
|                     |             | ent, final result is sufficiant    |
|---------------------+-------------+------------------------------------|
| After a suite crash | verdict     | first locate teh failure, then     |
|                     |    -> full  | inspect details                    |
|---------------------+-------------+------------------------------------|
| Performance-        | final       | minimal overhead, only one write   |
| critical runs       |             | operation                          |
|---------------------+-------------+------------------------------------|

Typical Workflow

CI run (final) : Fast, produces the final XML report.

Suite fails : Re‑run the suite in verdict mode to identify the failing test case.

Problematic test case found : Re‑run that test case in full mode for complete detail.

This workflow is deterministic, reproducible, and minimizes unnecessary runtime.

Running a TestSuite

A testsuite is simply a directory structure containing .tcase files.

Example:

../bin/runner --onError=continue regression

Options:

--onError=continue : Continue executing testcases even if errors or failures occur. (Default)

--onError=halt : Stop after the first test case reporting ERROR or FAILED

--writeMode={full|verdict|final} : logger persistence mode

Runner Behaviour

The runner automaticlly:

• discovers testcases to run sequentially in alphabetical order
• enforces deterministic execution order
• starts the logger
• executes each testcase
• determine testcase's verdict
• sends verdicts to the logger
• isolates testcase failures from logger integrity
• wait for the logger to finish
• waits for the logger to finish writing the final XML report

Logger Output

The logger produces a JUnit XML file such as (simplified structure):

testsuites/
  testsuite/
    system-out
    system-err
      testcase/
        system-out
        system-err
          objective
          references
            reference
          preconditions
            precondition
          postconditions
            postcondition
          steps
          assets
          blobs (CDATA)

Design Principles

• All communication is explicit
• Failures are visible and reproducible
• Output is deterministic

The framework is intentionally small, explicit and transparent, making it suitable for long-term maintenance and reproducible regression testing of command-line tools.

Logger Implementations

The framework provides two Logger implementations:

Bash Logger (Reference Implementation)

The Bash Logger serves as a transparent reference model demonstrating the Logger architecture and event protocol.

Characteristics:

• prioritizes readability and transparency
• invokes xmlstarlet for each event
• optimized for understanding, debugging, and experimentation

Due to the per-event XML processing cost, write modes (full, verdict, final) do not provide meaningful performance differences.

The Bash Logger should be considered a reference implementation, not a production logger.

Python Logger (Production Implementation)

The Python Logger is optimized for performance and large test suites.

Characteristics:

• persistent in-memory state model
• efficient incremental XML generation
• full support for Logger write modes

Recommended for CI environments and large regression runs.