move task documentation into task directories & refresh wiki automati…

…cally

.github/workflows/build-master.yml

@@ -35,6 +35,43 @@ jobs:
       DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
       DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
 
+  update_project_wiki:
+    name: Generate project documentation
+    needs: [check_source]
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout base code
+      uses: actions/checkout@v2
+      with:
+        path: code
+    - name: Checkout wiki code
+      uses: actions/checkout@v2
+      with:
+        repository: ${{github.repository}}.wiki
+        path: wiki
+    - name: Generate wiki from docs
+      run: |
+        touch ./wiki.md
+
+        for filename in code/docs/*.md; do
+          while IFS= read -r line; do
+          if [[ "$line" =~ ^"#!!" ]]; then
+            bash -c "cd code && ${line:3}" >> ./wiki.md
+          else
+            echo "$line" >> ./wiki.md
+          fi
+          done <<< $(cat $filename)
+        done
+
+        cp ./wiki.md ./wiki/Home.md
+    - name: Push to wiki
+      run: |
+        cd wiki
+        git config --local user.email "[email protected]"
+        git config --local user.name "GitHub Action"
+        git add .
+        git diff-index --quiet HEAD || git commit -m "Add changes" && git push
+
   create_snapshot_release:
     name: Create snapshot release
     needs: [build_binaries]

docs/01-setup.md

@@ -0,0 +1,71 @@
+# Installing Assertoor
+
+## Use Executable from Release
+
+Assertoor provides distribution-specific executables for Windows, Linux, and macOS.
+
+1. **Download the Latest Release**:\
+   Navigate to the [Releases](https://github.com/ethpandaops/assertoor/releases) page and download the latest version suitable for your operating system.
+
+2. **Run the Executable**:\
+   After downloading, run the executable with a test configuration file. The command will be similar to the following:
+    ```
+    ./assertoor --config=./test-config.yaml
+    ```
+
+
+## Build from Source
+
+If you prefer to build Assertoor from source, ensure you have [Go](https://go.dev/) `>= 1.21` and Make installed on your machine. Assertoor is tested on Debian, but it should work on other operating systems as well.
+
+1. **Clone the Repository**:\
+	Use the following commands to clone the Assertoor repository and navigate to its directory:
+    ```
+    git clone https://github.com/ethpandaops/assertoor.git
+    cd assertoor
+    ```
+2. **Build the Tool**:\
+	Compile the source code by running:
+	```
+    make build
+    ```
+	After building, the `assertoor` executable can be found in the `bin` folder.
+
+3. **Run Assertoor**:\
+	Execute Assertoor with a test configuration file:
+
+    ```
+    ./bin/assertoor --config=./test-config.yaml
+    ```
+
+## Use Docker Image
+
+Assertoor also offers a Docker image, which can be found at [ethpandaops/assertoor on Docker Hub](https://hub.docker.com/r/ethpandaops/assertoor).
+
+**Available Tags**:
+
+- `latest`: The latest stable release.
+- `v1.0.0`: Version-specific images for all releases.
+- `master`: The latest `master` branch version (automatically built).
+- `master-xxxxxxx`: Commit-specific builds for each `master` commit (automatically built).
+
+**Running with Docker**:
+
+* **Start the Container**:\
+To run Assertoor in a Docker container with your test configuration, use the following command:
+
+  ```
+  docker run -d --name=assertoor -v $(pwd):/config -p 8080:8080 -it ethpandaops/assertoor:latest --config=/config/test-config.yaml
+  ```
+
+* **View Logs**:\
+To follow the container's logs, use:
+  ```
+  docker logs assertoor --follow
+  ```
+
+* **Stop the Container**:\
+To stop and remove the Assertoor container, execute:
+  ```
+  docker rm -f assertoor
+  ```

docs/02-global-config.md

@@ -0,0 +1,77 @@
+# Configuring Assertoor
+
+The Assertoor test configuration file is a crucial component that outlines the structure and parameters of your tests. \
+It contains general settings, endpoints, validator names, global variables, and a list of tests to run.
+
+## Structure of the Assertoor Configuration File
+
+The configuration file is structured as follows:
+
+```yaml
+endpoints:
+  - name: "node-1"
+    executionUrl: "http://127.0.0.1:8545"
+    consensusUrl: "http://127.0.0.1:5052"
+
+web:
+  server:
+    host: "0.0.0.0"
+    port: 8080
+  frontend:
+    enabled: true
+
+validatorNames:
+  inventoryYaml: "./validator-names.yaml"
+  inventoryUrl: "https://config.dencun-devnet-12.ethpandaops.io/api/v1/nodes/validator-ranges"
+  inventory:
+    "0-199": "lighthouse-geth-1"
+    "200-399": "teku-besu-1"
+
+globalVars:
+  validatorPairNames:
+  - "lighthouse-geth-.*"
+  - "teku-besu-.*"
+
+tests:
+# test test1
+- name: "Test with local tasks"
+  timeout: 48h
+  config: {}
+  tasks: []
+  cleanupTasks: []
+
+externalTests:
+- file: ./check-block-proposals2.yaml
+  name: "Test with separate list of tasks"
+  timeout: 48h
+  config: {}
+
+```
+
+
+- **`endpoints`**:\
+  A list of Ethereum consensus and execution clients. Each endpoint includes URLs for both RPC endpoints and a name for reference in subsequent tests.
+
+- **`web`**:\
+  Configurations for the web frontend, detailing server host and port settings.
+
+- **`validatorNames`**:\
+  Defines a mapping of validator index ranges to their respective names. \
+  This mapping can be defined directly in the configuration file, imported from an external YAML file, or fetched from a specified URL. \
+  These named validator ranges can be referenced in tests for targeted actions and checks.
+
+- **`globalVars`**:\
+  A collection of global variables that are made available to all tests and tasks. \
+  These variables can be used to maintain consistency and reusability across different test scenarios. \
+  For an in-depth explanation, see the "Variables" section of the documentation.
+
+- **`tests`**:\
+  A list of tests, each with a specific set of tasks. \
+  Every test is identified by a unique name and may include an optional execution timeout. \
+  The `config` section within each test allows for the specification of additional variables, which are passed down to the tasks within that test. \
+  Moreover, a suite of cleanup tasks can be defined for each test, ensuring orderly and thorough execution, regardless of the test outcome. \
+  Detailed guidelines on task configuration can be found in the "Task Configuration" section of the documentation.
+
+- **`externalTests`**:\
+  This feature enables the integration of tests that are defined in separate files, fostering a modular and scalable test configuration approach. \
+  It allows for better organization and management of complex testing scenarios.

docs/03-task-config.md

@@ -0,0 +1,91 @@
+# Task Configuration
+
+Each task in Assertoor is defined with specific parameters to control its execution. Here's a breakdown of how to configure a task:
+
+## Task Structure
+
+A typical task in Assertoor is structured as follows:
+
+```yaml
+name: generate_transaction
+title: "Send test transaction"
+timeout: 5m
+config:
+  targetAddress: "..."
+  # ...
+configVars:
+  privateKey: "walletPrivateKey"
+  # ...
+ ```
+
+- **`name`**:\
+  The action that the task should perform. It must correspond to a valid name from the list of supported tasks. \
+  This can be a simple action like running a shell script (`run_shell`) or a more complex operation (`generate_transaction`). \
+  For a comprehensive list of supported tasks, refer to the "Supported Tasks" section of this documentation.
+
+- **`title`**:\
+  A descriptive title for the task. It should be unique and meaningful, as it appears on the web frontend and in the logs.\
+  The title helps in identifying and differentiating tasks.
+
+- **`timeout`**:\
+  An optional parameter specifying the maximum duration for task execution. \
+  If the task exceeds this timeout, it is cancelled and marked as a failure. This parameter helps in managing task execution time and resources.
+
+- **`config`**:\
+  A set of specific settings required for running the task. \
+  The available settings vary depending on the task name and are detailed in the respective task documentation section.
+
+- **`configVars`**:\
+  This parameter allows for the use of variables in the task configuration. \
+  In the given example, the value of `walletPrivateKey` is assigned to the `privateKey` config setting of the task. \
+  To make this work, the `walletPrivateKey` variable must be defined in a higher scope (globalVars / test config) or set by a previous task (effectively allowing reusing results from these tasks).\
+  This feature enables dynamic configuration based on predefined or dynamically set variables.
+
+With this structure, Assertoor tasks can be precisely defined and tailored to fit various testing scenarios.\
+Some tasks allow defining subtasks within their configuration, which enables nesting and concurrent execution of tasks.\
+The next sections will detail the supported tasks and how to effectively utilize the `config` parameters.
+
+# Supported Tasks in Assertoor
+
+Tasks in Assertoor are fundamental building blocks for constructing comprehensive and dynamic test scenarios. They are designed to be small, logical components that can be combined and configured to meet various testing requirements. Understanding the nature of these tasks, their states, results, and categories is key to effectively using Assertoor.
+
+## Task States and Results
+
+When a task is executed, it goes through different states and yields specific results:
+
+**Task States**
+
+1. **`pending`**: The initial state of a task before execution.
+2. **`running`**: The state when a task is actively being executed by the task scheduler.
+3. **`completed`**: The final state indicating that a task has finished its execution (regardless of success or failure).
+
+**Task Results**
+
+The result of a task indicates its outcome and can change at any point during its execution:
+
+1. **`none`**: The initial result status of a task, indicating that no definitive outcome has been determined yet.
+2. **`success`**: Indicates that the task has achieved its intended objective. \
+This result can be set at any point during task execution and does not necessarily imply the task has completed.
+3. **`failure`**: Indicates that the task encountered an issue or did not meet the specified conditions. \
+Similar to `success`, this result can be updated during the task's execution and does not automatically mean the task has completed.
+
+It's important to note that while the task result can be updated at any time during execution, it becomes final once the task completes. After a task has reached its completion state, the result is conclusive and represents the definitive outcome of that task.
+
+## Task Categories
+
+Tasks in Assertoor are generally categorized based on their primary function and usage in the testing process. It's important to note that this categorization is not strict. While many tasks fit into these categories, some tasks may overlap across multiple categories or not fit into any specific category. However, the majority of currently supported tasks in Assertoor follow this schema:
+
+1. **Flow Tasks (`run_task` prefix)**: 
+   These tasks are central to structuring and ordering the execution of other tasks. Flow tasks can define subtasks in their configuration, executing them according to specific logic, such as sequential/concurrent or matrix-based execution. They remain in the `running` state while any of their subtasks are active, and move to `completed` once all subtasks are done. The result of flow tasks is derived from the results of the subtasks, following the flow task's logic. This means the overall outcome of a flow task depends on how its subtasks perform and interact according to the predefined flow.
+
+2. **Check Tasks (`check_` prefix)**: 
+   Check tasks are designed to monitor the network and verify certain conditions or states. They typically run continuously, updating their result as they progress. A check task completes by its own only when its result is conclusive and won’t change in the future.
+
+3. **Generate Tasks (`generate_` prefix)**: 
+   These tasks perform actions on the network, such as sending transactions or executing deposits/exits. Generate tasks remain in the `running` state while performing their designated actions and move to `completed` upon finishing their tasks.
+
+The categorization serves as a general guide to understanding the nature and purpose of different tasks within Assertoor. As the tool evolves, new tasks may be introduced that further expand or blend these categories, offering enhanced flexibility and functionality in network testing.
+
+The following sections will detail individual tasks within these categories, providing insights into their specific functions, configurations, and use cases.
+
+#!! cat pkg/coordinator/tasks/*/README.md

pkg/coordinator/tasks/check_clients_are_healthy/README.md

@@ -0,0 +1,39 @@
+## `check_clients_are_healthy` Task
+
+### Description
+The `check_clients_are_healthy` task is designed to ensure the health of specified clients. It verifies if the clients are reachable and synchronized on the same network.
+
+### Configuration Parameters
+
+- **`clientNamePatterns`**:\
+  An array of endpoint names to be checked. If left empty, the task checks all clients. Use this to target specific clients for health checks.
+
+- **`pollInterval`**:\
+  The interval at which the health check is performed. Set this to define how frequently the task should check the clients' health.
+
+- **`skipConsensusCheck`**:\
+  A boolean value that, when set to `true`, skips the health check for consensus clients. Useful if you only want to focus on execution clients.
+
+- **`skipExecutionCheck`**:\
+  A boolean value that, when set to `true`, skips the health check for execution clients. Use this to exclusively check the health of consensus clients.
+
+- **`expectUnhealthy`**:\
+  A boolean value that inverts the expected result of the health check. When `true`, the task succeeds if the clients are not ready or unhealthy. This can be useful in test scenarios where client unavailability is expected or being tested.
+
+- **`minClientCount`**:\
+  The minimum number of clients that must match the `clientNamePatterns` and pass the health checks for the task to succeed. A value of 0 indicates that all matching clients need to pass the health check. Use this to set a threshold for the number of healthy clients required by your test scenario.
+
+### Defaults
+
+These are the default settings for the `check_clients_are_healthy` task:
+
+```yaml
+- name: check_clients_are_healthy
+  config:
+    clientNamePatterns: []
+    pollInterval: 5s
+    skipConsensusCheck: false
+    skipExecutionCheck: false
+    expectUnhealthy: false
+    minClientCount: 0
+ ```

pkg/coordinator/tasks/check_consensus_attestation_stats/README.md

@@ -0,0 +1,48 @@
+## `check_consensus_attestation_stats` Task
+
+### Description
+The `check_consensus_attestation_stats` task is designed to monitor attestation voting statistics on the consensus chain, ensuring that voting patterns align with specified criteria.
+
+### Configuration Parameters
+
+- **`minTargetPercent`**:\
+  The minimum percentage of correct target votes per checked epoch required for the task to succeed. The range is 0-100%.
+
+- **`maxTargetPercent`**:\
+  The maximum allowable percentage of correct target votes per checked epoch for the task to succeed. The range is 0-100%.
+
+- **`minHeadPercent`**:\
+  The minimum percentage of correct head votes per checked epoch needed for the task to succeed. The range is 0-100%.
+
+- **`maxHeadPercent`**:\
+  The maximum allowable percentage of correct head votes per checked epoch for the task to succeed. The range is 0-100%.
+
+- **`minTotalPercent`**:\
+  The minimum overall voting participation per checked epoch in percent needed for the task to succeed. The range is 0-100%.
+
+- **`maxTotalPercent`**:\
+  The maximum allowable overall voting participation per checked epoch for the task to succeed. The range is 0-100%.
+
+- **`failOnCheckMiss`**:\
+  Determines whether the task should stop with a failure result if a checked epoch does not meet the specified voting ranges. \
+  If `false`, the task continues checking subsequent epochs until it succeeds or times out.
+
+- **`minCheckedEpochs`**:\
+  The minimum number of consecutive epochs that must pass the check for the task to succeed.
+
+### Defaults
+
+These are the default settings for the `check_consensus_attestation_stats` task:
+
+```yaml
+- name: check_consensus_attestation_stats
+  config:
+    minTargetPercent: 0
+    maxTargetPercent: 100
+    minHeadPercent: 0
+    maxHeadPercent: 100
+    minTotalPercent: 0
+    maxTotalPercent: 100
+    failOnCheckMiss: false
+    minCheckedEpochs: 1
+```