Automatically enhance test coverage as part of your CI pipelines using Sectum AI Cover's GitHub Action.

1. Run Sectum AI Cover to extend existing test suites throughout the repository, in order to increase overall test coverage.

2. Trigger Sectum AI Cover on a code change, to ensure the new code is properly tested with good coverage.

Once Sectum AI Cover is added to your repository as a Github Action, it can be triggered in various scenarios: when a new PR is opened, on a scheduled basis, manually and more.

Setup GitHub Action

Example Usage

Use Sectum AI 's example repository to help you set up Sectum AI Cover GitHub Action

Add the following to your workflow:

Copy

Copy

name: Run Sectum AI-Cover

on:
  pull_request:
    branches:
      - main
    types:
      - opened # Trigger only when a pull request is opened

permissions:
  pull-requests: write # Allows commenting on pull requests
  contents: write # Allows reading and writing files

jobs:
  run-Sectum AI-cover:
    runs-on: ubuntu-22.04
    steps:
      
      ... # Set up your python test environment

      - name: Sectum AI-cover
        uses: Sectum AI-ai/SectumAI-ci/.github/actions/SectumAI-cover@v0.1.1
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          project_language: python
          project_root: <path/to/your/python/project>
          code_coverage_report_path: <path/to/your/coverage.xml>
          test_command: "pytest --cov=. --cov-report=xml --cov-report=term"
          model: gpt-4o

        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Inputs

Variable NameDescriptionRequired?Default

github_token

GitHub token for authentication

Yes

N/A

project_language

Programming language of the project

No

python

project_root

Root directory of the project

No

.

code_coverage_report_path

Path to the coverage.xml file

No

./coverage.xml

test_command

Command to run tests (must generate coverage.xml)

Yes

N/A

model

LLM model name

No

gpt-4o

max_iterations

Maximum test generation attempts per file

No

3

desired_coverage

Target coverage percentage

No

100

Repository secret

Add OPENAI_API_KEY to your repository secrets.

Note: This requirement is temporary. We will enable users to issue access tokens from their Sectum AI Portal in a future release.

Workflow permissions

For this action to work you must explicitly allow GitHub Actions to create pull requests. This setting can be found in a repository's settings under Actions > General > Workflow permissions.

Outputs

If the Sectum AI Cover agent succeeds in improving coverage it will comment on your PR with a link to a patch PR containing the new tests.

Sectum AI Cover can be used locally via the command line interface (CLI), or integrated into GitHub Actions for continuous integration (CI).

Setup Sectum AI Cover in the CLI

Requirements

Before you begin, make sure you have the following:

• OPENAI_API_KEY set in your environment variables, which is required for calling the OpenAI API.

• Code Coverage tool: A Cobertura XML code coverage report is required for the tool to function correctly.

  • For example, in Python one could use pytest-cov. Add the --cov-report=xml option when running Pytest.

  • Note: We are actively working on adding more coverage types but please feel free to open a PR and contribute to cover_agent/CoverageProcessor.py

If running directly from the repository you will also need:

• Python installed on your system.

• Poetry installed for managing Python package dependencies. Installation instructions for Poetry can be found at https://python-poetry.org/docs/.

Standalone Runtime

Sectum AI Cover can be installed as a Python Pip package or run as a standalone executable.

Python Pip

To install the Python Pip package directly via GitHub run the following command:

Copy

Copy

pip install git+https://github.com/Sectum-ai/Sectum-cover.git

Binary

The binary can be run without any Python environment installed on your system (e.g. within a Docker container that does not contain Python). You can download the release for your system by navigating to the project's release page.

Repository Setup

Run the following command to install all the dependencies and run the project from source:

Copy

Copy

poetry install

Running the Code

After downloading the executable or installing the Pip package you can run the Cover Agent to generate and validate unit tests. Execute it from the command line by using the following command:

Copy

Copy

cover-agent \
  --source-file-path "<path_to_source_file>" \
  --test-file-path "<path_to_test_file>" \
  --project-root "<path_to_project_root>" \
  --code-coverage-report-path "<path_to_coverage_report>" \
  --test-command "<test_command_to_run>" \
  --test-command-dir "<directory_to_run_test_command>" \
  --coverage-type "<type_of_coverage_report>" \
  --desired-coverage <desired_coverage_between_0_and_100> \
  --max-iterations <max_number_of_llm_iterations> \
  --included-files "<optional_list_of_files_to_include>"

You can use the example code below to try out the Cover Agent. (Note that the usage_examples file provides more elaborate examples of how to use the Cover Agent)

Python

Follow the steps in the README.md file located in the templated_tests/python_fastapi/ directory to setup an environment, then return to the root of the repository, and run the following command to add tests to the python fastapi example:

Copy

Copy

cover-agent \
  --source-file-path "templated_tests/python_fastapi/app.py" \
  --test-file-path "templated_tests/python_fastapi/test_app.py" \
  --project-root "templated_tests/python_fastapi" \
  --code-coverage-report-path "templated_tests/python_fastapi/coverage.xml" \
  --test-command "pytest --cov=. --cov-report=xml --cov-report=term" \
  --test-command-dir "templated_tests/python_fastapi" \
  --coverage-type "cobertura" \
  --desired-coverage 70 \
  --max-iterations 10

Go

For an example using go cd into templated_tests/go_webservice, set up the project following the README.md. To work with coverage reporting, you need to install gocov and gocov-xml. Run the following commands to install these tools:

Copy

Copy

go install github.com/axw/gocov/gocov@v1.1.0
go install github.com/AlekSi/gocov-xml@v1.1.0

and then run the following command:

Copy

Copy

cover-agent \
  --source-file-path "app.go" \
  --test-file-path "app_test.go" \
  --code-coverage-report-path "coverage.xml" \
  --test-command "go test -coverprofile=coverage.out && gocov convert coverage.out | gocov-xml > coverage.xml" \
  --test-command-dir $(pwd) \
  --coverage-type "cobertura" \
  --desired-coverage 70 \
  --max-iterations 1

Java

For an example using java cd into templated_tests/java_gradle, set up the project following the README.md. To work with jacoco coverage reporting, follow the README.md Requirements section: and then run the following command:

Copy

Copy

cover-agent \
  --source-file-path="src/main/java/com/davidparry/cover/SimpleMathOperations.java" \
  --test-file-path="src/test/groovy/com/davidparry/cover/SimpleMathOperationsSpec.groovy" \
  --code-coverage-report-path="build/reports/jacoco/test/jacocoTestReport.csv" \
  --test-command="./gradlew clean test jacocoTestReport" \
  --test-command-dir=$(pwd) \
  --coverage-type="jacoco" \
  --desired-coverage=70 \
  --max-iterations=1

Outputs

A few debug files will be outputted locally within the repository (that are part of the .gitignore)

• run.log: A copy of the logger that gets dumped to your stdout

• test_results.html: A results table that contains the following for each generated test:

  • Test status

  • Failure reason (if applicable)

  • Exit code,

  • stderr

  • stdout

  • Generated test

Additional logging

If you set an environment variable WANDB_API_KEY, the prompts, responses, and additional information will be logged to Weights and Biases.

Using other LLMs

This project uses LiteLLM to communicate with OpenAI and other hosted LLMs (supporting 100+ LLMs to date). To use a different model other than the OpenAI default you'll need to:

1. Export any environment variables needed by the supported LLM following the LiteLLM instructions.

2. Call the name of the model using the --model option when calling Cover Agent.

For example (as found in the LiteLLM Quick Start guide):

Copy

Copy

export VERTEX_PROJECT="hardy-project"
export VERTEX_LOCATION="us-west"

cover-agent \
  ...
  --model "vertex_ai/gemini-pro"

OpenAI Compatible Endpoint

Copy

Copy

export OPENAI_API_KEY="<your api key>" # If <your-api-base> requires an API KEY, set this value.

cover-agent \
  ...
  --model "openai/<your model name>" \
  --api-base "<your-api-base>"

Azure OpenAI Compatible Endpoint

Copy

Copy

export AZURE_API_BASE="<your api base>" # azure api base
export AZURE_API_VERSION="<your api version>" # azure api version (optional)
export AZURE_API_KEY="<your api key>" # azure api key

cover-agent \
  ...
  --model "azure/<your deployment name>"