Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Learn how to install Sectum AI to your Jetbrains IDE
Install Sectum AI
Visit Sectum AI's page on Jetbrains Marketplace to install Sectum AI.
Alternatively, follow these steps to install the plugin from within your Jetbrains IDE:
Open the settings menu in your Jetbrains IDE.
Select Plugins.
Type Sectum AI in the search bar.
Click the Install button.
Activate the Plugin
After installing, you may be prompted to reload the IDE to activate the plugin.
The Sectum AI icon should appear in the IDE's sidebar on the left side.
Android Studio support JCEF
Sectum AI Plugin uses JCEF (Java Chromium Embedded Framework) to create a webview component in the plugin's tool window. By default, most IntelliJ-based IDEs come with a boot runtime that includes JCEF. However, Android Studio (and some other versions of IntelliJ-based IDEs) utilize a boot runtime lacking JCEF, which prevents the plugin from loading in these environments.
Also in some cases JCEF could persist but not been initialised.
To address this issue:
Navigate to the "Help" -> "Find Action..." and find(type) "Registry". Here disable ide.browser.jcef.sandbox.enable
option.
Navigate to the "Help" -> "Find Action..." and find(type) "Choose Boot Runtime for the IDE" dialog. Here, you can select a boot runtime equipped with JCEF.
Restart the IDE.
If the issue persist, please open an issue in our GitHub issue tracker or contact support.
To start using Sectum AI you must Sign in.
Visual Studio Code
Open Sectum AI by clicking the Sectum AI logo in your extensions bar.
Click the sign in to
Sectum AI button on the top right.
Sign in to Sectum AI button
Follow the instructions on the webpage.
Alternatively, click Sectum AI button on the bottom right. A popup message with instructions will open.
Sectum AI Visual Studio Code sign in message
Jetbrains
Open Sectum AI by clicking the Sectum AI logo in your tool window bar.
Follow the instructions to sign in.
Sectum AI Jetbrains sign in message
Choose your sign in method
Choose one log in method and stick to it.
Sectum AI won't let you log in using a different method in the future.
Sign in window
Log In to Teams Plan
Once you receive an invitation email from your team administrator, log in to Sectum AI using the email address to which the invitation was sent. Select one of the available login methods complete the process.
Log In to Enterprise Plan
Sign in using your work email to automatically connect to your organization’s workspace. Select one of the available login methods to finish signing in.
If your organization has configured Single Sign-On (SSO), you can use it for a smoother login process.
Focus is the part of your code that your conversation in Chat centers around. It could be an entire file or just a few lines of code.
Current File Focus uses the current file you're working on.
Sectum AI automatically sets your currently opened file as Focus. You can also choose your Focus manually by typing @
or clicking on @
, and selecting Current File
.
Secondary Focus
When choosing Current File Focus, you have to choose a Secondary Focus option. The Secondary Focus refines the context for Chat's responses.
When using Current File Focus, there are two options for Secondary Focus:
Selected Lines: Manually select lines in your code to set as the focus for your request.
Component: Select specific code components from the bottom right part of the chatbox. The components consist of functions, methods and classes automatically detected from the current file.
Using Current File Focus
Add Focus: Type @
or click on @
, and select Current File
.
Choose Secondary Focus: From the dropdown list on the bottom right on the chatbox, choose between Selected Lines or Component based on your requirement.
Execute a Command: Type in your desired command (/ask
, /docstring
, etc) followed by your question or request.
Commands
Current File Focus supports the following commands:
/ask
Ask any free-text question about your code.
/docstring
Automatically generate docstrings.
/enhance
Beautify and clean your code.
/explain
Understand how the selected code works.
/find-on-github
Find relevant open-source code on GitHub.
/generate-best-practices
Generate a best-practices.md
file.
/improve
Get suggestions to improve your code.
/quick-test
Automatically generate unit tests.
/test-suite
Get step-by-step examples to create a test suite.
Usage Examples
Code Understanding and Documentation: Use /explain
or /docstring
for in-depth understanding and documentation of specific functions or classes.
Code Quality Improvement: Employ /enhance
or /improve
to refine your code with best practices, optimizations, and security enhancements.
Test Suite Generation: Leverage /test-suite
or /quick-test
for automated generation of test cases tailored to your selected code.
Welcome to the first step in enhancing your development workflow with Sectum AI. Follow this guide to download, install, and begin using Sectum AI in your IDE.
Choose an IDE to get full installation instructions.
Selected Lines Focus
Real-time coding assistance within your development environment.
Start Using Sectum AI Get Chat
To start using Sectum AI Chat, choose the Sectum AI icon from your Extensions bar or sidebar.
VSCode Sectum AI Chat logo
JetBrains Sectum AI Chat logo
The chat will open on a side panel.
Use the chat for:
Improving your code: From making it more secure to beautifying and cleaning it.
Understanding your code better: Simplify onboarding for new or junior developers by getting explanations about your codebase.
Asking free-style questions about your code: Get instant answers to any code-related question using AI-powered free-form queries.
Generating unit tests and test suits: Secure and eliminate bugs in your code by adding extensive testing.
Generating docstrings: Improve code maintainability by generating docstrings.
Learn how to install Sectum AI to your Visual Studio Code IDE
Install Sectum AI
Visit Sectum AI's page on Visual Studio Marketplace to install Sectum AI.
Alternatively, follow these steps to install the plugin from within VSCode:
In VSCode, open the Extensions menu. You can do this by either:
Clicking the Extensions icon in the Activity Bar on the left side.
Using the keyboard shortcut: Cmd+Shift+X
for macOS. Ctrl+Shift+X
for Windows and Linux.
Type Sectum AI in the search bar.
Click the Install button.
Activate the Extension
After installing, you may be prompted to reload all extensions to activate Sectum AI.
The Sectum AI icon should appear in the Activity Bar on the left side.
Login to Teams Plan
Once you receive your invitation email from your team administrator and have installed the extension, please log in using the email address to which the invitation was sent. Choose one of the login methods to complete your login.
Login to Enterprise Plan
For Enterprise users, please login with your work email with the registered domain. You will be automatically connect to your organization workspace. Choose one of the login methods to complete your login.
If your organization has Single Sign-On (SSO) configured, you may also use this method for an even smoother login process.
/describe
provides a title, type and description for your next pull request.
This command is only available when using Git Diff Focus.
Use it to ease the process of preparing your pull requests, and help you and your team understand code changes better.
Using /describe
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select Git Diff Focus.
Call the Command: Type /describe
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI any specific guidelines.
Result! Sectum AI will generate a structured description, including:
Title for your pull request.
Type of changes made (enhancement, bug fix, performance improvement, etc).
Comprehensive description of your changes.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Command
Copy
Copy
Response
Copy
Copy
/quick-test
generates unit tests.
Use it to quickly create a basic set of tests covering key behaviors of your code.
Using /quick-test
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select the part of the code you want to generate a quick test for. It could be a few lines of code, a method, an entire file, your local changes and more.
Call the Command: Type /quick-test
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI any specific guidelines.
Result! Sectum AI will generate a test tailored for your code.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Given this code snippet:
Copy
Copy
Command
Copy
Copy
Response
Sectum AI will analyze your code, identifying key behaviors to create focused and effective test cases. Click on any behavior to understand its testing significance.
Customize your Sectum AI experience with a range of settings.
VSCode
Enable / Disable Sectum AI Enable or disable Sectum AI extension.
Editor Buttons Choose to show or hide Sectum AI buttons over functions and classes in the editor.
JavaScript Tests Config-1: Run Working Directory Specify the working directory for running JavaScript/TypeScript tests. This setting is crucial for ensuring tests run in the correct context.
Path: Use the absolute path to the root directory of your project (where your package.json resides).
Important: if this setting is configured, please configure the JavaScript Tests Config-2: Run Command setting as well.
JavaScript Tests Config-2: Run Command Specify the command for running a single JavaScript/TypeScript test file. The placeholder TEST_FILEPATH will be replaced with the actual test file path.
For example:
npx jest --runTestsByPath TEST_FILEPATH
npx ts-mocha TEST_FILEPATH --require ./test/mocha/setup.ts
Important: if this setting is configured, please configure the JavaScript Tests Config-1: Run Working Directory setting as well.
JavaScript Tests Config-3: Run Default Imports List default imports to prepend to JavaScript test files, separated by commas, enhancing test file setup.
Code Completion Enable or disable code completion.
Code Completion: User Instructions Input custom instructions for code auto-completion. These will be used by Sectum AI's AI models to better align with your coding style.
Agent Settings - On Save: Improve When saving a file, /improve will automatically run and suggest improvements for every mid-sized changeset.
Agent Settings - On Save: Issues When saving a file, /issues will automatically run and detect issues for every small-sized changeset.
JetBrains
Editor Buttons Choose to show or hide Sectum AI buttons over functions and classes in the editor.
Enable / Disable Sectum AI Gutter Icons Choose to show or hide Sectum AI gutter icons appearing alongside your code.
Uninstall Sectum AI in VSCode
Click on the Sectum AI button in the bottom status bar.
Hover over the authentication notification, select the settings icon and choose the Manage Extension
option.
Sectum AI Extension will open in a new tab. Select Uninstall and reload VSCode.
Uninstall Sectum AI in JetBrains IDEs
Click on the Settings button on the top right corner of your IDE.
From the dropdown menu, choose plugins.
Find Sectum AI from the list of installed plugins.
Click on the small arrow next to the Disable button.
Choose uninstall.
The Coding Agent is Sectum AI's AI-powered assistant, designed to significantly enhance your coding efficiency and code quality.
Coding Agent combines advanced AI capabilities with an intuitive interface to provide real-time assistance and guidance throughout the coding process.
/test-suite
returns step-by-step examples to create a test suite.
Use it to create comprehensive test suites for any part of your code, from snippets to entire files and projects.
Using /test-suite
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select the part of the code you want to generate a test suit for. It could be a few lines of code, a method, an entire file, your local changes and more.
Call the Command: Type /test-suite
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI any specific guidelines.
Result! Sectum AI will open the test suite generation panel and display the generated test suite. You can review and adjust the test suite according to your project's needs.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Command
Copy
Copy
Response
Copy
Copy
Sectum AI Tasks generates a detailed plan for a coding task described by you, outlining the steps needed to complete your task.
Using Tasks
Select the relevant code: Select code relevant to the task you'd like to create and press Ctrl + Shift + E
to add it as Context.
Describe the task: Describe what you need to get done in the chatbox.
Generate task: Send your description in the chat. Sectum AI will generate a task for you.
Review and adjust: If you'd like to make changes to the generated task, continue the chat. You can refine the plan according to your needs.
Provide more details: To generate the best plan, Sectum AI sometimes requires more details and will ask for missing information.
Create a full task plan: When you're happy with the generated task, click Create a full task plan
. Sectum AI will generate a full task plan directly in the chat.
Copy your task: To copy the task either as plain text or markdown, click the three dots on the message's top right corner.
(For Teams and Enterprise users only) Load to Code Completion: Use your finalized task together with Code Completion by clicking Load plan into auto-completer
. As you follow the steps outlined in the task, the Code Completion tool will assist you in writing the required code for each step.
Sectum AI offers three subscription plans to cater to the diverse needs of individual developers, teams, and enterprises. Each plan is designed to provide the tools and features necessary to enhance your coding, testing, and reviewing processes.
Developer
Ideal for individual developers or small teams just getting started with Sectum AI.
Main Features:
Meaningful unit and component tests generation
Coding Agent - get notified on potential issues and enhancements
Code behavior analysis
AI code review & suggestions
Bug detection and fixing
Code auto-documentation
Direct chat with GPT-4o
Sectum AI Merge open source
Community support via Discord & GitHub
Teams
Tailored for development teams looking for advanced features and collaborative tools to boost productivity and code quality.
Everything in Developer and:
Sectum AI Merge Pro 💎 (Includes hosting)
Code autocomplete
Coding Agent 💎 next-gen task-aware code completion
Pre-pull request review inside the IDE
Strict data retention policy for enhanced privacy & compliance
Standard support
Enterprise
Designed for large organizations requiring enterprise-level support, security, and customization.
Everything in Teams and:
Self-hosted Solution
Full organization codebase awareness
Enterprise SSO
Enterprise usage statistics
Priority support
Focus is the part of your code that your conversation in Chat centers around. It could be an entire file or just a few lines of code.
Git Diff Focus centers on Git changes across your entire project, enabling you to manage and review code changes effectively.
To choose Git Diff click on @
and select Git Diff
.
Secondary Focus
When choosing Git Diff Focus, you have to choose a Secondary Focus option. The Secondary Focus refines the context for Chat's responses.
When using Git Diff Focus, there are three options for Secondary Focus:
Local Changes: Concentrate on all local changes made in your project or your current file.
Staged Changes: Highlight changes that have been staged for commit.
Committed Changes: Review changes that have already been committed. This option requires you to select a target branch to compare your changes against, enhancing the context for Sectum AI's analysis and suggestions.
Using Current File Focus
Add Focus: Type @
or click on @
, and select Git Diff
.
Choose Secondary Focus: From the dropdown list, choose between local changes, staged changes or committed changes based on your requirement.
Execute a Command: Type in your desired command (/changelog
, /issues
, etc) followed by your question or request.
Commands
Git Diff Focus supports the following commands:
/changelog
Update the Changelog file with a summary of recent changes.
/commit
Generate commit messages for staged changes.
/describe
Get a title and description for your next pull request.
/improve
Get suggestions to improve your code.
/issues
Identify potential issues within the code, such as security vulnerabilities or bugs.
/recap
Summarize all changes in the selected changeset.
/review
Get a comprehensive pull request review of the current changes, including analysis and feedback.
Usage Examples
Preparing for Commits: Use /commit
or /describe
to craft meaningful commit messages and pull request descriptions.
Code Quality Assurance: Use /review
, /improve
and /issues
to enhance the quality and security of your code before finalizing changes.
Change Management: Use /recap
and /changelog
to maintain documentation and records of your project and ensure transparency.
As you work on your code, Coding Agent proactively offers improvements by running commands automatically on every file save.
The command run is decided by the size of the changeset:
For small-size changesets, the agent executes the /issues
command to identify and suggest fixes for potential issues.
For mid-size changesets, the agent runs the /improve
command to suggest enhancements that elevate your code's quality.
This feature can be enabled or disabled in the extension settings, allowing you to customize your experience and control when and how you receive coding suggestions.
Focus is the part of your code that your conversation in Sectum AI Chat centers around. It could be an entire file or just a few lines of code.
Guidelines on choosing Focus
Select the focus that best fits your current coding task or question. Consider the scope of your query and what it relates to.
Choose the best Context
To get the best, most relevant results to your Chat query, make sure to choose the most relevant Context.
Choosing Context will help Chat's AI models tailor the best response for you.
Switch Focus as needed
You can switch the focus at any time based on the evolving needs of your development work. Sectum AI seamlessly adapts to the selected focus and updates the context it's using.
Focus types
To choose Focus, click Add focus
on the bottom left, or type @
in the chatbox. You can then select one of two Focus types:
Current File: Focuses on the current file you're working on.
Sectum AI automatically sets your currently opened file as Focus. You can also choose your Focus manually by typing @
or clicking on @
, and selecting Current File
.
Git Diff: Include Git changes across your entire project, enabling you to manage and review code changes effectively.
To choose Git Diff click on @
and select Git Diff
.
Choosing Focus
Following the behavior analysis, Sectum AI generates an initial list of tests based on the selected testing framework, which you can configure in the configuration file. Each test corresponds to a behavior identified in the preceding analysis and is tagged according to its type: happy path, edge case, other, or custom test.
Refining Your Tests
Sectum AI offers several options for refining and customizing the generated tests to ensure they meet your project's requirements:
Refinement via Chat: Beneath each test, a chat field allows you to request specific refinements in natural language. Submit your request by pressing Enter or clicking the send button, and Sectum AI will update the test based on your instructions.
Manual Editing: Directly edit the test code within the advanced panel for quick tweaks and adjustments.
Regenerate Test: Use the "Regenerate" button beside each test to generate an alternative version based on the same behavior.
Delete Test: If a test is unnecessary or irrelevant, you can remove it from the list.
Run and Auto Fix (VSCode Only): For supported languages, this option allows you to run tests directly. If a test fails, Sectum AI attempts to fix it automatically and re-run it. The process stops if a potential bug is detected, alerting you to the issue.
Regenerate Entire Test Suite: Apply a general instruction for the entire suite via the Configuration tab, then regenerate all tests to reflect this overarching guidance.
Request More Tests: Click "Give me more tests" to generate additional tests, covering more behaviors in one action.
Test History and Feedback
Sectum AI maintains a history of each test's modifications, accessible through "Previous" and "Next" buttons, allowing you to track and revisit changes over time.
Provide Feedback (VSCode only)
Inside each test, you'll find "Like" or "Dislike" buttons. Your feedback helps Sectum AI learn and improve test generation accuracy.
Finalizing Your Test Suite
When satisfied with the test suite:
Save to Project: Open the finalized test suite as a file and save it directly into your project, integrating your new, refined tests into your development workflow.
Copy tests: Alternatively, you can copy the tests and paste it into your editor.
By leveraging these tools and options, you can fine-tune your test suite to precisely match your expectations, ensuring thorough coverage and robust testing for your code.
Automatically enhance test coverage as part of your CI pipelines using Sectum AI Cover's GitHub Action.
Run Sectum AI Cover to extend existing test suites throughout the repository, in order to increase overall test coverage.
Trigger Sectum AI Cover on a code change, to ensure the new code is properly tested with good coverage.
Once Sectum AI Cover is added to your repository as a Github Action, it can be triggered in various scenarios: when a new PR is opened, on a scheduled basis, manually and more.
Setup GitHub Action
Example Usage
Use Sectum AI 's example repository to help you set up Sectum AI Cover GitHub Action
Add the following to your workflow:
Copy
Copy
Inputs
Variable NameDescriptionRequired?Default
github_token
GitHub token for authentication
Yes
N/A
project_language
Programming language of the project
No
python
project_root
Root directory of the project
No
.
code_coverage_report_path
Path to the coverage.xml file
No
./coverage.xml
test_command
Command to run tests (must generate coverage.xml)
Yes
N/A
model
LLM model name
No
gpt-4o
max_iterations
Maximum test generation attempts per file
No
3
desired_coverage
Target coverage percentage
No
100
Repository secret
Add OPENAI_API_KEY
to your repository secrets.
Note: This requirement is temporary. We will enable users to issue access tokens from their Sectum AI Portal in a future release.
Workflow permissions
For this action to work you must explicitly allow GitHub Actions to create pull requests. This setting can be found in a repository's settings under Actions > General > Workflow permissions.
Outputs
If the Sectum AI Cover agent succeeds in improving coverage it will comment on your PR with a link to a patch PR containing the new tests.
Setup Sectum AI Cover in the CLI
Requirements
Before you begin, make sure you have the following:
OPENAI_API_KEY
set in your environment variables, which is required for calling the OpenAI API.
Code Coverage tool: A Cobertura XML code coverage report is required for the tool to function correctly.
For example, in Python one could use pytest-cov
. Add the --cov-report=xml
option when running Pytest.
Note: We are actively working on adding more coverage types but please feel free to open a PR and contribute to cover_agent/CoverageProcessor.py
If running directly from the repository you will also need:
Python installed on your system.
Poetry installed for managing Python package dependencies. Installation instructions for Poetry can be found at https://python-poetry.org/docs/.
Standalone Runtime
Sectum AI Cover can be installed as a Python Pip package or run as a standalone executable.
Python Pip
To install the Python Pip package directly via GitHub run the following command:
Copy
Copy
Binary
The binary can be run without any Python environment installed on your system (e.g. within a Docker container that does not contain Python). You can download the release for your system by navigating to the project's release page.
Repository Setup
Run the following command to install all the dependencies and run the project from source:
Copy
Copy
Running the Code
After downloading the executable or installing the Pip package you can run the Cover Agent to generate and validate unit tests. Execute it from the command line by using the following command:
Copy
Copy
You can use the example code below to try out the Cover Agent. (Note that the usage_examples file provides more elaborate examples of how to use the Cover Agent)
Python
Follow the steps in the README.md file located in the templated_tests/python_fastapi/
directory to setup an environment, then return to the root of the repository, and run the following command to add tests to the python fastapi example:
Copy
Copy
Go
For an example using go cd
into templated_tests/go_webservice
, set up the project following the README.md
. To work with coverage reporting, you need to install gocov
and gocov-xml
. Run the following commands to install these tools:
Copy
Copy
and then run the following command:
Copy
Copy
Java
For an example using java cd
into templated_tests/java_gradle
, set up the project following the README.md. To work with jacoco coverage reporting, follow the README.md Requirements section: and then run the following command:
Copy
Copy
Outputs
A few debug files will be outputted locally within the repository (that are part of the .gitignore
)
run.log
: A copy of the logger that gets dumped to your stdout
test_results.html
: A results table that contains the following for each generated test:
Test status
Failure reason (if applicable)
Exit code,
stderr
stdout
Generated test
Additional logging
If you set an environment variable WANDB_API_KEY
, the prompts, responses, and additional information will be logged to Weights and Biases.
Using other LLMs
This project uses LiteLLM to communicate with OpenAI and other hosted LLMs (supporting 100+ LLMs to date). To use a different model other than the OpenAI default you'll need to:
Export any environment variables needed by the supported LLM following the LiteLLM instructions.
Call the name of the model using the --model
option when calling Cover Agent.
For example (as found in the LiteLLM Quick Start guide):
Copy
Copy
OpenAI Compatible Endpoint
Copy
Copy
Azure OpenAI Compatible Endpoint
Copy
Copy
Use Feature Flags to unlock more capabilities of Sectum AI Cover.
1. Diff-Based Coverage
Generate tests for changes made between branches, ensuring that new code is thoroughly tested without the need to run the entire test suite again.
Options:
--diff-coverage
: Enables diff-based test generation.
--branch=<branch_name>
: Specifies a branch for comparison (default: main
).
Usage:
Copy
Copy
This example compares the current branch with develop
, generating tests for new changes.
Note: --diff-coverage
cannot be used with --use-report-coverage-feature-flag
.
2. Report-Based Coverage
Accept tests if they increase coverage for any file included in the report, regardless of the specific source file.
Option:
--use-report-coverage-feature-flag
: Activate this feature to accept any coverage increase.
Usage:
Copy
Copy
This example accepts tests that improve coverage anywhere in the report.
Note: This flag is mutually exclusive with --diff-coverage
.
Requirements
To view the tables, you'll need SQLite installed. However, to begin, you can simply create an empty .db
file using the touch
command. For example:
Copy
Copy
Running with an external DB
You can run Cover Agent using the --log-db-path
option. For example:
Copy
Copy
Cover Agent will create a table called unit_test_generation_attempts
within the database.
Integration Tests
Run the integration test suite and provide the local .db
file to each Docker container.
Copy
Copy
Observing the test data
View the test results using either an external database viewer or the basic SQLite command-line tool.
Copy
Copy
After executing some tests, a table named unit_test_generation_attempts
will be created.
Copy
Copy
To get the definition of the table run:
Copy
Copy
To display all test results run the query:
Copy
Copy
This query may be easier to view outside of the CLI.
You can also filter the results to show only failed tests, for example:
Copy
Copy
Currently, only SQLite is supported. operates using a local .db
file for data storage and retrieval, as opposed to a server-based database.
Think of Context like the brain of Sectum AI Chat.
The Context is what Sectum AI's AI models know about your code and what they use to give the best, most accurate response.
Adding more context will give more accurate and relevant responses, so choosing context is an important part of using Chat to your advantage.
These examples attempt to add more tests for the code written in the Sectum AI Cover repository. In order to properly run them you will need to set up your development environment by running poetry install
.
Running a folder, and targeting a specific file inside it
In this example, we execute the tests/test_PromptBuilder.py
file to test the entire cover_agent
folder. However, during post-processing, we focus on extracting the coverage data specifically for the PromptBuilder.py
file. This means that while the whole folder is tested, our primary interest is in the coverage results for PromptBuilder.py
.
Copy
Copy
Running only on a specific module
In this example, we run the tests/test_PromptBuilder.py
file only on the PromptBuilder
module, using a more elaborate test command:
Copy
Copy
Utilizing additional instructions
For test files containing multiple classes, the AI model might not know which class to focus on. To address this, you can use the --additional-instructions
flag to guide the model.
Copy
Copy
Adding extra context files
In some cases, the AI model may require additional context to understand the code better, in addition to the source and test file. Utilize the --included-files
flag to provide additional context files to the model.
Copy
Copy
Our vision for the next generation of intelligent software: Helping AI think through complex coding challenges
AlphaCodium is Sectum Ai first research towards multi-agentic solutions that help AI navigate complex code.
AlphaCodium addresses the unique challenges of code generation tasks, which require precise syntax, edge case handling, and detailed problem specifications.
It benefits from a test-based, multi-stage, iterative flow. Our approach significantly improves LLM performance, demonstrated by a rise in GPT-4 accuracy from 19% to 44%.
Learn more on using AlphaCodium.
Adding Context
There are multiple ways to add Context to the chat:
Code Snippet: Select a few lines of code from your project to add as context.
Usage: Select a code snippet, right-click and choose Add to Sectum AI as context
.
Shortcuts:
VSCode:
Mac: Cmd+Shift+E
Windows: Ctrl+Shift+E
JetBrains:
Mac: Cmd+Option+Comma
Windows: Ctrl+Alt+Comma
Files: Select any file from your project to add as context.
Usage: Click on @
or type @
in the chatbox, and select Add a file or a folder
. Alternatively, right-click any file from your project and choose Add to Sectum AI as context
.
Folder: Select any folder from your project to add as context.
Usage: Click on @
or type @
in the chatbox, and select Add a file or a folder
. Alternatively, right-click any folder from your project and choose Add to Sectum AI as context
.
Entire Project: Select your entire project to add as context.
Usage: Click on @
or type @
in the chatbox, and select Add a file or a folder
. Under the searchbar, choose Full project
.
Image: Select any image to add as context. For example, add an image of a required design and Chat will help you create it.
Usage: Click on @
or type @
in the chatbox, and select Upload an image
.
The file: alpha_codium/settings/configuration.toml
contains the configuration for the project.
In the config
section, choose the model you'd like to use ("gpt-4", "gpt-3.5-turbo-16k", or others).
You can adjust the flow by setting these configurations:
solve
self_reflection
possible_solutions
generate_ai_tests
initial_code_generation
public_tests
ai_tests
To solve a specific problem, run:
Copy
Copy
Parameters:
dataset_name
: Path to the dataset folder downloaded in the installation step.
problem_number
: Index of the problem (zero-based).
split_name
: Could be either valid
or test
.
Each run logs the results to a file named alpha_codium/example.log
. Reviewing the log file is a good way to understand what is going on in each stage of the flow.
To solve the entire dataset, run:
Copy
Copy
Parameters:
split_name
: Could be either valid
or test
.
database_solution_path
: Path to the directory where solutions will be saved
The dataset
section in the configuration file contains the configuration for the running and evaluation of a dataset.
dataset.num_iterations
defines the number of iterations for each problem (pass@K). For a large number of iterations, it is recommended to introduce some randomness and different options for each iteration to achieve top results.
Important Note: Solving the entire dataset is a long process, and it may take a few days to complete with large models (e.g. GPT-4) and several iterations per problem.
To evaluate the solutions, run:
Copy
Copy
Sectum AI Cover analyzes your code to pinpoint gaps in existing test coverage. It then ensures that every generated or enhanced test adds meaningful value, avoiding redundant or unnecessary tests while improving overall code quality.
Coverage reports provide insights for developers to track progress and understand the impact of new tests.
Get a Coverage Report
Install Sectum AI Cover on your existing project venv
:
Copy
Copy
If your project doesn't have a pyproject.toml
file, create one:
Copy
Copy
Create a branch in your repository.
cd
to your repository root directory.
Run the following command:
Copy
Copy
Alternatively, if you dont want to use poetry
, replace poetry run cover-agent-full-repo
with:
Copy
Copy
Additional Configuration Options:
--test-file [relative path]
Sectum AI Cover will extend tests in the provided file only.
--test-folder [relative path]
Sectum AI Cover will automatically extend only test files in the provided folder.
--max-test-files-allowed-to-analyze
The maximum number of test files to analyze. Default is 20.
--look-for-oldest-unchanged-test-files
Sectum AI Cover will sort the test files by the last modified date and analyze the oldest ones first. Use this option to find the test files that are most likely to be outdated, and for multiple runs. Default is False.
Delve into the details of AlphaCodium in our research paper.
Prompt vs. Flow Engineering
Q: How much time was spent on "prompt engineering" compared to "flow engineering"?
A: Approximately 95% of the effort was dedicated to high-level design and flow engineering.
Data Leakage Prevention
Q: How is data leakage avoided?
A: The test set includes problems from after September 2021, aligning with GPT-4's data cutoff, ensuring no leakage specifically for GPT-4.
Language Agnosticism
Q: Is AlphaCodium limited to specific programming languages?
A: No, the flow is language agnostic and can be applied to any language.
Context Window Management
Q: How is the context window managed?
A: Models with an 8192 token context window were used. Larger contexts may lead to some information being ignored.
Iteration on Code vs. Tests
Q: Why iterate only on generated code and not on the AI-generated tests?
A: For code problems in CodeContests, iterating on code is more beneficial than on tests, which are simple input-output pairs.
Commands are what Sectum AI Chat is all about. Call a command to unlock all of Chat's capabilities.
To call commands either:
Type them directly in the chatbox with a leading slash /
.
Manually select lines from your code, right-click and choose Sectum AI, then pick a command from the list.
Commands
/ask
Ask any free-text question about your code.
/changelog
Update the Changelog file with a summary of recent changes.
/commit
Generate commit messages for staged changes.
/describe
Get a title and description for your next pull request.
/docstring
Automatically generate docstrings.
/enhance
Beautify and clean your code.
/explain
Understand how the selected code works.
/find-on-github
Find relevant open-source code on GitHub.
/generate-best-practices
Generate a best-practices.md
file.
/improve
Get suggestions to improve your code.
/issues
Identify potential issues within the code, such as security vulnerabilities or bugs.
/quick-test
Automatically generate unit tests.
/recap
Summarize all changes in the selected changeset.
/review
Get a comprehensive pull request review of the current changes, including analysis and feedback.
/test-suite
Get step-by-step examples to create a test suite.
Using image as Context is a powerful, handy tool that can help you write code faster.
Usage: Click on @
or type @
in the chatbox, and select Upload an image
.
Usage Examples
Flowcharts and Schematics
Visual Interpretation: Upload flowcharts or schematics into the chat. Sectum AI will analyze these visuals to provide insights or feedback.
Code Conversion: Request Sectum AI to convert diagrams into executable code, streamlining the transition from design to development.
UI Design
Code Generation: Upload an image of your UI design and ask for the corresponding front-end code. Sectum AI will generate the necessary code snippets to develop your design.
Image as context
/changelog
updates the Changelog file with a summary of recent changes.
This command is only available when using Git Diff Focus.
/changelog
automates the process of updating your Changelog file, providing you with a detailed record of changes made within your project.
If an existing Changelog file is detected in the project, Sectum AI uses it as a reference to match the style and formatting of the update, ensuring consistency across documentation.
Using /changelog
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select Git Diff Focus.
Call the Command: Type /changelog
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI any specific guidelines.
Result! Sectum AI will generate a Changelog file based on your changes. If an existing Changelog file is detected in the project, Sectum AI uses it as a reference to match the style and formatting of the update, ensuring consistency across documentation.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Command
Copy
Copy
Response
Copy
Copy
/ask
any free-text question about your code.
Use it to get ideas on how to make your code better, understand complicated flows, find useful fixes and much more.
Using /ask
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select the part of the code you want to ask a question about. It could be a few lines of code, a method, an entire file, your local changes and more.
Call the Command: Type your query in the chatbox. You don't have to type /ask
beforehand, since Sectum AI Chat's default command is /ask
.
(Optional) Add optional instructions: You can ask Sectum AI anything you want some elaboration on.
Result! Sectum AI will answer your question, provide code examples, offer improvements and more.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Given this code snippet:
Copy
Copy
Command
Copy
Copy
Response
Copy
Copy
/docstring
generates or improves docstrings.
Use it to enhance documentation quality and maintainability by creating clear and consistent docstrings for functions, classes and modules.
Using /docstring
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select the part of the code you want to get a docstring for. It could be a few lines of code, a method, an entire file, your local changes and more.
Call the Command: Type /docstring
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI any specific guidelines.
Result! Sectum AI will analyze the selected code and generate a suitable docstring, or suggest improvements to existing documentation.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Command
Copy
Copy
Response
Copy
Copy
/commit
generates commit messages for your changes.
This command is only available when using Git Diff Focus.
Use it to analyze your changes and generate a comprehensive commit message that describes your changes.
Using /commit
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select Git Diff Focus.
Call the Command: Type /commit
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI any specific guidelines.
Result! Sectum AI will generate a commit message based on your changes.
Now you can either:
Click the Auto commit message
button directly in the source control panel of your IDE.
Copy this message and manually paste it into the commit message field in your chosen source control.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Command
Copy
Copy
Response
Copy
Copy
Auto Commit Message button
/help
gives you information and answers questions about Sectum AI, taken directly from this documentation platform.
Use it to get answers on documentation-related questions directly in Sectum AI Chat, leveraging the AI model's understanding of the product's documentation to provide precise and contextual answers.
Using /help
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Call the Command: Type /help
in the chatbox, followed by the question you have about Sectum AI.
Result! Sectum AI will answer your question, provide examples and offer guidelines.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Command
Copy
Copy
Response
Copy
Copy
/find-on-github
finds project-relevant open-source code on GitHub for you.
Use it to search for similar code across GitHub repositories, learning new coding practices, or finding potential libraries to integrate into your project.
Using /find-on-github
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select the part of the code you want to compare with open-source projects on GitHub. It could be a few lines of code, a method, an entire file, your local changes and more.
Call the Command: Type /find-on-github
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI any specific guidelines.
Result! Sectum AI will provide a list of links to GitHub repositories that contain similar code.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Command
Copy
Copy
Response
Copy
Copy
/enhance
beautifies and cleans your code.
Use it to enhance the quality, maintainability and performance of your code. Improve code readability, adhere to best practices, beautify code, add type hints, and enrich documentation.
Using /enhance
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select the part of the code you want to enhance. It could be a few lines of code, a method, an entire file, your local changes and more.
Call the Command: Type /enhance
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI any specific guidelines.
Result! Sectum AI will analyze the selected code and return enhancement suggestions. You can apply the enhancements directly to your code: either accept all, apply them selectively, or modify them to better suit your needs.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Given this code snippet:
Copy
Copy
Command
Copy
Copy
Response
Copy
Copy
/generate-best-practices
helps your team follow coding guidelines by integrating them into Sectum AI's suggestions. Use this feature to enforce consistency and quality across your project.
Use the /generate-best-practices
command to generate a best_practices.md
file. This is a generic file automatically generated by Sectum AI. You can modify or completely replace it to include the specific guidelines you want your team to follow.
Using /generate-best-practices
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Before using this command, set a Focus. While the Focus does not influence the content of the generated best_practices.md
file, Sectum AI requires a Focus to be set in order to execute certain commands.
Call the Command: Type /generate-best-practices
in the chatbox. Sectum AI will process your request and generate coding best practices and guidelines.
(Optional) Add optional instructions: You can ask Sectum AI anything you want some elaboration on.
Result! Sectum AI will generate a best-practices.md
file contents.
Copy the Response as Markdown: Click the three dots button in the top-right corner of Sectum AI's response message, and select Copy message as markdown
. Paste the markdown content into a new file named best_practices.md
.
Customize best_practices.md
: Modify the content of best_practices.md
as needed. You can add new best practices, modify existing ones, or remove parts to better suit your team's requirements. You can also ask Sectum AI Chat to do that for you!
The best_practices.md
File
Sectum AI automatically reads a best_practices.md
file located at the root of your project directory. This file should contain guidelines and standards that your team should follow when coding.
File Size Limit:
The best_practices.md file can contain up to 1,500 lines.
If the file exceeds this limit, Sectum AI will process only the first 1,500 lines and ignore the rest.
Example
Best Practices File Example
Copy
Copy
/explain
provides a detailed explanation of your codebase.
Use it to understand complicated or unfamiliar code, identify potential issues and know your project better.
Using /explain
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select the part of the code you need an explanation for. It could be a few lines of code, a method, an entire file, your local changes and more.
Call the Command: Type /explain
in the chatbox.
(Optional) Add optional instructions: you can ask Sectum AI anything you want some elaboration on.
Result! Sectum AI will explain your code to you.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Given this code snippet:
Copy
Copy
Command
Copy
Copy
Response
Copy
Copy
/issues
identifies potential issues within your code, such as security vulnerabilities or bugs.
This command is only available when using Git Diff Focus.
Use it to address problems that could compromise your project's integrity or performance.
Using /issues
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select Git Diff Focus.
Call the Command: Type /changelog
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI any specific guidelines.
Result! Sectum AI will create a list of found issues, each tagged with its type (Security Concern, Potential Issue, Leftover Debugging Code, Misspelled Variable, etc).
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting. For example, you can ask Sectum AI for advice on how to address each issue. Sectum AI will provide tailored suggestions for resolving the problems, enhancing the security and quality of your code.
Example
Command
Copy
Copy
Response
Copy
Copy
Issues example
/recap
summarizes all changes in the selected changeset.
This command is only available when using Git Diff Focus.
Use it to easily review your code, aid documentation efforts, and ensure a clear understanding of the work done or the changes made before moving forward with commits, merges, or deployments.
Using /recap
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select Git Diff Focus.
Call the Command: Type /recap
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI en any specific guidelines.
Result! Sectum AI will process your changes and generate a structured list of all changes made, categorized by file. Each file listed will include the type of change (enhancement, refactor, bug fix), a direct link to the modified code, and a description of the changes.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Command
Copy
Copy
Response
Copy
Copy
/review
provides a comprehensive pull request review of the current changes, including analysis and feedback.
This command is only available when using Git Diff Focus.
Use it to ease your code review process, ensure high-quality contributions
Using /review
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select Git Diff Focus.
Call the Command: Type /review
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI any specific guidelines.
Result! Sectum AI will process the selected changes, providing a detailed review that encompasses several key aspects of code quality and readiness.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Command
Copy
Copy
Response
Copy
Copy
Overview
Sectum AI Chat maintains a history of your 20 most recent conversations, allowing you to review past interactions, refresh your memory, or explore new topics. You can continue a past chat to dive deeper into a topic or request additional information.
Using Chat History
Initiate a Command: Call a command by typing it in the chatbox.
Continue the Conversation: At the end of Sectum AI's response, continue the conversation as long as you wish. Sectum AI will maintain the context of your initial query, providing tailored responses to your follow-up questions.
Start a new Conversation: Click on the "New Chat" button on the top right of the chat interface to start a new conversation. Your current chat history will be cleared, and Sectum AI will be ready to start a new conversation.
Check your Chat History: Next to the "New Chat" button, find the History button and click on it. The Chat History interface will open. You can see your latest 20 chats with Sectum AI, switch to each one and continue the conversation if you wish.
/improve
provides suggestions to improve your code.
Use it to get insights and recommendations for improving code quality, maintainability, and performance, making the overall integrity and efficiency of your codebase better.
Using /improve
Open a file: Open a file in your project.
Activate Sectum AI: Click the Sectum AI logo in the Extensions bar.
Focus: Select the part of the code you want to improve. It could be a few lines of code, a method, an entire file, your local changes and more.
Call the Command: Type /improve
in the chatbox.
(Optional) Add optional instructions: You can give Sectum AI any specific guidelines.
Result! Sectum AI will offer improvements for your code.
(Optional) Continue your chat: To give Sectum AI more instructions, just type a command or ask a question to keep chatting.
Example
Given this code snippet:
Copy
Copy
Command
Copy
Copy
Response
Copy
Copy
Unleash the full potential of AI: seamlessly switch between the world's most advanced AI models in real-time to get the best, most relevant chat experience for your query.
Available Models
GPT-4.0 - The gold standard for advanced reasoning. Reliable for everyday, iterative coding tasks requiring up-to-date knowledge.
GPT-o1-preview - Enhanced reasoning and extensive knowledge base. Ideal for planning, difficult debugging, and deep reasoning about code.
GPT-o1-mini - Lightning-fast coding specialist, optimized for efficiency. Best for quick
Claude 3.5 Sonnet - Anthropic's latest, built for precision. Ideal for everyday coding tasks with excellent flexibility and speed.
Gemini 1.5 Pro - Google's cutting-edge multimodel AI. Best suited for tasks that need the whole project in context, such as large-scale refactoring or generating project-wide documentation.
Using Model Selection
Select a model from the dropdown menu on the bottom left below the chatbox. The selected model will be used for your next query.
Sectum AI leverages advanced AI technology to generate comprehensive tests for any programming language.
Test generation boosts productivity, enhances code quality and simplifies the development process, helping you deliver reliable, high-performance software with ease.
Key Features
Universal Language Support
Sectum AI proudly offers test generation capabilities for all programming languages, ensuring every developer can benefit from automated test creation.
Initiating Test Generation
Initiating test generation is straightforward, with multiple entry points to suit your workflow. All test generation activities converge in the advanced panel, where behaviors are analyzed, and tests are crafted. Generate tests with:
Sectum AI Test Lens: Click the "Sectum AI: test" lens button appearing above every function, class, or method.
Context Menu: Right-click on any selected code or component name and select "Sectum AI - Test this component" from the context menu.
Command Palette (In VSCode): Highlight the desired code, open the command palette, and execute "Sectum AI: Generate tests."
Sectum AI Button (In JetBrains): Click on the Sectum AI icon located near every identifiable component.
Refining Your Test Suite
Sectum AI doesn't stop at generating tests. It offers an array of features for refining and customizing your test suite:
Example Tests: Guide the style of your generated tests by providing Sectum AI with an example test.
Context Awareness: Sectum AI gathers context from your code to generate relevant and accurate tests.
Running and Auto-fixing Tests: Validate and automatically fix your tests with Sectum AI 's run and auto-fix feature, available for certain supported languages.
Running Tests
Sectum AI integrates directly with your development environment to offer a streamlined process for running tests. This feature, available for Python, JavaScript, and TypeScript, enables you to:
Run and Auto-Fix Tests: Automatically run generated tests with the option for Sectum AI to attempt fixes on failing tests, enhancing efficiency by iteratively improving test success rates.
Manual Test Execution: Choose to manually run tests without auto-fix interventions, providing you with direct insight into test outcomes.
Configuration
Tailor Sectum AI's test generation to your project's needs through the Configuration tab, and continuously improve the test generation process by providing feedback on generated tests.
What is Company Codebase?
Company Codebase helps Sectum AI Chat to know your codebase better, enabling you to ask complex, code-specific questions and get better answers.
For example, with Company Codebase you can ask Sectum AI Chat "How do I use our auth service to set up authentication for my new app?" and get the right response.
How does Company Codebase work?
Retrieval-Augmented Generation (RAG) is a technique that combines retrieval-based methods with generative models to enhance the quality and relevance of generated content.
Sectum AI uses RAG to understand your company's codebase better, gain deeper context about your projects and answer more complicated or specific questions.
This feature is available for Enterprise users only.
Using Company Codebase
To start using Company Codebase:
Open Sectum AI Chat: From your extensions bar, choose the Sectum AI logo to open Sectum AI Chat.
Choose Extra Context: Click on the Extra Context
button located under the chatbox.
Select Company Codebase: From the dropdown menu, select Company Codebase
.
What can you do with Company Codebase?
Using Company Codebase in Sectum AI Chat, you can get answers to deeper, more complicated questions.
For Example, you can try asking:
"Where in our codebase do we have a function with similar functionality to this one?"
"How do I use our auth service to set up authentication for my new app?"
"What are the best practices for using our logging library?"
"How do I integrate our new API into this project?"
Template Generation: Use RAG to generate templates for common tasks, such as creating a Kotlin class with logging. This can help standardize code across your project.
Avoiding Code Duplication: Use RAG to check for code duplication by selecting the context of the current file and asking relevant questions.
Company Codebase Selection
Sectum Ai's Code Completion helps speed up your coding by reducing the need for repetitive tasks such as copying and pasting code from the internet.
It learns from your codebase and suggests real-time code completions that fit your writing flow.
Using Code Completion
Code Completion is a Pro feature, exclusively available for Teams and Enterprise users.
As you type, Sectum Ai analyzes your code in real-time to grasp your intention, and then presents code completions in a grayed-out format directly in your editor.
The suggestions aim to complete your line of thought or offer snippets that fit the context of your work.
To accept a suggestion press the Tab key. The completion will be inserted into your code.
Customization
Customize Code Completion to match your coding style by adjusting Sectum Ai Extension settings. This helps ensure that suggestions align with your project needs and preferences.
In the journey of test generation, Sectum Ai employs a sophisticated approach to thoroughly analyze your code. This analysis extends beyond mere syntax, incorporating an understanding of dependencies and imports. This comprehensive view helps grasp the full context of your code's functionality, leading to the identification of various behaviors your code exhibits.
Behavior Categories
Sectum Ai categorizes identified behaviors into three main types, each representing a different aspect of how your code operates:
Happy Path: Behaviors under this category represent the ideal and expected use cases of your code, where everything operates as intended without any errors or exceptions.
Edge Case: Behaviors that occur at the boundaries of your code's logic, handling unusual or extreme inputs or scenarios that might not be immediately obvious.
Other: A catch-all category that encompasses behaviors not fitting neatly into the first two categories, including less common use cases or those requiring special consideration.
Exploring Sub Behaviors
Each identified behavior can be expanded to reveal its sub behaviors, which are more specific instances or variations of the main behavior. This allows for a granular understanding of how your code functions in different scenarios. To explore sub behaviors:
Click the Arrow: Next to each behavior, an arrow icon allows you to expand the behavior and view its sub behaviors.
Review Sub Behaviors: Each sub behavior represents a more explicit use case of the parent behavior, providing insight into detailed operational nuances of your code.
Generating Tests for Behaviors
Sectum Ai not only identifies behaviors but also automatically generates tests for a selected set of initial behaviors to jumpstart your testing process. You have the flexibility to:
Select More Behaviors: Beyond the initial selection, you can choose additional behaviors or sub behaviors for which you want tests to be generated.
Create Custom Behaviors: If a specific behavior you're interested in is not listed, Sectum Ai allows you to define it manually:
Add Behavior: In the behavior analysis section, enter a natural language description of the desired behavior in the "Add behavior" field.
Generate Test: Click the "Add and Generate" button to create a test based on your custom behavior description.
By leveraging the behavior analysis and test generation capabilities of Sectum Ai, you can ensure comprehensive test coverage, capturing the full spectrum of how your code operates—from the most common scenarios to the edge cases that could lead to unexpected behavior.
There are multiple pathways to initiate test generation, catering to your workflow and preferences:
From the Chat:
/test-suite
Command: Execute the /test-suite
command on selected code within the chat interface to trigger test generation.
From the Editor:
For a function, class or method: Click the Test this function
button above the function declaration (can be disabled from the extension settings).
For code snippets: Select the code you want to test. Right-click on your selection and choose Codiumate
. From the dropdown menu choose Generate Tests
.
Command Palette (VSCode): use command palettes to generate tests.
The Sectum AI Testing tab will open.
The Advanced Panel
Upon initiating test generation, the advanced panel opens, marking the beginning of your test creation process. The panel is divided into two main sections:
List of Behaviors
The panel starts by presenting a list of behaviors detected in your selected code. Each behavior corresponds to a specific use case or functionality that your code is expected to perform, serving as a foundation for test coverage.
Generated Tests
Following the behavior list, you will see generated tests covering a selection of the identified behaviors. These tests are designed to provide immediate value, offering examples of how each behavior can be tested.
Example Test
At the top of the tab, there's an Example Test field, that lets you guide the style of generated tests by providing a sample test. Insert a test in the designated field or save it in the settings. If unspecified, Sectum AI chooses an existing project test as a template, ensuring consistency across your test suite.
Next Steps
After reviewing the generated tests and behaviors, you can further refine the tests, add new tests for uncovered behaviors, or modify existing tests to better suit your project's needs. The advanced panel provides a powerful, interactive environment for enhancing your project's test coverage and ensuring the reliability of your code.
Increase code coverage and generate meaningful tests locally or in a CI.
Sectum AI Cover is an agent that creates and extends test suites using CLI or within the CI process.
It analyzes your existing test coverage and intelligently generates additional tests to improve coverage while ensuring high-quality, meaningful test cases.
With Sectum AI Cover, developers and teams can maintain consistent test practices, ensure comprehensive and efficient test suites, and keep them up to date.
Sectum AI Cover is currently in preview and is available for free for a limited time for Python projects, leveraging your own LLM API key from your favorite LLM provider.
The Configuration tab within Sectum AI's Advanced Panel provides a suite of options to tailor the test suite generation process to your specific needs. These configurations ensure that the generated tests align with your project's conventions, requirements, and preferences.
Configuration Options
1. General Instructions
A free text field where you can specify general instructions that apply to the entire test suite. Use this space to request specific styling, documentation inclusion, or any other overarching guidelines you'd like Sectum AI to follow during test generation.
2. Example Test
Provide an example test in this field to guide Sectum AI on your preferred naming conventions, styling, use of mocks, etc. Sectum AI will analyze this example to align the generated tests with your project's existing patterns and practices.
3. Number of Tests
Set the desired number of tests for Sectum AI to generate initially. This allows you to control the volume of tests based on your project's scale, coverage goals, or other considerations.
4. Testing Framework
Select your preferred testing framework from the available options. This ensures that the tests Sectum AI generates are compatible with your project's testing environment and conventions.
Regenerate Test Suite
After making any changes to the configurations, it's essential to apply these updates by clicking the "Regenerate" button. This action prompts Sectum AI to regenerate the test suite based on the new configuration settings, ensuring that all modifications are accurately reflected in the generated tests.
Saving Configuration to a File
To preserve your configurations and ensure consistency across your project or team, Sectum AI allows you to save these settings to a TOML file:
Save Configurations: Within the Configuration tab, find the option to export your settings.
File Name: Save the exported configurations to a file named .codiumai.toml
.
File Location: It's recommended to place this file in the root folder of your project for easy access and recognition.
Version Control:
Collaboration: Consider adding .codiumai.toml
to your project's git repository to share these configurations with your team, promoting uniformity in test generation across all contributors.
Local Preferences: Alternatively, if the configurations are intended for personal use, you might opt to add .codiumai.toml
to your .gitignore
file to keep them local.
See an example of configuration file
Available Sections and Configuration Keys
[tests]
framework
:
Description: Specifies the testing framework to be used, affecting both the content of the generated tests and the command used to run them.
Possible Values:
Python: Pytest
, Unittest
JavaScript / TypeScript: Jest
, Mocha
, Karma
, Jasmine
, QUnit
, React Testing Library
Note: Test execution in JavaScript / TypeScript is currently supported only for Jest
, Mocha
, and React Testing Library
.
utility_library
:
Description: An additional JavaScript utility library used for testing, if any.
Possible Values: None
, Testing Library
, Enzyme
, Chai
.
Applicability: Not applicable to Python projects.
reference_test
:
Description: A multiline string, enclosed in triple quotes ("""
), providing an example test to guide the style, setup, etc., of the generated tests.
use_mocks
:
Description: Indicates whether to use mocks in the generated tests.
Possible Values: true
, false
.
num_desired_tests
:
Description: Specifies the default number of tests to be generated. Selecting fewer tests results in faster generation. Currently, this does not apply to extending test suites.
[tests.javascript]
For JavaScript / TypeScript projects, the following configuration values control the test runner:
overrideTestRunCwd
:
Description: Specifies the directory to use as the "current working directory" when running JavaScript / TypeScript tests.
Default Value: The directory containing the config file.
Note: It is common practice to place the config file in the same directory as the package.json
file and to leave this option as the default.
overrideTestRunScript
:
Description: Defines the command used to run tests.
Important: Sectum AI generates a temporary file containing the test code for a single test and runs that file. After testing, this file is deleted. For component-oriented tests, the temporary file is created next to the file being tested. For suite-extension tests, it is created next to the test suite file.
Note: You should start the command with 'npx' (e.g., 'npx jest'), and ensure the test command can run test files in the same directory as the file under test. Adjust your package.json script to avoid exclusions that could cause Sectum AI tests to be "not found".
Placeholder: TEST_FILEPATH
will be replaced with the actual path of the test file.
Examples:
Mocha:
Copy
Copy
Jest:
Copy
Copy
Debugging Note: To debug test run issues, consult the run logs in VSCode's OUTPUT (select Sectum AI from the dropdown). Clearing the output before running tests again can be helpful.
overrideImports
:
Description: A multiline string, enclosed in triple quotes ("""
), containing import declarations that will be prepended to each test file.
This problem taken from the CodeContests dataset (test-set, problem 1), illustrates the complexity and challenges faced by LLMs.
Copy
Copy
Effective test generation in Sectum AI relies heavily on understanding the context of your project. When initiating test generation, Sectum AI meticulously collects context based on your code's dependencies and interactions. This rich context is crucial for generating accurate and meaningful tests that closely align with your project's specific requirements.
Context Collection
Sectum AI's context collection process encompasses several aspects:
Code Dependencies: Analyzes the dependencies within your code to understand the relationships and functionalities that need to be tested.
Existing Tests: Searches for existing tests within your project that could serve as templates for new tests, enhancing style consistency across your test suite. Learn more about example tests.
Reviewing Context
To gain insights into the context Sectum AI has collected for test generation:
Access the Context Tab: The advanced panel features a "Context" tab where the collected context is displayed.
Explore Context Tags: Each piece of context is tagged, providing a clear indication of its source and type.
Navigate to Source: Clicking on a context tag will take you directly to the corresponding code or test, allowing you to review the basis for Sectum AI's test generation decisions.
Context Types
Each context tag is associated with a type, which helps in understanding the role it plays in test generation:
Reference Test: Existing tests in your project that provide styling cues or functional insights for generating new tests.
Imported File: Files that your code depends on, offering structural and functional context that influences test generation.
Referenced in Files: Identifies locations where the code is used or referenced across your project, providing insight into real-world usage and potential edge cases that need testing.
Benefits of Context Awareness
By leveraging detailed context, Sectum AI ensures that the generated tests are not only syntactically correct but also deeply integrated with the logic and architecture of your project, resulting in a robust and comprehensive test suite.
Understanding the context behind test generation allows you to appreciate the depth of Sectum AI's analysis and its impact on producing high-quality, relevant tests tailored to your project's unique environment.
Similar to other popular generative-AI tools, we transmit code snippets to our servers. You can learn more in our terms of use and privacy policy.
Paying users, in Teams and Enterprise plans, are opted out from Sectum AI's model training. Free users have the option to request for opt-out.
To opt out:
Log in to https://app.SectumAI.ai/.
Go to Account Settings.
Toggle the "data utilization" switch off.
OptOut direcly from our JetBrains plugin In Sectum AI's plugin settings, check the box Opt out of sharing my data with
Sectum AI.
Installation Guide
Virtual Environment Setup:
Create and activate a virtual environment.
Install dependencies:
Copy
Copy
API Key Configuration:
Copy alpha_codium/settings/.secrets_template.toml
to .secrets.toml
.
Insert your OpenAI API key:
Copy
Copy
Dataset Preparation:
Download and extract the CodeContest dataset from Hugging Face.
Place the extracted folder in the project root directory.
For VSCode users, Sectum AI offers an integrated feature to run tests directly from the advanced panel. This ensures your generated test suite performs as expected right off the bat. You have two primary options for running tests: with automatic fixes or without.
Running Tests Options
Run and Auto Fix: This default option attempts to automatically fix any issues if a test fails.
Run Only: Accessible by clicking the arrow next to the "Run and auto-fix" button, this option runs the test without attempting any automatic fixes.
Both single tests and all tests in the suite can be executed using the "Run all tests" button located above the test list.
Run and Auto-Fix Flow
When opting for "Run and auto-fix", Sectum AI acts as a testing agent through the following steps:
Initial Test Run: Sectum AI runs the selected test(s).
Success Path: If the test passes, the process concludes successfully.
Failure Analysis:
If a test fails, Sectum AI analyzes the failure and displays a summary under the POSSIBLE TEST ISSUE
section.
Test Issue: If the issue lies within the test, Sectum AI attempts a fix and reruns the test, looping back to step 1.
Code Issue: If the analysis suggests a bug in the code, Sectum AI halts auto-fixing and alerts the user. A bug summary and a "fix code" option are presented under the POSSIBLE CODE ISSUE
. Clicking "fix code" generates a diff view with the proposed fix, which the user can accept to update the code and regenerate the test suite.
Handling Test Run Issues
There might be instances where tests cannot run due to various reasons, such as:
Framework Installation: The testing framework might not be installed in your project, requiring setup before proceeding.
Import Issues: Problems with imports or dependencies can also prevent tests from running successfully.
In such cases, Sectum AI provides a summary of the issue without attempting auto-fixes, guiding you towards resolving the underlying problem to enable test execution.
Sectum AI enhances its test generation capabilities by allowing the use of example tests. This feature enables Sectum AI to generate new tests that follow a specific style or format you prefer, ensuring consistency and adherence to your project's testing conventions.
Providing an Example Test
There are several ways to provide Sectum AI with an example test, each designed to suit different workflows and preferences:
Direct Input: Copy and paste a test you like into the "Example Test" field located in the main test tab. Once you regenerate the test suite, Sectum AI will attempt to generate new tests mirroring the styling of your example.
Configuration Tab: For a more permanent setup, you can add the example test to the configuration tab. Saving this to the configuration file means all future tests generated by Sectum AI will use this example as a template.
Automatic Detection: If no example test is provided, Sectum AI will search your project for a suitable test to use as an example. The identified test will be displayed in the main test tab with a note explaining Sectum AI's choice. You have the option to delete or change this test if it doesn't meet your requirements.
Benefits of Using Example Tests
Consistency: Ensures that all generated tests adhere to your preferred style and formatting, maintaining consistency across your test suite.
Customization: Gives you the flexibility to influence the structure and content of the generated tests, tailoring them to better fit your project's needs.
Efficiency: Saves time by automating the generation of tests that align with your existing testing practices, reducing the need for manual adjustments.
Managing the Example Test
Change or Remove: At any point, you can modify or remove the current example test from the main test tab to adjust Sectum AI's test generation behavior.
Update Configuration: To change the default example test for future test generations, simply update the example test in the configuration tab and save your changes.