What this section is for
This section collects suggestions, acceptance rates and generated lines from connected AI tools. You can group developers and compare their metrics. The feature is in beta: an observed difference does not prove that AI caused it.
When to use
- To check how the team uses connected AI tools.
- Before renewing or expanding a tool licence.
- To compare metrics for developers with different levels of AI usage.
Quick start
- 1Connect the AI tools used by the team.
- 2Choose start and end dates; both are included. Review event counts and suggestion acceptance rate.
- 3Compare group means alongside the number of observations and starting values.
- 4Account for differences in tasks and experience. The product does not test statistical significance or identify the cause of a difference.
Key settings
| Setting | What it does | How to use it | Common mistake |
|---|---|---|---|
| AI integration | Connects a tool so its usage events can be collected. | Check that all tools in use are connected. Otherwise, missing data may be mistaken for low AI usage. | Comparing groups without accounting for missing data from a popular tool. |
| Group selection criteria | Selects members by AI usage, tools, dates or a list of users. | Compare the same period and similar tasks. Check differences in experience or specialisation between groups. | Building a "high AI usage" cohort that also happens to be the frontend team, and attributing the difference to AI. |
Usage example
Scenario: You want to compare pull request merge time for two groups of developers.
- 1Check group membership and connected AI tools.
- 2Use the same dates for both groups.
- 3Compare average times and observation counts. Record differences in tasks or membership that could affect the result.
What this shows: You can see differences between the groups and know what needs further investigation. The comparison does not prove an AI effect.
FAQ
Why is correlation missing?
The calculation needs observations with and without AI, and the outcome values must vary. If a group is missing or all results are the same, no coefficient is shown.
Where is the p-value?
The comparison reports observed means and sample sizes. It does not provide a significance test or establish independent randomized samples.
Does a positive association prove AI made delivery faster?
No. Experience, task complexity, team structure and tool coverage can explain differences. Use an independently designed experiment to estimate causality.
Tips
- Decide which question you want to check before comparing, such as whether to expand a licence.
- When sharing results, include the dates, group membership, connected tools and beta limitations.