Code Insights - from Beta to GA (Sourcegraph)

By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.

Preferences Deny Accept

Privacy Preference Center

When you visit websites, they may store or retrieve data in your browser. This storage is often necessary for the basic functionality of the website. The storage may be used for marketing, analytics, and personalization of the site, such as storing your preferences. Privacy is important to us, so you have the option of disabling certain types of storage that may not be necessary for the basic functioning of the website. Blocking categories may impact your experience on the website.

Reject all cookies Allow all cookies

Manage Consent Preferences by Category

Essential

Always Active

These items are required to enable basic website functionality.

Marketing

Essential

These items are used to deliver advertising that is more relevant to you and your interests. They may also be used to limit the number of times you see an advertisement and measure the effectiveness of advertising campaigns. Advertising networks usually place them with the website operator’s permission.

Personalization

Essential

These items allow the website to remember choices you make (such as your user name, language, or the region you are in) and provide enhanced, more personal features. For example, a website may provide you with local weather reports or traffic news by storing data about your current location.

Analytics

Essential

These items help the website operator understand how its website performs, how visitors interact with the site, and whether there may be technical issues. This storage type usually doesn’t collect information that identifies a visitor.

Confirm my preferences and close

PROJECT DETAILS

Company

Sourcegraph

Year

2021-22

Platform

Web

My role

Product Designer

Team

Product Manager, Engineering Manager, 5 Developers

PRODUCT

Sourcegraph

Code Intelligence Platform that lets developers search and explore all of their organization's code, perform large-scale changes and gain unique insights about the codebase.

Code Insights

Code Insights reveals high-level information about the codebase, based on both how the code changes over time and its current state. Code Insights is based on Sourcegraph's universal code search, making it precise and configurable. Users can track anything that can be expressed with a Sourcegraph search query: migrations, package use, version adoption, code smells, codebase size and much more, across 1,000s of repositories.

THE ORIGINS

In early 2020, Code Insights were created as a POC by one of Sourcegraph's engineers.

He learned that some customers manually track search results count to understand their code better. He proposed to automate this for the users by creating a new sub-product that tracks search results for a given query over time and visualises it.

Sourcegraph has an extremely precise search engine so the company was uniquely positioned to provide users with accurate insights about their code base.

the 'why'

Expanding the target audience by serving new use cases
So far, Sourcegraph has been heavily focusing on winning the IC devs - most of the features were the most appealing to individual engineers. Insights can be useful for engineering leaders: managers, tech leads, CTOs, etc. Those are the decision makers who ultimately buy Sourcegraph for their company.
Alignment with the product strategy of becoming a code intelligence platform
The long-term vision for Sourcegraph is to surface high-level information about the code to the users. We want to help people make decisions and support them with the necessary context. In addition, Sourcegraph should become a code intelligence platform by combining code search with other features built on top of it.

USER EXPERIENCE (v.0)

I've differentiated three functional spaces that we could use for extensions: top horizontal, bottom horizontal and vertical. Each of those spaces has its pros and cons in relation to the needs of the extension action items.

Viewing Code Insights

Only one general dashboard view - no grouping
Chart datapoints are not interactive; they don't lead to search results
Issues with charts performance, accessibility, and lack of flexibility due to the chosen charting library

The first version of the Code Insights general dashboard

Creating & Editing Insights

Configuring insights possible only via user settings editor. There is no dedicated creation UI
Users needed to add and fill in a special JSON property describing their new insight. There was no validation, autocomplete or hints available.

User settings editor where users had to configure insights manually

IDEA VALIDATION

After creating the POC and getting feedback from some of our users, we ha enough market validation to start planning for Beta. Together with the PM and EM, we needed to perform wide product research to define what we want to build, why, for whom and how.

Exploratory research (scope)

Problems Insights can solve - define the most popular use cases to focus on first
Target audience - who should we focus on: ICs or managers
Business and user value - how does insight fit into our existing strategy
Market analysis - competitors, similar products, existing workarounds
*Implementation-related research (excluded from this case study)

Methodologies

Interviews with existing customers who expressed interest in Insights or fall into the company profile we're targeting
Interviews with sales team and prospects
Desk research - competitors, other solutions, similar tools, etc.

Validated hypothesis

Insights hold value for nearly every level of an engineering organisation
Use cases that resonated the most: migration, adoption, depreciation, versions (languages, packages, patterns, etc.), code reuse, code health, security
Sourcegraph is uniquely positioned to provide many of those insights because of its robust and accurate search
Insights are valuable enough to positively impact our sales process and the bottom line

CREATION UI

After talking to some of the Insight's early adopters among out customers, we quickly validated that creating insights via JSON in user settings is the biggest blocker. For the Beta release to be successful, we needed to provide users with a creation UI that would guide them through the process.

Step 1: type of insight

We've differentiated 3 types of insights that vary when it comes to the setup: based on custom search query, language insight and based on extensions
In the first step of the creation flow, users need to choose the path. I've also used it as a moment to showcase the graphs and what Insights can be can be used for.

Creation UI: Choosing the type of Insight

Step 2: Creation UI

Creation UI: Flow of creating an Insight

Repositories

Users can list up to 20 repositories
We support users with autocomplete and field validation
Users can select to run insight over all their repositories

Creation UI: Targeted repositories

Data series

Users define each data series individually
Data series is based on Sourcegraph search query - everything that can be searched for, can be tracked as a Code Insight
Users can choose a name and a color for each data series

Creation UI: Data series

Live preview

For scope up to 20 repositories, we are able to show the insight preview as user is creating it
Live preview prevents creating 'in the dark'. We got a lot of positive feedback about this solution during usability tests

Creation UI: Live preview

Chart settings

Users can adjust granularity of the chart and its name
Taking into consideration the variety of desired timelines that came out of our research, I've decided to keep the granularity as customisable as possible, without enforcing any default values

Creation UI: Chart settings

USABILITY TESTING

Creation UI went through two rounds of usability testing with internal and external testers. Key findings:

I moved naming an Insight to the last section of the form. Users signalised that they would like to create a chart first, and then calmly think about the name.
Testers were immediately asking about defining repositories using a search query. We should focus on removing the technical limitations and providing this experience as son as possible.
Insight preview is a must-have, all the users paid attention to it and welcomed it with enthusiasm
Initially, testers struggled to understand the granularity. Insight preview was very helpful in those cases.

CHARTS

We migrated to a different chart library that gives us better experience and more flexibility:

Improved keyboard navigation
Better accessibility - the new API allowed us to mark our chart with aria labels so that voice over read the charts.
Improved look-and-feel and more flexibility of styling for the future

Code Insight: chart interactions

BETA VALIDATION

We worked closely with a group of customers who decided to experiment with Code Insights in Beta. The overall reception of the feature was extremely positive but we've repeatedly heard feedback about improvements needed before Insights can be seen as a fully valuable feature.

Main issues to address before GA (General Availability)

Viewing Code Insights - one dashboard is not enough for big customers. They would like to group their insights by topic or group of interest;
Access - so far we have 2 visibility settings: public (visible for everyone on the instance) and private (visible only to a given user). Customers need more granularity so that they can control which groups of users (departments, teams, etc.) have access to insights)
Sharing Code Insights - users would like to share a link to an insight and make sure this person sees exactly the same data as them. Currently, Insights scope is adjusted for repositories a given user has visibility into. We need to either change the model or communicate it better to avoid confusion.

DASHBOARDS

Allowing users to group insights and control access to those groups became a clear next step in the process of becoming ready for the GA. We've decided to combine sharing, privacy and grouping in one solution - dashboards.

Code Insights Dashboard

Privacy & access

We've decided to move from insight-based to dashboard-based privacy settings
Insights inherit settings from dashboard(s) to which they are assigned. If an insight is assigned to dashboards with different privacy settings, it inherits the broader setting.

Adding a dashboard

When adding a dashboard, users can choose between 3 privacy settings: private (only this user), shared (one of the organisations) or global (visible for everyone on a given Sourcegraph instance). Those settings will be inherited by the insights included in the dashboard.

Add new dashboard

Selecting a dashboard

Users can choose a dashboard from the dropdown. Dashboards are grouped by visibility and the list is searchable.

Dashboard select (dropdown)

Assigning insights to a dashboard

Managing insights assigned to a dashboard is done via a popup. In addition, users can remove insights from a dashboard using the insights contextual menu.

Assigning insights to a dashboard

THE MARKETING LAUNCH

We've collaborated with our Marketing Team to prepare the Code Insights launch. In addition to providing input for the landing page, documentation, press release, and so on; I've created a promotional video that was used for social media posts, articles and on our website.

Promotional video for Code Insights launch

FUTURE IMPROVEMENTS

Filtering the insight - 'drill-down'
Our users would like to narrow down the scope of already created insights and be able to compare results between repositories.
Automatically generated data series
Another type of insight we're planning to introduce is 'detect and track patterns'. Data series will be generated dynamically for each unique value from the regular expression capture group included in the search query. Chart will be updated as new values appear in the code base.
Scoping repositories via search query
Currently, we provide ability to scope insight to all repositories or a manually defined list of repositories. Our users would like to also define groups of repositories using dynamic search query.

MEASURING THE SUCCESS

KPIs & metrics

Insights usage measured by hovers and clicks on the insights
Insights creation measured by the number of insights created per user and per instance
Dashboards usage measured by number of visits on every dashboard
Dashboards creation measured by number of dashboards created on the instance

Impact on company goals

Contribution to ARR - both as a separately sold product and as a feature
Impact on expansion within the existing customers
Impact on prospects - are Insights contributing to potential customers understanding the power of the code intelligence platform