Some time ago, I organized a talk on “Code Quality and why developers should care about” for my company. In that presentation, I demonstrate some tools, unfamiliar to the audience: young developers and battle-hardened IT developers.

One among these tools has brought to me a lot of questions: Codacy. I will present its features and perform a comparison between Codacy and SonarQube from SonarSource.

How the choice of a Application Quality Measurement solution is influencing software teamwork?

Disclaimer: This article is not endorsed by Codacy. It reflects my personal opinion and experience.

According to a study ordered by Codacy, Codacy’s product may improve the code quality perception up to 20% and optimize the code review process up to 30%. We can assume that it reflects the disruption brought by the new code quality solutions in contrast with the traditional ones. The power of the code analysis also seems to be a minor criterion in the choice of a solution. Interviewed users are justifying the ROI of such a solution in its ability to ease and optimize the software development process.

Codacy is a company that started as Qamine with funding from Seedcamp in 2012. Its clients include Adobe, Deliveroo, Intel, Paypal. more than 30.000 developers are daily using their products and more than 1000 companies use their services. It’s more than a billion lines of code that are analyzed per day.

Features

Here are Codacy’s main features :

• Code review automation
• Code Quality analytics
• Security Code analysis
• Cluster installation / multiple instances

The first time, I have tried Codacy, I signed in using my GitHub account and launched some analysis on my public repositories.

At a first glance, we can observe that most of my repositories are analyzed without a line of configuration. Codacy is detecting most of what it needs automatically: handy.

One of my projects has not been analyzed. This project is developed in Groovy using a Gradle packaging script.

Officially, the supported languages in Codacy are :

• Scala
• Java
• JavaScript
• Python
• Ruby
• PHP
• Apex
• JSP
• XML
• Velocity
• VisualForce
• C#

According to Codacy, the language support includes several community plugins working out-of-the-box.

• CoffeeScript
• CSS
• C/C++
• Dockerfile
• SASS
• Shell Script
• Swift
• TypeScript
• LESS
• Go

Reference

For each one of my projects, Codacy is providing metrics on the quality of my project.

Quality evolution metrics

One of the best Codacy’s features is the “pre-commit strategy”. Codacy is triggering an analysis for every commit of my repository. It is even analyzing the past commits of my repository to draw the whole tendency. For each commit, we can see the evolution of the quality: number of created/deleted issues…

The dashboard is clean and really easy to use. The widgets provide the necessary information to assess the present project quality. Both metric and violation statistics are integrated with Codacy’s to build a simple quality model (Complexity, Style, Compatibility, Error-Prone, Performance, Security, Unused code).

This quality model is specific to Codacy even though it is close enough to the ISO 250001/9126 specification. Codacy does not seem to offer any SQALE / ISO or Security standard models.

Codacy dashboard

A specific dashboard in Codacy allows the developers to evaluate the security characteristic of my project.

Codacy is strongly mapped to your repository and your files. Most of the metrics and violations are associated with your files like on this screenshot. Abstract representations such as packages, modules, namespaces are not represented and the analysis tends to be complex for large projects

Each information is clearly reported and each file receives a score based on a letter from A to F.

The new dashboard better represents your code quality data and basically answers two important questions:

• How does my code quality look right now, and…
• How does my code quality evolve?

To answer the first question, you have the project grade (A to F) and the four main metrics (static analysis issues, complexity, duplication, and coverage). Each issue is in one of six categories (Security, Error-Prone, Code Style, Compatibility, Unused Code, and Performance). The overall technical debt is also estimated.

Duplicate code tracking

A nice feature in Codacy is that duplicate code is easy to track and to follow like in this screenshot :

File code quality summary

An important feature in Codacy is the possibility to handle natively branches, tags, and other important things provided by your SCM. It won’t be necessary to create several projects for handling the branches or to imagine tricky project naming.

Codacy can be used as an efficient quality gate to accept/deny your team commits or other external contributions. It becomes very handy with the mechanism of triggers that controls the code quality when the push has been submitted.

Whether you are using Github, GitLab, Bitbucket, or your own repository, Codacy provides some integration to perform a priori checks of your commits, pull-requests.

Another interesting feature is that your pull requests can be analyzed and a quality report inserted into the PR discussion.

We also find in Codacy, an interface to manipulate and configure the rules used to analyze your code source. However, the functionality is missing flexibility (Parameters seems to be missing from rules) and the rule documentation has not been imported from the 3rd addons.

SonarQube has mostly deleted from their default dashboard the possibility to evaluate the tendency of your code quality (remember the little arrows…) and the default charts to read the metric evolutions. While you may customize your dashboard in SonarQube, Codacy is providing defacto an interface to track efficiently the evolution of the quality of your projects and locate the source of your defects per commit, tags, and so on.

The line graph shows the evolution of every metric over the past 31 days and then commits the have been made. In addition to that, you can define a standard value for each metric, which represents your expected code quality (e.g. not more than 20% issues, not less than 70% coverage, etc). If any of the metrics are not good enough. You can think of it as the quality gate/threshold for your project.

After using Codacy deeper, I think that their developers are recommending including your configuration files directly in your GIT repository. The configuration can be easily tracked thanks to the versioning of your configuration (and a bonus to be shared with your IDE.)

SonarQube vs Codacy : an alternative ?

I am a frequent user of SonarQube and for that reason, I wish to share my personal comparison between the two solutions to help anyone that could hesitate for another solution.

Review automation

For a long time, SonarQube has been the easiest way to perform code quality analyses. Some years ago – an eternity in this domain -, tools like Squale were requiring a long and difficult process to tune, configure and finally obtain the code metrics. SonarQube brought a new model, “code analysis as a service” where all details from the project creation could be either imported from build scripts like Maven or modified a posteriori. This feature has been the great force of SonarQube.

However, this approach also contains a major drawback. SonarQube is lacking the adaptability to be used as a quality gate. An example? The possibility exists to use SonarQube to perform analyses per commit. It’s a clever trick using your CI Build like Jenkins. However, SonarQube won’t hold any metrics per commit, tags, branches. Branches are a rather new feature in SonarQube though its usage is quite cumbersome, even more, to perform the comparison between branches (link1, link2). In the end, technologies like GIT or Bitbucket have created the need for tools that produce metrics for each branch, commit, pull requests to help the Product Owner.

Security code analysis

Security and vulnerability detection is a tough problem to solve by static analysis tools. This niche market is already populated with well-established leaders (Coverity, Fortify, Klocwork…) These kinds of tools are really difficult to implement: misprediction, lack of information, and the complexity to understand and to fix the issues are common issues.

Codacy is therefore not a new security code analyzer. Codacy integrates many existing tools and accumulates all analysis results in one interface. The diagnostic is built with the data from the plugins and other integrated tools inside Codacy.

Language coverage and the power of the static analyses

At the first glance, the coverage by Codacy is quite impressive. Codacy is well integrated with many open-source tools.

SonarQube is beating Codacy by offering to companies several languages support for proprietary / specialized languages like Cobol, ABAP.

That is a crucial difference between the two solutions. If you are developing legacy software or vendor-locked languages, Codacy is probably not your solution. If you want an application portfolio analysis that offers you a score whatever the languages present in your code, SonarQube will be the best at this task.

However, if you are developing new software with trendy languages (TypeScript, Scala), Codacy is fitting better. Even if SonarQube is developing their own code analyzers, usually more powerful than the opensource alternative, the consequence is that SonarQube is less reactive to the new languages and frameworks. Several open-source plugins may complete the SonarQube possibilities but it requires that your administrators allow it. That’s the main restriction that prevents some SonarQube users to be able to analyze their new projects.

User-friendliness

Along with the SonarQube releases, the UX has greatly evolved from a simple but efficient GUI to a highly customizable product allowing Software Quality managers to build their own dashboards.

Although the usage of SonarQube becomes more complex in the recent releases (6): complexity to obtain the full list of violations, to browse the list of rules violated. These are some use cases difficult to reproduce without using the REST API. SonarQube is in a full transition that deteriorates lightly the user experience.

In contrast, Codacy has been designed as a simple but efficient UI to see your data, your indicators simply. It’s easy to use by a developer team, in an agile context that does not want to spend several hours to align two widgets.

Conclusion

SonarQube or Codacy? I think both are answering two different usages and expectations.

SonarQube (at least until its Cloud-based offer becomes mature) has been an on-premise solution, giving the power to companies to analyze their projects, whatever their organization could be. SonarSource has developed proprietary code analyzers to extend the possibility of their platform to better fulfill their customer needs: security, vendor languages, quality models… The strengths of SonarQube are: proprietary languages support, market leadership, open-source/proprietary licensing model. Its weaknesses are an expensive/ slightly complex licensing model, an complex UI, an on-premise solution, and possibly a lack of responsiveness to the market needs.

Codacy in comparison has been designed as a Cloud-based solution thought to fit the needs of remote software developers, using mainly Github, Bitbucket Saas service, working, building, communicating, and deploying into the Cloud. Their solution is deeply mapped to the organization of your code repositories.

Codacy’s strengths are: easy to use, a friendly UI, the possibility to analyze every contribution instantly, the cheapest price. This solution also provides a good integration of the available open-source code analyzers.

Codacy’s weaknesses are the lack of integration of other Saas services (Blackduck, Sonatype, UI/E2E testing Saas services or API QOS metrics from AWS API Gateways…), the complexity to contribute to the ecosystem, the relatively small community. The impossibility to cipher the project information or to limit the access to the source code in the UI.

I hope that this article has been useful for you. If so, please write a little comment or please share it with your colleagues.

Feedback of this article on Codacy: a great example of the technological post