Comparison: AWS CodePipeline, GitHub Actions, and Jenkins
May 14, 2020 · 10 min read
This post has come about as I’ve written up this content in different formats multiple times, and it’s easier to write this up once, and just reference back. The comparison is valid as of May 2020. If you visit this page after that, the comparison table may be different.
The tools this post is going to compare are:
- AWS CodePipeline (As I happen to be working in the AWS space at the moment)
- Jenkins
- GitHub Actions
A couple of things to note:
- This post almost exclusively talks about AWS CodePipeline, but you will almost certainly require AWS CodeBuild too.
- GitHub Actions:
- Is free for open source projects.
- Chargeable in paid for plans.
- Can run on GitHub infrastructure, but also on your infrastructure.
Comparison
For an easily consumable comparison, this table aims to quickly highlight differences.
| Category | AWS CodePipeline | GitHub Actions | Jenkins |
|---|---|---|---|
| Environments | Linux | Linux/Windows/macOS | Linux/Windows/macOS |
| Parallelism | ✅ | ✅ | ✅ |
| API | ✅ | ✅ | ✅ |
| Config as code | ✅ | ✅ | ✅ |
| Runtime discovery | ❌ | ✅ | ✅ |
| Branching Support | ❌ | ✅ | ✅ |
| Serverless | ✅ | ❌✅ | ❌ |
| Auth | AWS SSO | GitHub | GitHub |
| Status visibility | ❌ | ❌✅ | ✅ |
| Ad hoc workflows | ❌ | ❌ | ✅ |
| Cost | AWS | GitHub + Cloud Provider | Cloud Provider |
Installation
Installation of a build tool can be the first hurdle to overcome.
If you are in AWS, then the easiest tool to “install” is AWS CodePipeline, as there is no installation. It’s a first class service AWS provide. As long as you have permissions to access the service, you can start to build a pipeline.
The GitHub Actions installation experience will be determined by your account type and what you want to achieve. If you are running an open source project, then there is no installation. It’s bundled into the repo. If you need to run the actions on your own Infrastructure, then you will need to install some runners. How you do this is likely to depend on the tooling you already use. The team I am currently working in has configured the infrastructure with AWS Cloud Developer Kit (CDK), and the agents with Ansible. This means we have a repeatable workflow to scale the runners up, if required.
With Jenkins, you have a few options to install the application.
- Using your package manager - Probably the easiest way to get up and running, if your Docker knowledge is limited.
- Running a WAR file.
- Docker.
This will be entirely your preference based on the knowledge you have. In order to scale, you will need to understand the primary/agent configuration of Jenkins, and decide how you are going to scale the platform, so the installation can become quite involved.
Environments
Understanding what environments you need to run builds on, can help you determine which tool is right for you. If you want to use the same build tool across projects building for windows, linux, and macOS, then that would limit the tool choice. You may decide to pick a tool that suits a platform, or stitch many tools together.
Parallelism
Before you know it, you will be discussing build times, and how you can decrease them. One of the easiest ways to achieve this, is to run steps/tasks in parallel.
This is possible within AWS CodePipeline, but that is where the parallel execution stops. You cannot run many steps within the AWS CodeBuild stage. For example, let’s say you have a static analysis stage within the pipeline. Maybe you run a linter, security checker, and code style checker. This is likely to be in a buildspec.yml file, and you will have to execute these sequentially.
In GitHub Actions you can run a matrix build, which is similar. In your workflow file, you can define a matrix strategy for job execution. This currently can only be done at the workflow level. This is handy, if you want to check that your build and tests work on many versions of a language, for example. It doesn’t help with the static analysis scenario mentioned above. However, in GitHub Actions you could define a workflow per tool, which in turn would turn it into parallel execution. Be aware, however, that it would mean you are spending more money on the action runner (depending on the pricing plan you are on). This is because you would have to clone the repo, install all the dependencies etc many times, each incuring “runtime costs”.
Jenkins caters for this scenario the best. It can do parallel execution within the Jenkinsfile and at various levels. You can also use environment variables to decide what to execute, and on what type of operating system. The example in their documentation shows that they run the “Test” stage on Linux and Windows.
API
This may not be of interest to you, but it’s worth stating that each tool has an API you can interact with, should you wish.
This is worth knowing, if you need to stitch the build tool into other processes, like change approval.
Config as code
This has been rather a big ticket item in the industry for many years. This is the idea that resource configuration should be managed like source code in your repository. There is far too much to talk about on this topic, so we shall leave it there for this post.
All of the tools cater for this. To see how they differ, please see the next chapter, which covers the slight differences in their approach.
Runtime discovery
This section is about maintenance essentially. Can the tool “discover” what needs to be executed by the code and configuration within the branch and codebase.
AWS CodePipeline has to have a static pipeline defined. Each stage is defined either in something like CDK/Terraform, or manually. This can then delegate to a buildspec.yml file within the repo, but the pipeline itself is static. This means if your pipeline needs to change slightly between branches you need to deploy the static changes.
GitHub Actions discovers the workflow, per branch and executes what it finds in the workflow file. This is desirable, as everything is within the codebase/branch/pull request. There is no hidden configuration.
Jenkins uses the Jenkinsfile to define the pipeline per branch and execution.
For both GitHub Actions and Jenkins this means there is less overhead on the team defining static infrastructure, and allows a more fluid build pipeline, based on code/config within the repo.
Branching support
This is only really being mentioned, because of a limitation within AWS CodePipeline, at the moment. Right now, a CodePipeline can be linked to one git branch [1].
Because of this, it means you will need to manage many pipelines in order to cover some branching strategies, e.g. GitFlow. This is an overhead on your team, so keep this in mind.
GitHub Actions has an "[on]" syntax that allows you to define patterns to match. You can also run if statements within the workflow, to protect certain steps depending on the branch.
Jenkins is very similar, but uses regex patterns to determine what to run and when.
GitHub Actions and Jenkins seem more appropriate to handle different branching strategies.
Serverless
AWS CodePipline, and AWS CodeBuild are provided as a Serverless feature within AWS. Meaning, you will not be required to look after any servers. As long as you define your pipelines, either manually, or as code, the rest will be looked after for you.
GitHub Actions is similar, unless you want to run agents on your own servers. If you do, then you need to manage that piece of infrastructure.
If you want to run Jenkins then you will require some compute resource within your Cloud Provider or on premise data centres.
Auth
Authentication and authorization for AWS CodePipline and GitHub Actions are covered by the normal access to that tool.
For AWS CodePipeline, the people who need to interact with a pipeline will need IAM access to the accounts that are running the pipelines. This may or may not be desirable.
This is similar for GitHub Actions, but allows you to restrict access to certain repos within your organisation.
Jenkins, however, can be configured in many ways. This will depend on your business needs.
Status visibility
You’ve picked a tool, and you now have many jobs defined, covering many repositories and projects. Being able to see, at a glance, the status of your projects is key. You want quick, actionable feedback.
AWS CodePipeline has only just recently displayed the more recent execution of a pipeline in the pipelines screen.
This is a much needed addition. This provides an overview of all your pipelines in the account/region you are accessing. It’s worth noting that all members of your team, or anyone who needs to see this data will need access to your AWS Account, which may not be ideal. The suggestion here would be to create IAM roles to lock this down, but this comes with an overhead.
GitHub Actions is a tab on the repo. Currently you cannot get a view, within GitHub, that would show you the status across many repositories for a given project.
It cleanly shows each execution for each branch within the repo.
Jenkins has the best support for this feature, as you can define your own views, and use plugins to provide different types of view, e.g. a TV dashboard.
I would certainly recommend installing the Build Monitor View plugin.
Ad hoc workflows
If you are in a team that likes to automate a lot of things, then that automation is going to have to run somewhere. An obvious place seems to be your build tool.
AWS CodePipeline and GitHub Actions do not cater for ad hoc jobs. AWS CodePipeline needs a trigger, and then runs a static pipeline. GitHub Actions is listening to git events.
Jenkins, on the other hand, caters for many types of job. You can run bash scripts, maven, ant, powershell etc, to name just a few. If you need a job to run on a regular cadence, and for it not to be linked to a codebase, then Jenkins is capable of doing this.
Cost
AWS CodePipline is chargeable within the AWS pricing mode. However, as mentioned previously, you will also require the AWS CodeBuild service, which is also chargeable.
GitHub Actions is chargeable for organisations. If you need to run agents on your servers, you will incur costs for the infrastructure.
Jenkins is a free open source project, but you will have infrastructure costs.
Conclusion
As with all decisions, the decision can be a hard one, and any recommendations will depend on a multitude of factors specific to your needs.
To provide some guidance, this is my recommendation.
AWS CodePipeline is recommended if you have limited knowledge of Jenkins, GitHub Actions, or any experience in other tooling. If you have a single pipeline e.g. running from your master branch, and it’s not likely to get more complicated, you will be fine. If you’re happy to have basic functionality, and grow with the service, then AWS CodePipeline is a good fit. If however, you have used other tooling, you may quickly find AWS CodePipeline to be lacking in many areas. Of course, over time, AWS may build these features, so be mindful of this.
GitHub Actions is a great product, easy to use, configure, baked into a great platform. If you can run all of your workloads on their servers, then this is a perfect fit. If you require more control about your build environment, and where it is run, GitHub Actions is capable, but will require some more work. The documentation is great, and it quickly becomes intuitive. The community of actions you can use, is rich, and creating your own isn’t that difficult [1], [2].
For the most flexible/extensible experience, then Jenkins is the outright winner in my opinion. It’s a blank canvas that allows you to use it in many scenarios. Projects like Jenkins X are certainly worth looking at. Yes, it requires more knowledge, but with that knowledge comes more features. The community is rich, and the product has been around a long time. It’s agnostic of Cloud Providers, so if you are a multi cloud customer, this may be of interest.
Hopefully this provides some insights to help you chose a build tool.