Running pa11y in Jenkins for accessibility testing

Apr 24, 2019 · 5 min read
#accessibility #ci #automation #jenkins #testing #a11y

The best thing about being a software engineer is all the learning you can do. I’m now working on a web based product where accessibility is critical to the success of the team, and therefore I’m in full learning mode at the moment.

Backstory

One of the blogs I’ve been following whilst trying to grok all things accessibility is A11y with Lindsey. A couple of weeks ago Lindsey explained the tooling she was using from within the browser to help test accessibility. After I started to investigate all of those tools and get use to using them, it was great to see a follow up article about the CLI tools being used. Since we have all our other checks running in Jenkins, and those checks have to pass before we can merge into master, it makes sense to try and add these tools too. This is to keep myself, and the team, honest. There is a lot to remember in modern software development, so having automated checks eases the burden.

What has become clear to me, and Lindsey calls out in her articles, is that you cannot fully automate accessibility (a11y) testing. The Web Content Accessibility Guidelines (WCAG) 2.1 is a substantial tome of recommendations, and I’m not currently aware of a tool that can automate the lot.

Web Content Accessibility Guidelines (WCAG) 2.1 covers a wide range of recommendations for making Web content more accessible.

By having these tools in your pipeline, it does not mean your a11y testing is complete, far from it. But what it does give is clear intent, within your team, that a11y is important, as it’s part and parcel of getting a feature merged. This is a win for me, and since the current sprint of development we are working on is really focussed on a11y, we wanted to get pa11y running in our Jenkins pipeline.

This post is going to cover off what was required to run pa11y in Jenkins.

Installation

So, first off pa11y-ci

Pa11y CI is a CI-centric accessibility test runner, built using Pa11y.

Installing this test runner is the familiar one liner, via npm:

npm install -g pa11y-ci

This will install what is required for us to run the command in our pipeline.

Automation

We are using a Makefile in our team to provide the automation hooks required, so we wrap pa11y in a make target:

PALLY_ACCEPTABLE_ISSUE_COUNT = 70

.PHONY: test-a11y
test-a11y: ## Test what we can from an accessibility point of view
  ./node_modules/.bin/pa11y-ci \
    -c pa11y.json \
    -T $(PALLY_ACCEPTABLE_ISSUE_COUNT) \
    --sitemap http://localhost:1313/sitemap.xml \
    --sitemap-find "http://0.0.0.0" \
    --sitemap-replace "http://localhost:1313"

That’s potentially a lot there, so let’s unpack this a little.

Within the make target of test-a11y we are calling the pa11y-ci command. We have installed this locally within the application, and not globally, so we define the path to the application:

./node_modules/.bin/pa11y-ci

The -c pa11y.json argument is to pass in a pa11y configuration file. Our Jenkins instance is configured to delegate builds to run in Docker containers defined by the engineering teams (think TravisCI). This meant we hit this issue about how we run the command.

To resolve this issue, we have defined the following configuration file

// pa11y.json
{
  "defaults": {
    "chromeLaunchConfig": {
      "args": ["--no-sandbox"]
    }
  }
}

Since our containers are running with minimal privileges, this was acceptable to us.

Unfortunately, we have some existing accessibility issues we will need to resolve (which is going to be the case for most projects if you are adding this tool into your process - unless you have been very rigorous in testing). With this in mind we have defined PALLY_ACCEPTABLE_ISSUE_COUNT and passed this in as the -T argument, which defines the threshold of errors we are willing to accept.

We have defined this as the number of existing issues. Anything above this will fail the build, and therefore block the Pull Request being merged into master. We are also going back to fix the existing issues, which is a vital learning experience for the team. Once we are at 0 issues, we will remove this variable from the make target.

Lastly, we want to hit all URLs without really managing a separate URL list (because humans forget to do things). This is where we leverage features from Hugo. Hugo is a static site generator which we are using in various ways (another blog post maybe) to aide our product development. Luckily, Hugo generates a sitemap out of the box for you, which we can use.

--sitemap http://localhost:1313/sitemap.xml \
--sitemap-find "http://0.0.0.0" \
--sitemap-replace "http://localhost:1313"

Our build process in Jenkins spins up the Hugo site in serve mode, which is running on http://localhost:1313, so we can point pa11y to that URL to run against all the URLs we want to test.

All of that is then wrapped in our Jenkinsfile. The relevant part for this post, is as follows.

stage('Test') {
  steps {
    parallel(
      "Validate the HTML": {
        sh 'make test-html'
      },
      "BackstopJS visual testing": {
        sh 'make test'
      },
      "Test a11y": {
        sh 'make test-a11y'
      }
    )
  }
}

Once that has run, we are provided with this output in Jenkins.

Pa11y Jenkins Output

Summary

The main call outs for me are:

Content

Social

Recent