NC State held it’s first ever Global Accessibility Awareness Day Challenge to see which campus Web sites could correct the largest number of accessibility errors. The challenge was based upon automated scans of Web sites which evaluated for both Section 508 and WCAG 2 conformance. The winners were the sites which corrected the highest percentage of errors. The challenge was just one part of the larger accessibility scanning system we have set up.
I could write up a lot about the system that I’ve created to do this, and I’ll have more details about the mechanics of the larger system, but I wanted to share some lessons learned about one of the more novel aspects about this system – the game aspect. What happens when you introduce gaming aspects into accessibility?
For this project, the game aspects I introduced were the following.
- the ability to see where you rank anonymously among all other sites on campus
- guides for how to improve your site
- the ability to quickly see how changes you make impact your accessibility ranking
- awards and recognition, not just for the “winners” in each category, but for all sites which show significant improvement
1. Sometimes it’s all about the game…friendly competition is a motivator
In fact, some people will fix things without you even having to ask them. If I had approached a group on campus and said, “You have 28,000 accessibility errors spread across 8000 pages and I need you to correct them,” let’s face it, I would have gotten no traction. Perhaps they feel that if they’ve had 28,000 errors for this long, why do they need to fix them now? Perhaps they had other pressing matters they had to deal with. Who knows why.
Instead, I sent out an email that in essence said, “We are having a contest to see who can correct the greatest percentage of accessibility errors in their site over the next 2 weeks.” When they went to view their current standing they saw that they were ranked as the 371st most accessible site out of 385 and ranked in the bottom 10% in all categories for all sites on campus. At the end of two weeks the site in question had corrected about 27,500 errors, ranked as the 40th most accessible site out of 385, and was in the top 5% in all categories for all sites on campus. I didn’t have to ask for this group to do anything – they just did it themselves. They ended up doing quite well in the contest. After the contest was over I saw one of the directors which oversees this group who had only learned about the contest after it was over and I told him that I was co-opting his employees for my own purposes.
2. But sometimes it’s all about the beer…it’s about building relationships
Yes, we play games because we like to win, but we also play games with people because we like to be with the other people. When I started the contest I knew who several of the participants would be because I’ve gotten to know them over the years and knew they would jump all over this. The great surprise though is who else shows up to play the game. It’s like a “pick-up” game of accessibility fixes. You bring a ball to the playground and you never know who is going to want to play.
Lots of people decided to come play. Currently 65% of all of the Web sites on campus have had someone claim them as their own and look at their accessibility report. What percentage of people normally would hit “delete” or “mark as spam” when they receive an email from “the accessibility guy” saying he has an accessibility report for them? (I’m not saying that would ever happen here at NC State – it’s just a hypothetical question.)
Not all of those Web site owners decided to play in the contest, but them coming to the game and the discussions I had with them led to the next three lessons learned.
3. Have training and tutorials immediately available to users that solve the specific problems they are trying to fix
WCAG 2.0 is great, but let’s face it, not everyone was meant to have a deep meaningful relationship with the WCAG documentation. The documentation is extremely thorough, but that’s also what makes it so unapproachable for so many people. The documentation is War and Peace, but what people really want is the Cliff’s Notes version.
Most people aren’t like me. They don’t like to read about all of the nuances and possible implementations for each success criteria in WCAG or come up with creative implementations that haven’t been thought of yet. They have other jobs they want to do.
The moral of the story – don’t send people to the WCAG documentation to learn more about their errors or how to fix them. Tell them in your own words. You know what problems they are facing, so give them the solutions they need, not EVERY solution and nuance possible.
In our system when users view a list of their accessibility errors, along with that they get a brief description of the error that I wrote along with a link to a tutorial that I wrote that gets them exactly what they need. Many of the examples I use come straight out of the WCAG documentation, but I’ve streamlined the process. As an example, when users see that they don’t have the language of the page defined, they get a link for Defining the Language of the Document
Many of these tutorials were written before the game began, but they were updated as new questions came in.
4. Different people come to the game with different skill levels, so you have to figure out how to meet people where they are
I would love to play basketball with Michael Jordan, but Michael Jordan might not like playing with me. How do you create a game space where people with varying skill levels can play?
When the game was first envisioned it was designed with Web developers in mind – people who were comfortable with viewing the source code of documents. I always knew I wanted to reach out to people beyond the traditional Web developer group, like our communications people and content creators. I thought one of my “ins” with these groups was I could also report to people how many broken links and misspelled words they had. While people might not make accessibility a top priority in their jobs, correcting misspelled words and broken links were a high priority for professionally produced content.
When I sent the communicators and content creators information about the game, I stated that they might want to forward much of this information to their Web developers. Then came the day when I taught a class on Interpreting your Accessibility Scan Results, and the only people that showed up were content creators. To them source code was a foreign language. So how do you teach about an accessibility report that talks in technical terms and gives line numbers of where errors occur? Or an even bigger problem, how do you speak to someone about an accessibility report who doesn’t even understand the concept of accessibility?
We spent the hour looking at their reports and starting to break out which errors were probably things their Web developers would have to fix and which were things that they as content creators would have to fix. Out of that discussion we were able to start generalizing probabilities of whether certain errors applied to them or applied to their developers. That information will soon make it’s way into everyone’s report, which leads us to the next lesson learned.
5. Don’t get too attached to your game design, because it probably could have been designed better and should have been designed better
Creating good games is hard and the best ones change when they need to. Did you know that in the original rules for basketball dribbling was not allowed?
You might not get all of your rules right the first time either. Be open to suggestions that people give you for improvements. Also be very responsive with changes, especially when you have one of those moments when accessibility is getting their undivided attention.
Ever since the game went live, here are some of the suggestions that people gave that I implemented, usually within a day or two of receiving the request.
- ability to request rescans as often as you want them as opposed to once a quarter – this took a major overhaul of some of the back-end processing
- showing historical data for how a site has improved over time
- instead of showing line numbers where misspelled words are, show the actual misspelled words – this was not as easy as it seems given the tool we were using
- added distribution graphs to show where there site falls in terms of total errors
6. Automated scans miss some of the biggest accessibility problems
I knew this going in, but it was really obvious when I saw some of the results. One of our sites which did quite well in the contest still had significant accessibility problems with it that the scan didn’t pick up. There are certain aspects of accessibility that automated scans simply cannot accurately asses. For example, if a site uses a table-based layout, how do you determine if an actual data table embedded in the layout table is coded correctly? How do you even know if it’s a data table if there are no table headers, captions, or scope or summary attributes? Automated tests also fail at assessing user interaction with a Web page, like testing for true keyboard accessibility.
This automated scan doesn’t solve all of our problems, but it lets us take a step, which leads us to the next lesson.
7. Mario didn’t save the princess in world 1-1
I think one of the things that often holds us back in the accessibility field is we want to make sure we say and do everything as perfectly as possible and that we don’t leave anything out. I think being accurate and complete is very important, however, if completeness becomes the thing which prevents us from making any progress then we aren’t helping anyone.
I think this problem impacts both people who are trying to create accessible content and those of us trying to help them do it. We will always find more accessibility problems to fix. We will always find a more accessible way to implement something. As teachers we will always find better ways to say something and more complete examples to give. But you have to get it out the door. I believe in release early, iterate often.
In fact one of the aspects of games is the ability to slowly improve yourself over time. When you are able to slowly yet meaningfully make progress on improving your Web site’s accessibility, that can be very rewarding. Being able to say, “two months ago I was here, but now I’m here” is powerful.
In game design I think this approach is important too. Like I said, the tool as it is now doesn’t solve all of our problems, but we’ve made meaningful steps. I can now add new features into the game to help address some of the remaining deficiencies we have. Here are some of the planned new features.
- There will soon be a set of manual tests that you can do and earn extra points for your score once you complete and pass the test. For example, if you do a test to see if the keyboard focus is visible at all times you will get some bonus points. If you actually make the keyboard focus visible at all times you will get even more points.
- I am looking at adding additional evaluation tools into the mix that are better at detecting certain errors than our current toolset is able to do.
- And as a teaser, I’m also looking to add in some artificial intelligence and statistical techniques to assess for some problems that there are no good ways to test for yet. Stay tuned for this one.
8. If something good comes out of the game, share it with the world
The final lesson for this round – if you had fun and something good came of the game, share it with others. We had two projects come out of our challenge that are now available to everyone.
First, we now have a new Drupal module that automatically looks for links that are coded to open in new windows and it appends some text to the link to alert the user to this. The link is hidden offscreen for screen reader users and comes into view when the link receives keyboard focus or a mouse hover.
Second, we now have a bookmarklet that lets you easily assess the reading level of published Web pages.
I’ve written a lot and could write more, but it will have to wait for the next post.