First impressions of automated checkers for WCAG 2.0 AA

A week ago I finished my assignment on automated checkers for WCAG 2.0 AA. I asked it to check the home page of another blog of mine and it spewed out 223 “potential problems.” (A classmate told me she got more than a thousand.)

I drudgingly went through the list and at the end what did I find?

Three legitimate concerns.

Yes, three out of 223 were all I could find. That’s an accuracy of just slightly over 1%.

Granted, from an AI point of view I know that we don’t talk about accuracy but rather about recall and precision, and a lot of the bogus warnings do concern deep AI problems such as how human language can be understood. Or impossible-to-solve ones like guessing the author’s intent. That said, some of those warnings—especially those related to standard third-party Javascript library API calls, standard icons, or, incredibly, non-breaking spaces—are just incredulous.

I’m not hating the checker or have anything against it per se, but for these checkers to be taken seriously they really have to get better. An accuracy of 1% is not going to work.