Five Fatal Flaws in Dane County Election Audits

DaneCountyElectionAudit.jpgSummary: Once again, something we hoped would be an improvement is much less than it appears. Dane County Clerk Scott McDonell has been saying that he now conducts voting machine audits after every election. But what might look like an audit to a casual observer is at best only a spot check. The county's efforts follow neither GAB instructions for voting-machine audits nor national recommendations for election audits. 

The inefficiency of the county effort is astounding. The Wisconsin Election Integrity Action team, with only two hours of active vote-counting, completed our audit confirming the countywide April 5 outcomes in both the Supreme Court race and the Democratic primary (because we followed national authorities' recommendations.) In contrast, McDonell's idiosyncratic process kept his auditors counting for 7.5 hours in an effort that, at best, could confirm output from only two machines.

Worse, McDonell's insistence on declaring election results accurate first and checking later means that the process cannot protect Dane County's final election results from undetected miscounts. If such a delayed spot-check was to detect an electronic miscount, it would cause only chaos, unnecessary controversy, and possible legal action.

The following post explains five of the flaws in McDonell's process.


1 - The big problem--TIMELINESS

Certification of election results needs to mean something. We cannot issue certificates of election to candidates only to ask for their return two weeks later if we find an error. Statutes provide the county board of canvass with two weeks or so following every election for a very good reason: to make sure the results are accurate because they won't be able to change them later. Final is final.

On April 13, 2016, Dane County Clerk Scott McDonell and the county board of canvassers certified the April 5 election results as final by signing their names to a statement declaring that they had reviewed the results and found them to be 'true and correct.' On April 20, McDonell asked several experienced poll workers to check to see whether or not they were.

What exactly is the point of auditing if results have already been certified? You cannot fix any errors you might find. And you won't convince any suspicious citizens of the election's integrity: No one is reassured when a public official checks information he has already sworn is accurate. 

Accuracy needs to be established during the canvass process, as part of that process.

"Post-election audits must be completed prior to finalizing official election results." Principles and Best Practices for Post-Election Audits

2 - Too few machines; miscounts might not be detected -VALIDITY

StandardPracticesAudit.jpgSpot-checking the accuracy of only two machines is good enough if the audit's purpose is to create an appearance of careful management for casual observers. It also works if the purpose is limited to detecting malfunctions that uniformly affect every one of the 150+ machines in Dane County.

But if the purpose is to make sure that the voting machines identified the correct winners,* two machines isn't enough. Election outcomes can be altered even when some machines count correctly. Overheating, dust and dirt, loss of calibration, and other malfunctions can affect some voting machines and not others. A corrupt insider with access to the voting machines' memory sticks can tamper with some and leave others alone--in fact, some forensic experts say that not-everywhere-at-once fraud is the most likely kind.

That's why national election-administration authorities, including the Presidential Commission on Election Administration, recommend risk-limiting auditing, a method developed by members of the American Statistical Society specifically for the purpose of confirming election outcomes--that is, who won. I won't bore you with the details, but if you come to any of the Wisconsin Election Integrity Action Team's Citizen's Audits (the next one is scheduled for September 10, 2016), you can see it demonstrated.

Briefly: A random sample of either whole precincts or individual ballots is selected that is large enough so that the manually counted sample results can be compared to the electronically tabulated results and, with the application of a statistical formula, reveal the probability that the two counts accurately represent the same thing--the verdict of the voters. The sample size will vary depending upon how many ballots were cast and the reported margin of victory. (Closer races need larger samples than landslides.) If the probability is 99% (or whatever level of certainty you're aiming for), the audit is complete. If the manual count of the sample does not produce 99% probability of accuracy, the sample size is expanded until the electronically tabulated outcome is confirmed as either correct or incorrect.

The point is to make sure that the audit confirms the election results, not just the output of a few  machines.

"The Commission endorses both risk-limiting audits that ensure the correct winner has been determined according to a sample of votes cast, and performance audits that evaluate whether the voting technology performs as promised and expected." Report of the Presidential Commission on Election Administration

3 - No one can tell whether the auditors are accurate or inaccurate, honest or cheating - TRANSPARENCY

To provide confirmation that elections have integrity, the audits must also be confirmed to have integrity. Outside elections, auditors often provide that confirmation by being independent--having no stake in the outcome of the audit. But for election audits, no truly disinterested auditor exists because we all have a stake in the outcome. For that reason, election audits need full transparency. People other than the auditors need to be able to tell--for themselves--that the audit is being conducted honestly and correctly.

Transparency means having written procedures (so that observers can tell the auditors are not just making steps up as they go along); secure internal control of the audit documents (to prove to observers that no one had an opportunity to alter or replace the audit records without being detected); and a way for observers to see that the tasks are honestly and accurately performed during the audit.

The Dane County audits have none of those.

"Any election where all post-election activity is done in secret or in a manner that doesn't support oversight and accountability will engender zero confidence in the electorate." Transparency in Post-Election Audits

No written procedures. With regard to written procedures, we can cut the Dane County auditors some slack. McDonell didn't hire the chief inspector who is helping him develop the audit procedures until late December 2015, and they have conducted audits after only two elections--February and April 2016. They're still working out the kinks, so I wasn't surprised when observers at the April audit were given nothing in writing to help them know what the auditors were supposed to be doing. The auditors themselves weren't provided with any. The reports that followed both the February and April audits contained after-the-fact written descriptions of what had happened, but both reports also contained recommendations that the procedures change before the next audit.

Weak internal controls.  Professional auditors know that "internal controls" are those measures that ensure the records used in an audit can be proven to be true and complete. Most election workers in Wisconsin--at least those with no management or audit experience--know internal control only as "chain of custody." Using what Joseph Hall of the Center for Democracy and Technology calls a "dotting the i's and crossing the t's" mentality," audit-naive election workers consider that they achieve adequate internal control simply by following a prescribed ritual involving security bags and seals. Election records are placed in security bags, closed with numbered seals in such a way that the records cannot be taken from the bag without breaking the seal. Comparison of the seal number to a paper record upon opening the bag is intended to ensure that the seal has not been opened and replaced with another. Some commercial seals used are difficult but not impossible to defeat, but in Wisconsin little care is typically taken to make sure no one ever has solitary access to the sealed records during the time they are stored. I don't know what security is provided to the paper record of the seal numbers.

McDonell's amateurish appreciation of internal control shows in how he allowed his auditors to handle the records. In the February audit, which relied on digital ballot images, the chief auditor documented that he took the sealed bag containing the audit records home with him overnight, documented that he opened the bag and worked alone with the ballot images all night, documented that he resealed the bag in the morning, and then at the next day's audit, the seal was ceremoniously broken in compliance with the ritual. This is Keystone Cops internal control: The auditor wasn't documenting that he had no opportunity to tamper with the records; he was documenting that he had and in fact did. 

At the April audit, the naivete about internal control took another form. Observers were carefully informed that the paper ballots (no digital images in this audit) had been transported from Stoughton, where they were being stored, by two of the auditors. In states more knowledgeable about standard audit precautions, transportation is usually handled in a way that precludes any opportunity for the clerk's employees to alter or substitute the records, such as by having representatives of opposing political parties accompany the ballots. Any citizen who was skeptical about McDonell's motive for doing the audit would have just rolled his eyes when told the clerk had sent two hand-picked employees to retrieve the ballots, with no one to watch whether they switched out the ballot bags as they did.

Observers' inability to observe. The first big violation of transparency came the day after the election, when the county clerk and the chief auditor selected the precincts in the privacy of the clerk's office--randomly, they assure us. I don't need to explain what's silly about that.

Then, at the audit, the auditors insisted all observers stand at least three feet away at all times, and they employed no technology, such as document projectors, to make anything they were doing visible to anyone other than themselves. It's not as if county election officials don't know the importance of observation--Wisconsin recounts are conducted with members of the opposing parties viewing each ballot as it is counted. which ensures that no one has an opportunity to cheat or make mistakes. But the Dane County audits made no provision for anyone other than the auditors to see anything of consequence. To his credit, the lead auditor for a while turned his back to me as he held each ballot just above his head to read it. In this way he was allowing me, from three feet away, to discern where the votes were, even if I couldn't read any of the names. No way could either he or I have kept that up for the entire 7.5-hour duration of the hand count.

But it wasn't just observers who couldn't observe enough to tell whether votes were being counted accurately--it was most of the auditors themselves! One vote-reading auditor would read the votes aloud as two other auditors made hash marks on a tally sheet, without ever seeing an actual vote themselves. The obvious flaw in this method is that it requires everyone to trust that the reading auditor is reading the votes honestly. Another danger (the one most often cited as the need for having more than one set of eyes on each ballot) is that the reading auditor can misspeak so that both tallying auditors record the the same error. That brings us to the fourth weakness....

4 - Insufficient safeguards to make sure it's done right - RELIABILITY

When only one auditor sees each ballot, the audit becomes less of a test of the voting machines' accuracy than a test of the one auditor's ability to read votes aloud for hours and the tally-makers' ability to keep hash marks accurately. And because the 'votes' that the tally-makers were recording were nothing more than spoken words, there is no way at the end of the audit to determine the source of any errors if the hand-count totals mismatch the machine-count totals. In fact, the auditors' report from the April audit confirms what any experienced hand-counter would have predicted: Most candidates' hand counted totals mismatched the machine counts (by only a little) and the auditors could only guess at the reasons.

THIS IS BASIC: A redundant count--having at least two people, if not more, read every ballot--is a standard feature of any reliable hand count procedure, including the one recommended by GAB for voting-machine audits.  

5- It takes much more time than it needs to - INEFFICIENCY

Auditors at the Citizens' Audit, 10 days after the county audit, clocked themselves accurately counting votes at a rate of 100 ballots every 4 minutes. The auditors at the county audit didn't time themselves, but you can calculate it from their report. In 7.5 hours of active counting, the faster of the two teams counted 1,186 ballots--a rate of 100 ballots every 38 minutes.

The difference is the citizens' ability to use digital images instead of paper ballots, something the county auditors cannot do because McDonell insists on using impractical ballot-viewing software.

Now couple that with McDonell's refusal to consider using the risk-limiting audit method recommended by the Presidential Commission. Following those recommendations, the citizens' audit determined they needed to count 216 ballots, randomly selected from across the entire county, to confirm the outcome in the Supreme Court race and 524 ballots to confirm the Democratic primary result. In contrast, the county auditors set out to count 2,504 ballots. But because those ballots came from only two precincts, the county's relatively extravagant effort could not have confirmed the countywide outcome in either race.

The combined effect of the two audits' differences (digital images versus paper ballots, and statistical sampling versus whatever you want to call the county's sample) is astounding. The citizens completed their audit of the Supreme Court race in approximately 30 minutes and the Democratic primary result in approximately 90 minutes. performing both random selection and vote-counting during that time. Having done the random selection weeks earlier, the county auditors spent 7.5 hours in active counting, and still did not confirm anything beyond the accuracy of two individual machines. To match the citizens' audit result--confirmation of the countywide outcomes--using the county's method would have required county auditors to count at least 6 more randomly selected precincts, likely taking them three more days.

How much of a problem is this inefficiency? If the county has enough money to pay a team of six auditors for 4 days' work rather than two hours', is that a fatal problem? Looking back at timeliness, validity, transparency, and reliability, I'm going to say it is.

  • Timeliness: It's important to complete the audit during the two-week county canvass period, which places a premium on speed.
  • Validity: You need to audit a sample big enough to confirm the election outcome, so excessive time-per-precinct makes validity more difficult in a limited amount of time;
  • Transparency: You want volunteer observers to show up to confirm the audit's integrity, which means you cannot place too big a burden on them; and
  • Reliability: After two hours, human errors begin to creep in, and become more frequent as auditors tire.

 

The final report of the Dane County election audit can be found here.

The final report of the Citizens' Audit is here.

 -----------------------------------------------------------------

* The only purpose for the county audits I've heard McDonell articulate is to see whether the voting machines can count votes marked in different color inks. That's naivete from any angle:

  • Unaware of the capabilities of his equipment? The DS200 machine was designed and manufactured to count any color ink, a capability that has been confirmed in testing for years, and does not change over time or among machines.
  • Unaware that Dane County's municipal clerks do a reliable job of publicly demonstrating the machines' capabilities in pre-election tests before every election? It doesn't need his "double-checking." Almost all the municipal clerks take care to mark their test ballots with a variety of different inks.
  • Unaware of what his own auditors are doing? If he did, in fact, instruct them to make note of the ink color in which votes were marked, they didn't follow his instructions. Their methods wouldn't have allowed them to detect the causes of any miscounts (ink color or anything else),  and their final report contains no mention of the question.

Showing 2 reactions

Please check your e-mail for a link to activate your account.
  • commented 2017-04-07 09:22:05 -0500
    Thank you!
    Here’s a new link to replace the link with “national authorities’ recommendations” on auditing, since https://www.supportthevoter.gov was taken down: http://web.mit.edu/supportthevoter/www/files/2014/01/Amer-Voting-Exper-final-draft-01-09-14-508.pdf
  • commented 2016-08-29 17:58:50 -0500
    I’m recording this here, so that I can find this information about the August audit again when I need it.

    REDUNDANCY: They used a slightly different method of vote-counting—still not the one recommended by the state elections authority for audits under s.7.08(6)—but slightly better than the method they used in February, in that two auditors had at least a chance to see the votes, if they chose to. At the August audit, two teams (two auditors each) counted a single precinct. One person handled the ballots as they counted, and read each vote out loud. That person and one other would make a tally mark on a handwritten spreadsheet to record the vote. After each batch of 25 ballots, they would compare their tally sheets.

    On one of the teams, both the reading and non-reading auditor would look at the vote (GOOD—that’s the redundancy the February audit didn’t have, which they would have had built-in if they used GAB’s instructions). But on the other team, it seemed only the reader was viewing the voters’ marks. Unnecessarily, from this observer’s perspective—the reading auditor would place the ballot face down on the table between him and the non-reading auditor after he’d read the vote out loud. I have no idea why he would want to HIDE the vote from the non-reading auditor, but I did not ask. It seemed possible for the non-reading auditor to catch a glimpse of the vote while the reading auditor picked it up from the first stack or looked at it himself, but from the observers’ area, however, it did not appear that the non-reading auditor was looking at the ballots. It seemed she kept her eyes on her tally sheet.

    SPEED: I observed the first part of the August 2016 county audit, and timed the vote-counting. They were counting in batches of 25 ballots, and were counting only one race—the Dem US Senate primary.
    When they encountered no problems in a batch, they were counting at a fairly reliable rate of 25 ballots every 2 minutes and 43 seconds. (A rate of 100 ballots every 10:51—but their overall speed at the end of the audit will be greater.)
    When they encountered a problem, the rate was much more variable—one batch was resolved in 4:46, another took more than 6 minutes—the point at which I accidentally touched the wrong spot on my cell phone screen and closed the stopwatch app.
    These figures do NOT include any of the prep (random selection, ballot-bag opening, counting ballots and putting them into stack of 25, adding up the subtotals from the batches of 25 ballots, etc.)
    We’ll have to wait until they release their report to calculate the overall average speed at which they can count votes.

    OTHER COMMENTS:
    David Spies, lead auditor, did not this time invite observers to stand three feet from the auditors if they wanted to see better, and I did not ask. I have no reason to believe he would have refused had we asked, but there was a reporter present, and I wanted her to get an unmediated experience of what it would be like for a regular citizen who wanted to observe, without having an election-integrity advocate present.

    Spies was willing to take questions whenever we had one, and so at one point, I did ask—or tried to ask—why they were not using the GAB audit instructions. He didn’t answer directly—just kept saying that they were following the recount instructions. (But they weren’t—they would have made more provision for observers and for redundancy if they were). When I pressed, Spies kept repeating that they were following recount instructions, not giving any reason for not following the audit instructions. Finally he said something along the lines of “Because I’m following the instructions that were given to me.” I got the distinct feeling that he is unaware of the GAB audit instructions, but I did not ask that directly because I did not expect to get a useful, direct answer, and didn’t want to risk embarrassing him in front of the reporter.

    Another thing I noticed about speed and efficiency: when an audit is not done with a strong eye to public participation/observation (as the Dane County audits are not), it is very easy for the auditors to lapse into social chat and conversations not related to the task at hand. Even with two observers present (me and the reporter), the auditors several times lapsed into chatting among themselves, since our presence in the room was something extraneous to their perceived task.
    Social chat among the auditors doesn’t hurt the audit itself, but definitely makes it less rewarding and interesting for any observers who are present, and makes the audit more costly if the auditors are being paid hourly. In contrast, if an audit’s goals include the CLEAR, EXPLICIT benefit of the observers, auditors will remain more mindful, at all times, of the observers’ perspective and will be less likely to chat about things like what kind of snacks they like, etc.

get updates