Verification using digital ballot images

NOTE: The following is DRAFT material. It describes the process used by the Wisconsin Election Integrity Action Team in their two public tests of the Visual Verification System (April 2 and July 20, 2015).  These tests used Dane County's ballot images from the February 2015 primary and the April 2015 general elections, but these procedures could be used with any automatically created digital images in any county. Verification using digital images created manually after Election Day requires additional chain-of-custody safeguards (add link here to Michigan work.)

The Slide-show Verification System, step by step

The following section provides a description of the slide-show verification system as we believe it could be implemented in Dane County, Wisconsin. Some steps cannot be performed by citizens’ groups, so citizen audits will be unable to achieve the reliability and credibility possible with an official verification.

At this writing, the system has been tested only with a random sample of entire precincts, not a random sample of individual ballots, and the steps below describe only that process. With the consultation of a statistician, which we have not yet been able to arrange, the process below can be adapted to check the visually counted vote totals against the statistical targets rather than the electronically tabulated precinct totals. 

As noted, this system is still in development. At this writing (September 2015), it has been publicly tested only twice, first with a random sample of City of Madison reporting units to verify the outcome of the February 17 mayoral primary (completed in less than 45 minutes, timed from when the ballot-image display began to when results were confirmed), and later to verify the exact total in one City of Madison precinct in an April 7 yes-no referendum (completed in 6 minutes). It is likely refinements and improvements are possible for any of the following steps.

1.      Announcing the process and criteria for sample selection

The clerk will have announced before Election Day the process and criteria by which contests and precincts will be selected for the verification.

  • For self-protection against accusations of selecting contests for verification based on whether the clerk liked the outcome, the clerk must make known ahead of the criteria and process for selection of contests and precincts to be audited, and the method by which the sample size will be determined. The process should use risk-limiting auditing, to reflect currently accepted best practices for post-election verification.

2.      Submission of digital images to the county clerk

In Dane County on Election Night as part of poll-closing procedure, only the preliminary vote totals are electronically transmitted to the county clerk’s office. The flash drive is removed from the voting machine and is physically transported to the county clerk’s office in a zipper bag locked with an initialed seal, along with the other election records. Elapsed time between poll closing and the delivery of election records to the clerk’s office is an issue for chain-of-custody considerations for both the paper ballots and the flash drives containing the ballot images, particularly if the election records are stored overnight or unattended at any point before delivery to the county clerk.

When the flash drive arrives at the county clerk’s office, its contents are copied to the clerk’s computer. Each precinct’s ballot images are saved in a separate folder.

3.      Making security copies of the digital-image files

As soon as all precincts’ flash drives have been copied to the clerk’s computer, the clerk should make multiple copies of the full set of ballot-image files. Each political party with a candidate on the ballot will receive a copy, as will the newspaper of record for the county and any citizen’s group that has requested one. Under current Wisconsin law, the clerk is required only to be timely in providing these public records over to requestors, but for his or her own protection should make multiple copies of the ballot-image files as soon as possible after receiving them from all precincts.

  • Quickly making multiple copies of the digital-image files and distributing them to independent parties creates a small window of opportunity for any alterations, protecting the clerk from credible accusations of tampering. This a strong security measure because making hard-to-detect alterations to the digital-image files would take much longer than altering the numerical tabulations alone.           

4.      Conduct the public display of ballots for transparent counting of votes

At the time and place announced for the public verification of the election outcomes, one of the flash drives with the ballot images is plugged into a computer loaded with the software that can project the ballot images as a slide show on a wall-sized screen.

The slide-show software and display

  • The slide-show software makes no changes to the digital-image files themselves; it merely reads and projects them.
  • The software is designed so that the projected image can be adjusted to include the entire ballot, front and back or any area on either side, so that the slide show can focus on only one race or several.
  • As each ballot is displayed, a bar at the top of the screen displays a number, e.g., ‘Ballot 341 of 412,’ and the ward/precinct, e.g., ‘City of Madison Ward 16.’

The auditors (the official counters) and observers

  • Ideally, the auditors or audience would include some of the elections inspectors who had initialed the ballots, since they will be able to examine the initials on the ballot images and confirm whether they are theirs or not.
  • A team of two official inspectors is assigned to each candidate whose votes will be counted. They sit in front facing the screen with hand-held clicker counters. They should be seated far enough apart, however, that they cannot discern the sound of the click made by their teammate’s counter.
  • Any number of public observers can sit behind the official inspectors and count along.

Checking the number of ballots

If counting for a full reporting unit (precinct, ward), the file directory for the flash drive is projected, the folder for the randomly selected precinct is displayed, and the number of ballot images in that folder is compared to the number of ballots reported to have been electronically tabulated at that precinct.  The numbers should match. If not, it indicates a problem that must be resolved before verification is possible with the digital images. If the problem cannot be resolved, verification with the paper ballots is necessary.

Checking the ballot images for obvious signs of alteration

The first few ballots in each precincts folder are projected and examined for signs of alteration or substitution—do the voters’ marks look natural and normally variable? Are the poll workers’ initials appropriate and natural-looking? If not, it indicates a problem that must be resolved before verification is possible with the digital images. If the problem cannot be resolved, verification with the paper ballots is necessary.

Counting the votes

Starting with the first ballot in the precinct, the ballots are individually projected at a variable rate of 0.5-2 seconds per ballot.

After the first few precincts, we found that 1 second per ballot was best speed. We achieved accuracy more readily at that speed than at 2 seconds per ballot.

The slide show automatically pauses after every 25 ballots, and the two members of each team compare their subtotals for that batch of 25 ballots. If there are no discrepancies, the agreed-upon subtotal is recorded and the slide show continues. If the members on any team disagree, the last 25 ballots are displayed again, and everyone in the room counts for the same candidate. If any votes are ambiguous, the number of the ballot is recorded (e.g., Ballot #78 in Ward 34), but no effort is made to resolve the differing interpretations of the vote. When the two official inspectors agree on the count for that batch of 25 or have identified the ballot on which they disagree, the slide show resumes.

Over both demonstrations, we inspected approximately 1,500 ballots and found none on which the official inspectors and audience members disagreed on how the vote should be counted. We found two ballots that contained idiosyncratic marks that we initially questioned whether the machine would have counted as votes, but for which the voter intent was clear for all who were present.

Determine the accuracy of the outcome indicated by the electronic count

When the slide show reaches the end of the precinct (or the sample), the agreed-upon visually counted subtotals are added and compared to the electronically counted totals reported from that precinct on Election Night (or the expected result for the sample). If they agree, or if the counts are different and the difference can be explained by the ambiguous votes noted during the counting, that finding is noted on the tally sheet and the verification moves on. If they do not agree, the verification also moves on until it becomes apparent that the electronically tabulated results indicated the incorrect outcome.

A well-done risk-limiting audit continues to draw larger samples until the risk that the machines identified an incorrect outcome approaches zero. When the machines counted accurately, this can usually be achieved with fewer than 500 randomly selected ballots. When confident the results are accurate, the verification effort—whether done by citizens or elections officials—should communicate its written findings to the county board of canvass.

However, when the machine has miscounted, it will become evident that no sample will be able to achieve that probability. If the verification is conducted by the county board of canvass, the counting could simply proceed until all votes have been counted. If the miscount is detected during the county canvass period, the finding should be immediately referred to the county board of canvass for resolution under s.7.60(3), Wis. Stats, which allows the county board of canvass to direct the municipal clerk to resolve any noted defects.

If performed by a citizen group, however, the verification could stop when the electronic miscount becomes clear and report their findings to the county board of canvass, who could, if the defect was detected during the county canvass period, order a full hand count of paper ballots—not digital images—in the flawed contest.  If the verification is done after the county canvass, there is no way to correct the flawed election results.

Experience in the first two tests

When two teams of two inspectors (four people) counted votes for two candidates out of a field of five in the mayoral primary, they reached agreement on the visual count for both candidates at a rate of 125 ballots every five minutes, so that a reporting unit with 1,000 ballots could be verified start-to-finish in 40 minutes.

When compared to a hand count, this remarkable time savings came from enabling both members of each pair of inspectors to count simultaneously (cuts the time in half); eliminating the chain-of-custody procedures surrounding unsealing and resealing ballot bags; eliminating the paper-handling and moving (ballot counting and stacking, moving the ballots while counting); freeing up the inspector’s hands to enable them to use a clicker-counter; and enabling them to keep their visual focus on the screen instead of looking back and forth between a ballot and a tally sheet.

Issues relating to using digital images rather than paper ballots

If an audit of election results is to have any value beyond mere appearance, it needs to compare the electronically tabulated election results against an independent record of the votes. That record needs to be true, accurate, and complete. In other words, the record cannot have been replaced by a different record; it cannot have been altered; and no parts can be missing or added.

In addition, that record needs to be verifiably true, accurate, and complete. Steps must be taken from the moment the record is created to reduce or eliminate damage to the record, and a chain of custody must be documented to demonstrate that those security steps were faithfully performed.

Both paper ballots and digital images have security risks that may ruin their value as an auditable record. For both, however, the most important security measure is the most obvious: Someone must look at the record. No security measure of any type will be of any value if no one ever examines the record. A routine practice of sealing the records on Election Night and later destroying them with no review is a security risk in and of itself, because it provides would-be election thieves with solid assurance than tampering will escape detection. Each jurisdiction must be known to have some process—however perfunctory—that ensures discerning human eyes will at some point view any record intended to serve an election-integrity purpose.

Beyond that, the risks and countermeasures are different. If both paper ballots and digital images of those ballots have been handled appropriately, the original voter-marked paper ballots are the superior record, and audits conducted using those are more credible. However, considerations of ease of handling, efficiency, and the reluctance of clerks to access the paper ballots provide arguments in favor of using the digital images.

The risks to paper ballots are well-known. While the digital images are automatically saved to a flash drive and receive no handling or processing after poll closing, paper ballots must be repeatedly handled before being sealed on Election Night. They must be removed from the machine, straightened out, examined for write-in votes, and packaged. Experience has demonstrated risks from accident (valid marked ballots have been left in the machines or elsewhere, or inadvertently discarded) and from deliberate tampering or substitution. Accidents have destroyed paper ballots or rendered them unusable in other ways. Ballot bags have simply gone missing, or been mishandled in ways that ruined the ballots’ suitability as a credible audit record.

The risks to the digital images are similar, through both accident and deliberate sabotage.

Risk #1: The digital images may not be saved, or the files may be corrupted in some way.

(Still to determine: We do not know the reliability of the digital-image preservation function. We looked at the files for approximately two dozen reporting units and saw none in which images had not been saved or were not legible. We detected no reporting units in which the number of saved ballot images was different than the number of ballots cast on Election Day.

In the April 2015 election, the machines for the Town and Village of Black Earth saved no digital images. At this writing in September 2015, the Dane county clerk has not yet determined the cause of the problem.)

Risk #2: Substitution

Ballot images prepared before the election could be either pre-loaded into the voting machine or substituted after the polls close, and matching the correct number of ballot images would be a technically easy task. However, deliberate fraudulent substitution would be easy to detect if the jurisdiction views the digital images at some point

Undetectable substitution would need to use ballots that have been marked to mimic the natural variation in the way voters mark their ballots.  In addition, preparation of an undetectable batch of false images would require impossibly accurate forecasting of details such as which poll workers were going to initial the ballots; the handwriting they would use when they did; and the exact precinct-level results in all contests, not just the targeted race in which results will be manipulated.

An election thief could prepare more convincing false images after Election Day, when preliminary precinct-level results are known in all contests, but the task of preparing the images would need to wait until after preliminary election results are known, and would take substantial time.  False ballot images prepared after the election would have the same need to realistically mimic variation in voters’ marks and poll workers’ initials.

Security measures against substitution:

  1. Because of the difficulty of pre-loading convincing fake images into the voting machines or the flash drives, we do not believe any pre-election steps are needed specifically to prevent this remote risk.
  2. To prevent substitution after Election Day, we are recommending that county clerks make duplicate copies of the ballot images as soon as they are received in the clerk’s office. (The files are large, and after a high-turnout election, copying can be time-consuming.)  The duplicate copies should be promptly distributed to at least a few independent recipients, such as political parties, newspaper, or an independent repository that can prevent undocumented access even by the clerk.

Risk #3: Alteration

The voting machines can create digital images and can recognize which marks are votes for which candidates. Additional programming could be inserted to instruct the machine to:

  • Switch portions of the digital image, so that the marked vote appears beside the desired candidate and an empty oval beside the candidate the voter actually selected;
  • Create additional votes on the digital image without erasing unwanted votes; or
  • Erase unwanted votes from the digital image without creating new votes.

If this process is performed before the machines count the votes, it would need to take place quickly as the ballot is processed, in a way that does not detectably slow the voting process.

?        If a machine was hacked to alter the ballots, how much would it slow down the voting process?

?        How much additional programming would it take, if any, to add an ‘edit’ function to the machine’s built-in ‘read’ function?

?        Can you explain, in layman’s terms, why the programming needed for this hack is so much more difficult than that needed to do nothing more than change the tabulation output?

?        When and by whom could such programming be inserted in the voting machines?

?        If this is possible, I can think of no safeguard against it that doesn’t require a lot more IT security budget, expertise, and authority that the municipal and county clerks will ever have. If altering the digital image is anything more than a highly remote risk, it is the best argument against auditing with the digital image rather than the paper ballots (though still better than no audit at all).

Security measures against alteration:

  1. The county board of canvass should (as they are instructed to do regardless of whether they conduct a post-election audit) remain alert for signs of suspiciously high under-voting in high-profile contests and for signs of suspiciously high over-voting rates. Because Wisconsin’s voting machines are required to reject over-voted ballots, a review of the chief inspectors’ reports will be necessary to notice overvoting problems.
  2. Beyond that, the only way to detect alteration of the images is to match the images to the actual paper ballots.

The DS200 creates no index marks that would allow easy matching of images to ballots, and the task would be time-consuming. If you have to unseal the ballot bags to verify the accuracy of the digital images, you might as well do your audit with the paper ballots.

Risk #4: Damage or destruction

Deliberate action or unintentional error could prevent the images from being saved on Election Day. In addition, the digital image files could be corrupted, damaged, or erased in a way that could prevent their use for verification purposes in numerous ways, either deliberately or by accident.  If the voter-marked paper ballots have been securely stored and safe, however, damage or destruction of the digital-image files need not prevent verification of the election results.

(My understanding is that there is no required retention period for the digital ballot images, since they are just copies of the official record. I think all we need to say is that clerks should retain them until they are sure the election results were counted correctly. )

Security measures to prevent damage or destruction:

  1. Clerks should minimize the window of opportunity for destruction or damage of the digital record by:
  • Promptly examining the digital images to check to see whether they appear to be intact, complete and undamaged.  If they do appear to have been destroyed or damaged, and the damage cannot positively be attributed to known accident or known inadvertent error, it should be taken as a sign that the election results might have been tampered with, and election results should be verified with a hand count of paper ballots.
  • If the digital-image files appear to be complete and undamaged, the clerk should make duplicate copies of the ballot images promptly upon receiving them. The duplicate copies should be promptly distributed to at least a few independent recipients, such as political parties, newspaper, or an independent repository that can prevent undocumented access even by the clerk.

get updates