Viewing entries tagged
audit

Comment

Tabulator Troubles in Colorado

More tabulator troubles! In addition to the continuing saga in New York with the tabulator troubles I wrote about earlier, now there is another tabulator-related situation in Colorado. The news report from Saguache County CO is about:

a Nov. 5 “retabulation” of votes cast in the Nov. 2 election Friday by Myers and staff, with results reversing the outcome ...

In brief, the situation is exactly about the "tabulation" part of election management, that I have been writing about. To recap:

  • In polling places, there are counting devices that count up votes from ballots, and spit out a list of vote-counts for each candidate in each contest, and each option in each referendum. This list is in the form of a vote-count dataset on some removable storage.
  • At county election HQ, there are counting devices that count up vote-by-mail ballots and provisional ballots, with the same kind of vote-counts.
  • At county election HQ, "tabulation" is the process aggregating these vote-counts and adding them up, to get county-wide vote totals.

In Saguache, election officials did a tabulation run on election night, but the results  didn't look right. Then on the 5th, they did a re-run on the "same ballots" but the results were different, and it appears to some observers that some vote totals may be been overwritten. Then, on the 8th, with another re-try, a result somewhat like in NY:

... the disc would not load and sent an error message

What this boils down to for me is that current voting system products' Tabulators are not up to correctly doing some seemingly simple tasks correctly, when operated by ordinary election officials. I am sure they work right in testing situations that include vendor staff; but they must also work right in real life with real users. The tasks include:

  • Import an election definition that specifies how many counting devices are being used for each precinct, and how many vote-count datasets are expect from them.
  • Import a bunch of vote-count datasets.
  • Cross-check to make sure that all expected vote-totals are present, and that there are no un-expected vote-counts.
  • Cross-check each vote-count dataset to make sure it is consistent with the election definition.
  • If everything cross-checks correctly, add up the counts to get totals, and generate some reports.

That's not exactly dirt-simple, but it also sounds to me like something that could be implemented in well-designed software that is easy for election officials to use, and easy for observers to understand. And that understanding is critical, because without it, observers may suspect that the election has been compromised, and some election results are wrong. That is a terrible outcome that any election official would work hard to avoid -- but it appears that's what is unfolding in Saguache. Stay tuned ...

-- EJS

PS: Hats off to the Valley Courier's Teresa L. Benns for a really truly excellent news article! I have only touched on some of the issues she covered. Her article has some of the best plain-language explanation of complicated election stuff, that I have ever read. Please take a minute to at least scan her work. - ejs

Comment

3 Comments

Dude, What Is My Ballot, Really?

(Part 2 of 2: What's My Ballot?) Today, I'm continuing on from a recent post, which compared my in-person voting experience with one method of Internet-based voting: return of marked ballots by fax or email. Next up is a similar comparison with another form of Internet-based voting: Internet voting from home using a PC's Web browser.

Let's briefly recall the result at the end of the day in my polling place: 1. Some paper ballots in a ballot box. 2. Some digital vote totals in a computer, and set of paper rolls that provide a ballot-like paper trail of each voter's activity that led to those vote totals. The paper trails can be used to check the correctness of the digital vote totals. Let's also recall the result at the end of the day with email ballot return: 1. Some printed versions of faxed/emailed ballots, which are treated as ballots for counting purposes. While we're at it, let's recall the results of the old lever machines too: 1. Some mechanical vote totals in one or more machines 2. A hand-recorded paper transcription of the "odometer" readings. (Those machines were a lot harder to move than a computer is! So the transcriptions were the basis for vote totals.)

Now, on to home-based Web i-voting. Before doing the end-of-the-day comparison, let's start with what the experience looks like -- fundamentally, it's Web pages. You point your browser to a Web site; you type in your voter identification, a bit like the in-person poll-book signing experience; and then you get your ballot: one or more Web pages. Various Internet voting products and services differ, but they are all fundamentally similar to something that I bet many readers have seen already: online surveys. Take a look at this simple election-like survey about music in Cuyahoga County. The web page looks like a simple ballot, with contests for vocalist and guitarist instead of governor and dog-catcher. There are candidates, and you vote by selecting one with a mouse click on a radio button next to the name of your favorite.

So far, so familiar, but when I press that submit button, what happens? Where's my ballot? Let's take it step by step.

  • The submit button is part of an HTML form, which is part of the Web page. (You can see the HTML form if you "View Page Source" in your browser.)
  • Pressing the button tells your browser to collect up the form's data, which might include Rachel Roberts for Vocalist if you had clicked the radio button next to Rachel.
  • These parts of the forms data are something that in election lingo you might call a "vote" (or "contest selection" to be precise.)
  • The HTML form data, including the vote-oid data, is sent from your browser to the Web server via an HTTP POST operation.
  • The HTTP transaction is typically via an encrypted SSL session, to preserve privacy en route over the Internet.
  • The Web server passes the POST parameters to some election-specific Web application software, which interprets the data as votes, and stores the vote data in a database.

Now, let's be specific about that database stuff. In surveymonkey, there is a database record for each Cleveland Music survey response, and it's possible (if the survey was set up that way) that the record also includes some information about the person who responded. In actual government voting, though, of course we don't want that. So even though the i-voting server has a database of voters, and even though you had to log in to the i-voting server, and even though you were only allowed to vote if the voter record said you were allowed to vote, still your vote data shouldn't be stored with your voter record. So, the vote data is supposed to be anonymously and separately stored, becoming part of vote totals for each candidate in each contest.

Can you say "odometer"? Okay, maybe it's not that obvious, so let me juxtapose a couple images. As I recounted earlier, a much younger me is standing in the voting booth of a lever machine, looking a big bank of little switches next to candidate names, and thinking that is the ballot. Then the big lever is pulled, the little switches flip back, and it's like the ballot just evaporated! Though of course I was told that the counter dials in the back of the machine did tick over like the odometer on a car, recording each vote. The votes were stored on the odometers, but the ballot was gone without a trace. Now shift the scene to my first surveymonkey experience. I clicked some radio buttons, clicked submit, and poof! what I thought was a ballot just disappeared. I'm told that the counters in a database somewhere ticked over to record my "votes." Again, votes were supposedly recorded, but there wasn't really ever a durable ballot. Home-based web client-server Internet voting is just like that, regardless of varying technical implementation details. There's no durable ballot document.

So, at the end of the day, we have stored vote totals in a database of a system that also logged the voter logins. At that point I don't have an answer to "What's the ballot" anymore than I do for lever machines or the early paper-trail-less DREs. Unlike the (much-more-insecure) email ballot delivery, we don't really know what or where the ballots are. Recalling my experience in the Middlefield Road fire house, the vote data is similarly stored as bits on a computer, but!!! there is also the paper trail. That paper trail can be used to audit the system and detect errors and fraud, and serves as the durable record of the vote -- almost a ballot, except for being on flimsy paper with some ballot information left out. But with i-voting, there is nothing even similar. Any kind of auditing that's done, is done using data saved on the server computers, rather than looking at a ballot document that the voter also saw.

Is that so terrible? Maybe so, maybe not. A durable ballot is not a holy requirement for U.S. elections -- though in some parts of the country it almost is. And a durable ballot may not be a requirement for a voting system that is specifically and only for timely assistance of overseas and military voters. Such requirements are a matter of local election law and decisions of local election officials. But my critical observation here is about voter trust. Trust derives in large measure from comprehension. And for many voters, a voting system is comprehensible if the voter knows what the ballot is, where it goes, and what happens to it. That's why overseas voters like fax and email return. Despite the security and anonymity problems, the voter understands that ballot, how it pops out of the fax/printer on the other side of the planet, and how its counted as a paper ballot. The same can't be said for paperless home-based i-voting. As a consequence, I think that it will be harder to build trust, at least in some parts of the country that are paper-centric. However, it may be less of a big deal if limited to overseas and military voters, whose main concern is "get the the ballot home in time to be counted." The pilots are happening, and time will tell.

-- EJS

3 Comments

3 Comments

Dude, Where's My Ballot?

I just finished voting in CA's primary -- whew! 47 contests, 76 candidates total, and for on-paper voters, 4 sheets! But today, instead of hand-marking a ballot (my preference explained in an earlier posting), I used a DRE. This voting machine is part of the voting system that San Mateo County purchased from Hart Systems, the smallest of the 3 remaining vendors with a significant share of the U.S. voting systems market. Comparing with people voting on paper or turning in vote-by-mail packets at the polling place, I had to ask myself the question: where's my ballot? The answer is in two parts.

As a techie, part of my answer is that an electronic version of my ballot is stored as bits on magnetic storage inside one of the computers in the polling place. It may or may not be not be a "ballot" per se (a distinct collection of selections in the contests), but rather just votes recorded as parts of vote total, analogous to the odometers on the old lever machines. As jaded techie, this strikes me as not the most reliable way to store my ballot.

However, as an observant voter, I can also see that my ballot is also represented by the "paper trail" on the voting machine. As an informed voter (a trained poll worker who also talks to local election officials), I know that this paper is used by election officials as part of auditing the correct operation of the computers, by manually tabulating vote totals for a handful of randomly selected precincts -- an extremely important part of the election process here. However, as a jaded observant voter, the cheap paper roll (like a gas station receipt printer) strikes me as not a very durable way of recording the ballot information that I could have put on nice solid real paper ballots.

But leaving aside questions of paper stock, the combination of the two ballot recording methods is pretty good, and the audit process is great! Though I have to say: my thanks and condolences go to the hard working San Mateo County elections staff who wield scissors to cut the paper rolls into individual ballot-oid papers to be hand-tabulated in the audit.

So, as a paper ballot fan, I left reasonably satisfied, though glad of the ability to vote on paper in November. It's a bit of a conceptual leap to go from a tangible paper ballot in a locked ballot box, to the above non-short answer to "Where's my ballot?" But it's a leap that I think many voters can be satisfied with, or would be if the paper trial items actually looked like ballots (as in the system we're building at TrustTheVote). But it got me thinking about some of the overseas-voter Internet voting pilots I've been reading about. That's enough for today, but a good question for another day, about Internet voting, is the same question, "Where's my ballot?" More soon …

-- EJS

3 Comments

Comment

Childhood Ballot Confessions

I have to admit, I like paper ballots. But it wasn't always that way. As a small child, I remember going into the voting booth with a parent, and watching them use those fine old lever machines. They were cool. The curtain made it seem like something both secret and important was happening. The little flippy switches made a satisfying little "tick" sound when you flipped them down to make a selection, and nice "tock" sound if you changed your mind and flipped one back up again. And of course the best part was hearing the thing click and clack after you pulled the big lever. But although it was cool as a machine, and the whole voting thing was groovy, I had a twinge while looking at the little floppy switches flipping themselves back up again. It was like all this important secret stuff we did in the booth … just sort of evaporated. Sure, the clicking and clacking was the machine "remembering" each vote, but it was odd to see.

Years later when I started to vote myself, I found it very satisfying and reassuring to be using a paper ballot, especially after the run-around I got trying to vote for the first time. I felt more confident seeing a durable ballot recording my votes, and not evaporating.

More recently, experimenting with using a Direct-Record Election device (DRE), it was back to the future, with the ballot evaporating again -- and without even seeing a ballot per se, like front panel of the old lever machine. As Doug Jones wrote here recently, our touch-screens are digital DREs just as the lever machines were mechanical DREs. The little paper tapes were certainly an improvement, but flimsy enough that it was a small improvement. If you're going to print something for me, please have it be a real paper ballot, I thought the first time. So, I now understand that I like the approach of ballot-marking devices used by those that aren't able to or don't wish to mark by ballots by hand.

Is there a point to this personal history of feelings about ballots? A small one, both a link back to my posting about eMailed ballot return, and a future one on Internet voting. The point is that I think that voter confidence depends in part on the voters' understanding the voting method that they are using. If you ask or allow voters to do something new, but which seems similar to voting that they already understand, then they can "get it" -- which is why eMail return makes sense for voters because it's like vote-by-mail that they understand. So, if people are used to a ballot -- as I am -- then a change is going to make the most sense if I can still understand where the ballot is, and what happens to it.

-- EJS

Comment

Comment

Kudos to Cuyahoga

I was very encouraged by recent election news from Ohio's Cuyahoga County, reported in the Cleveland Plain Dealer newspaper: Reason for election machine glitch found, officials expect things to be OK for the primary. At first blush, it might seem like bad news:

All told, 89 of the Cuyahoga County Board of Elections'  1,200 machines powered down and then froze during a specific test done to ensure the optical scanners were reading paper ballots correctly.

But as the Plain Dealer's Joan Mazzoli said in her folksy headline, the point is that the Cuyahoga BOE disclosed to the public exactly the problems that they encountered, the scope of the problems, and the possible effect on the election that will be conducted with machines that they tested. In fact BOE director Jane Platten explained the specific error logging procedures that they will use to audit the election, to make sure that no votes are lost even if the machines malfunction during the election. As Mazzoli quoted Platten:

We want to ensure everyone's votes are counted.

Of course that is every election official's goal, but this news is about a bit more: making sure that everyone's votes are counted, and "making sure" in a way that the voters can see and believe in despite known problems with the election technology being used. That is what I call helping people Trust the Vote, and for that I say - Kudos to Cuyahoga!

-- EJS

Comment

Comment

How to Trust a Voting Machine

[Today's guest post is from election technology expert Doug Jones, who is now revealed as also being an encyclopedia of U.S. elections history. Doug's remarks below were in a discussion about how to effectively use post-election ballot-count audits as a means to gain trust in the correct operation of voting machines -- particularly timely, given the news and comment about hacking India's voting machines. Doug pointed out that in the U.S., we've had similar voting-machine trust issues for many years. -- ejs] Lever machines have always (as used in the US) contained one feature intended for auditing:  The public and protective counters, used to record the total number of activations of the machine.  Thus, they are slightly auditable.  They are less auditable than DRE machines built to 1990 standards because they retain nothing comparable to an event log and because they do not explicitly count undervotes -- allowing election officials to claim, post election, that the reason Sam got no votes was because people abstained rather than vote for him.  (Where in fact, there might have been a bit of pencil lead jammed in the counters to prevent votes for Sam from registering).

One of the best legal opinions about mechanical voting machines was a dissenting opinion by Horatio Rogers, a Rhode Island supreme court judge, in 1897.  He was writing about the McTammany voting machine, one that recorded votes by punching holes in a paper tape out of view of the voter.  I quote:

It is common knowledge that human machines and mechanisms get out of order and fail to work, in all sorts of unforseen ways. Ordinarily the person using a machine can see a result.  Thus, a bank clerk, performing a check with figures, sees the holes; an officer of the law, using a gibbet by pressing a button, sees the result accomplished that he sought; and so on ad infinitum. But a voter on this voting machine has no knowledge through his senses that he has accomplished a result.  The most that can be said is, if the machine worked as intended, then he has made his holes and voted.  It does not seem to me that this is enough.

I think Horatio Rogers opinion applies equally to the majority of mechanical and DRE machines that have been built in the century since he published it.

-- Doug Jones

Mandatory disclaimer:  The opinions expressed above are mine!  The various institutions with which I am affiliated don't necessarily agree.  These include the U of Iowa, and the EAC TGDC. - dj

Comment

1 Comment

Transparency, Voting Machines, Choices

Today I provide the next step in clarifying TTV goals in relation to discussions with election transparency advocates. Regarding the previous posting, I want to emphasize that voting  machines -- in this case we focus on paper ballot scanning machines -- are a transparency problem, if there is no human involvement in counting paper ballots, and the public has no access to audit records of the counting process. Even with current systems, election officials can choose to mitigate these difficulties; and as I said before, we will deliver to them some technology that can make that a lot easier to do. Today, I wanted to talk about choices. In discussion about voting machines as part of the problem, it seemed like TTV might also be part of the problem too, because we are failing to advocate for either or both of the use of hand counting of paper ballots, or abandoning the use of paper ballot scanning devices. So let me be clear about that: it is true that we are not advocating for those positions, not influencing legislators to make such changes in election law, and not advocating that election officials should make those particular changes in their election methods. Such advocacy work may be to the public benefit, and is rightly performed by activists and advocates.

The choice is with election officials, on how to use available technology. In making available some new paper-ballot-counting technology, we are not advocating that a particular voting method be used. I've listed several voting methods below, as illustration of many choices that election officials could make, all of them choices in which new voting technology could be used, and could help with transparency. With the exception of advocates of completely zero machine count usage (and that is a worthy topic for another day), we hope that advocates of many positions might extend the benefit of the doubt that our efforts can help, at a minimum with some interesting "side effects" that I'll discuss next time.

-- EJS

PS: Here is that list of several kinds of voting methods:

  • Polling-place machine-counted paper ballots, centrally machine-counted other ballots, and minimum 2% partial hand-counting in a risk-limiting audit methodology;
  • Similar, but 100% hand-counting, for full benefit of each co-eval counting method checking the other (consilience), and a standard methodology for auditing and resolving differences;
  • Hand-counting, with machine-counting for consilience benefits in recounts, and in automaticly triggered audits of contests above a specified "close result" level;
  • Polling place electronic voting (no paper ballots), centrally machine-counted vote by mail ballots;

As you can see, that's a broad range, and with variants of each, there are dozens of choices. Paper ballot scanning/counting devices have a role in each, and do not preclude any of these choices. Again: 100% hand count, 0% machine count is a separate topic I promise to get to.

1 Comment

1 Comment

Virus in NY Voting Machine? Not Really

The reports of computer viruses in NY voting machines -- though spurious -- cause me to return to a basic mantra of TrustTheVote: we do technology development so that election tech helps inspire public confidence in elections, rather than erode it. The NY case is a great example of erosion, but also a cautionary tale for future inspiration. The caution comes from the significant and ongoing confusion about the term "virus". But first, the situation in question arose in Hamilton County, NY, part of the hotly contested NY 23rd Congressional District race between Hoffman and Owens. It's an ugly scene, because the vote was close, it's already certified, Owen is seated, but re-canvassing efforts highlighted some counting irregularities. These weren't large enough to effect the race, but were enough to spark Hoffman to un-concede defeat, and to issue a letter with some really disturbing claims of the election having been stolen. Now, add to this the claim that the election result is further tainted by the discovery of a computer virus in the voting system used in Hamilton. That's a real example of tech digging the confidence hole that much deeper - ouch!

But the really sad part of this, for me, is that the true story is a good story about election officials doing the right thing: when they found a software bug, they worked with the vendor and created an effective work-around -- maintaining the integrity of the system, the exact opposite of the story about the virus undermining the system. The real virus is that spurious story! The details, provided by NY State election official Doug Kellner, also provide another example of complexity of diligent election administration:

In pre-election testing several counties discovered the Dominion ImageCast machines froze when fed ballots that contained contests with multiple candidates to be elected.  It was determined during the week before the election that the cause was a source code programming error in the dynamic memory allocation of the function that stores ballot images--not the counting function.   Although only one line of source code needed modification, NYSBOE staff properly refused to approve any modification of source code without proper certification.  Dominion developed a work-around by changing the ballot configuration file--not the source code so that the machines using the new configuration files functioned on election day.  It is my understanding that a few county officials, who were using the machines for the first time, did not properly revise the configuration files and the machines were used in emergency ballot mode--that is, ballots were inserted in the emergency ballot boxes contained within the machine and were counted manually after the close of the polls.

Kudos to NY for doing their job right, in the real world of flawed equipment, not the fantasy land of viruses and stolen elections. New Yorkers should be thanking the NYSBOE for a job well done!

-- EJS

PS: For a detailed debunking of the virus claims, see the blog of NY election tech expert and advocate Bo Lipari. It's excellent. It got picked up in local press. But it can't catch up to the idea virus, as the tale continues to mutate through the blogosphere that Hoffman was cheated by corrupt election officials, or ACORN, or computer hackers, or viruses, or some combination. ;-(

1 Comment

Comment

Pennsylvania Paperless Recount

I wrote before that this month's re-count activity in Pennsylvania was notable because of the variety of voting methods used there, and hence the variety of recounting methods needed. In contrast to the Lackawanna county that I mentioned specifically, there are many counties in PA that use completely paperless DRE voting machines. In these cases, there are no actual ballots to recount, nor are there paper-trail tape-rolls to examine. As a result, the recount is more a matter of re-obtaining the vote totals from the DREs, re-doing the tabulation that adds up the machines' vote totals for the recounted contest, in order to re-compute the election result. This is similar in principle to re-counts of PA's old lever machines, where the re-count involved re-inspection of counters on the back of each lever machine. One difference in practice, though, is that the lever machine counters could be directly inspected by a person, who would have little doubt that the totals they gather from each machine were in fact recorded by that machine. The DRE's vote totals are stored re-writable digital storage media that are often separated from the machine itself. And as we saw recently in Myrtle Beach, human error can play a role in that separation.

So, election geek that I am, I'm waiting with interest to hear about the various re-counting methods used, the variances found, how the variance get accounted for, and so on. It should be a very interesting comparison of different means to the same end that one Lackawanna County candidate expressed so well:

Every vote should count. It's hard enough to get the people to come out and vote. ... The election process is under shadow.

Removing that shadow is what PA officials are working hard to do in a scant week of efforts, that along with the efforts of many public-spiritied observers, could teach us all a lot about how recount methods can create transparency as restore trust that every vote counts.

-- EJS

Comment

4 Comments

Levers, HAVA, and "Compliance"

Kudos to Brad Friedman for making a good call on a subtle point in his comment on my posting about Bo Lipari's coverage of the NY State testing of voting systems. Brad objects to my statement that lever machines are not compliant with the Help Amercia Vote Act (HAVA). And rightly so! The bad news about the adjective "HAVA compliant" is that people can and do disagree about the interpretation of that Act of Congress. The good news is that the noun "HAVA compliance" is well defined by facts on the ground, if not in the Act itself.

Those facts on the ground are composed of each state's implementation of its HAVA compliance plan, under the oversight of the U.S. Department of Justice. The DoJ has for years been working with states, including the lever-machine states of NY and CT, on each state's HAVA compliance plan. Those plans in NY include the use of machine-counted paper ballots, some hand-marked, and some from ballot marking devices that provide enhanced access for voters who are unable or unwilling to mark paper ballots by hand. Those plans do not include the continued of lever machines.

So we can say that lever machines are not part of HAVA-compliance (noun) in NY or CT.

Further, I got the impression, from talking to folks involved in HAVA compliance program implementations, that there was no chance of a compliance program being approved if it was based on the continued use of lever machines. If true, that might well based on what Brad would consider a misinterpretation of HAVA.

Would it be possible for a state to have an acceptable HAVA compliance plan that included lever machines? Perhaps a plan that included electronic DREs for enhanced access, lever machines (which are mechanical direct-record election devices), and tools for combining the results from both into an auditable election result? Possibly, but likely we'll never know, as the last few HAVA-compliance program engines pull into the station at the end of ride.

-- EJS

4 Comments

Comment

Identifying the Gold, Redux

I recently commented on specific connection, in the case of the TrustTheVote project, of open source methods and the issue of identifying a "gold build" of a certified voting system. As a reminder to more recent readers, most states have laws that require election officials to use only those specific voting system products that were previously certified for use in that state -- and not some slightly different version of the the same product. But recently, I got a good follow-up question - what is the role of the Federal government, in this "gold build" identification process? There is in fact an important role, that is potentially very helpful, and where openness can help magnify the benefit of this helpful role of the government. Here's the scoop. The EAC has the fundamental responsibility for Federal certification, which is used in varying degrees as part of some states' certification. Testing is the main body of work leading up to certification. Testing is performed by private companies, that have qualified in a  NIST-managed accreditation program as an official Voting Systems Test Lab. There are two key steps in the overall process in which a test lab verifies that it can re-do the "trusted build" process to re-create the soon-to-be "gold" version, so long as the lab can verify that the trusted build process did in fact re-create the same exact software that was tested. Then, as the EAC Web site briefly states: "Manufacturer provides software identification tools to EAC, which enables election officials to confirm use of EAC-certified systems."

But here is the fly in the ointment: for your typical PC or server, this is not easy! and the same is true for current voting systems. Yes, you could crack open the chassis, remove the hard drive, examine it as the boot medium, re-derive a fingerprint, and compare the fingerprint to something on the EAC web site. But in practice this is not going to happen in real election offices, and in any case it would be fruitless -- even if you did, you would still have no assurance that the device in the precinct was still the same as the gold build, because the boot media can be written after the central office tests the device, but before it goes into use in a polling place.

That's quite an annoying fly in the ointment, but it doesn't have to be that way. In fact, for for a carefully designed dedicated system, the fingerprinting and re-checking can be quite feasible -- and that applies to carefully made voting systems too, as we've previously explained. Such carefully made voting systems would be a real improvement in trustworthiness (which is why we're building them!), but they aren't a silver bullet, since you can never 100% trust the integrity of a computing system. That's why vote tabulation audits are an important ingredient, and why I periodically bang on about auditing in election processes.

-- EJS

Comment

Comment

Stalking the Errant Voting Machine: the Final Chapter

Some readers may sigh relief at the news that today's post is the last (for a while at least!) in a series about the use of vote-count auditing methods to detect a situation in which an election result was garbled by the computers used to create them. Today, a little reality check on the use of the the risk-limiting audit methods described earlier. As audit guru Mark Lindeman says,

Risk-limiting audits clearly have some valuable properties, yet no state has ever implemented a risk-limiting audit.

Why not? Despite the rapid development of RLA methods (take a quick glance at this paper to get a flavor), there are several obstacles, including:

  • Basic mis-conceptions: Nothing short of a full re-count will ever prove the absence of a machine count error. Instead, the goal of RLA is to reduce risk that machine count errors altered the outcome of any contest in a given election. Election result correctness is the goal, not machine operations correctness -- yet the common mis-perception is often the reverse.
  • Requirements for election audits must be part of state election laws or regulation that implements them. Details of audit methods are technical, and difficult to write into law -- and detailed enough that it is perhaps unwise to enshrine in law rather than regulation. Hence, there is some tension and confusion about the respective roles states' legislative and executive branches.
  • Funding is required. Local election officials have to do the work of audits of any kind, and need funding to do so. A standard flat-percent audit is easier for a state to know how to fund, rather than a variable-effort RLA that depends on election margins and voter turnout.
  • The variability itself is a confusing factor, because you can't know in advance how large an audit will have to be. This fact creates confusion or resistance among policy-makers and under-funded election officials.
  • Election tabulation systems often do not provide timely (or any) access to the data needed to implement these audits efficiently. These systems simply weren't designed to help election officials do audits -- and hence are another variable cost factor.
  • Absentee and early-voting ballots sometimes pose large logistical challenges.
  • Smaller contests are harder to audit to low risk levels, so someone must decide how to allocate resources across various kinds of contests.

As Lindeman points out, each of these problems is tractable, and real progress in RLA practice can be made without a solution to all of these problems. And in my view, one of the best ways to help would be to greatly increase transparency, including both the operations of the voting systems (not just the tabulation components!), and of the auditing process itself. Then we could at least determine which contests in an election are most at risk even after the audits that election officials are able to conduct at present. Perhaps that would also enable experts like Lindeman to conduct unofficial audits, to demonstrate effectiveness and help indicate efforts and costs for official use of RLA.

And dare I say it, we might even enable ordinary citizens to form their own judgement of an individual contest in an election, based on real published facts about total number of ballots cast in a county, total number of votes in the contest, margins in the contest, total number of precincts, precincts officially audited, and (crank a statistics engine) the actual confidence level in the election result, whether the official audit was too little, too much, or just right. That may sound ambitious, and maybe it is, but that's what we're aiming for with operational transparency of the voting system components of the TTV System, and in particular with the TTV Auditor -- currently a gleam in the eye, but picking up steam with efforts from NIST and OASIS on standard data formats for election audit data.

-- EJS

Comment

Comment

What's an RLA? What Does a Good One Look Like, and Why Would I Care?

Recently I've made a series of posts seemingly obsessed with chanting "audit, audit, ..." mantra-like, to put readers into a trance. For those of you still awake enough to want to know how to find out whether election results were garbled by the computers used to create them, today we have some more answers. The key word is "risk limiting audit" and here today to explain it is election expert Mark Lindeman of Bard College. Over to Mark ...

Many observers agree that electronic voting machines and optical scanners cannot be assumed to count votes accurately. A commonly proposed solution is to audit -- through a hand count -- a random sample of the paper ballots (or, perhaps, voter-verifiable paper records) from each election. For instance, California mandates a hand count in 1% of all precincts. Intuitively, if a "large enough" random sample of ballots uncovers no material counting errors, we can be confident that the count is accurate. But what counts as "material," "accurate," or "large enough"? Risk-limiting audits offer one answer: they are designed so that if miscounts have altered an election outcome, the wrong outcome is likely to be corrected via a full hand recount. "Likely" is not a weasel word: it is specified as a guaranteed minimum probability. If an audit guarantees at least a 99% chance of correcting an outcome when it is wrong, then we can say that there is a 1% risk level of of an undetected wrong outcome.

"Risk-limiting audits" may sound like a no-brainer, but most existing audits come nowhere near this standard. There are at least two big problems. One is getting a reasonable sample size. Suppose for a moment that in order to alter some election outcome, at least half the election precincts would have to be miscounted. At that miscount rate, a random sample of just seven precincts has about a 99% chance of including at least one miscounted precinct -- whether the contest being audited is in a single congressional district or an entire large state. Now suppose that miscounts in just 5% of election precincts could alter the outcome. To get that same 99% chance of detecting some miscount, one needs to sample about 90 precincts -- again, whether one is auditing a single CD or all of California. (The numbers do decrease for smaller contests, but not as fast as many people expect.) So a "1% audit" may be far larger than needed to confirm who won an election, or it may be far too small, depending on the size of the contest and the winning margin, among other things. Changing the percentage doesn't solve the problem, but only alters the balance of "too-large" and "too-small" samples.

Another big problem with existing audits is the gap between detecting miscounts and actually correcting incorrect outcomes. If an audit detects a miscount, what happens? In many states -- including California -- nothing happens. Some states do provide that sufficiently large miscounts lead to larger audit samples, and perhaps eventually to full recounts. A mere sample can never correct an incorrect outcome. But even the best of these rules is not very good: they don't always count more when they should, or they sometimes count much more than necessary to confirm election outcomes. This is not to say that fixed-percentage audits are useless, but they aren't tailored to the task of efficiently detecting and ultimately correcting most incorrect outcomes.

Many thanks to Mark for this explanation! Coming soon: practical use of risk limiting audit, and possibilities for DIY.

-- EJS

Comment

Comment

Voter Registration, Fraud, and Transparency

In this week's news we have a classic example of how transparency (a.k.a. "open government") has enormous potential to defuse some thorny political issues that can rise to the highest heights of U.S. political news.  The news is about Karl Rove's involvement in Bush-administration actions to dismiss some U.S. Attorneys, including David Iglesias. A New York Times article E-Mail Reveals Rove’s Key Role in ’06 Dismissals describes how Iglesias lost favor with the Bush Administration as a result of being perceived to be slack in pursuing cases of possible voter fraud.  In a PBS Interview, Mr. Iglesias described exactly what type of voter fraud was at issue, and how his investigation indeed sought, but did not find evidence of fraud to be prosecuted.  And the connection with voter registration?  The potential fraud in question was voter registration fraud. Mr. Iglesias said that New Mexico state GOP officials

singled out ACORN  as an entity that they thought was engaging in ... a plan to register individuals who were not legally entitled to vote... under-aged people, people who perhaps were felons, people who perhaps were not American citizens.

The concern was that if such fraud were occurring, then it would enable the further fraud of actual voting by people who were fraudulently registered and had no legal right to vote. If that were to occur, the election result could be swung -- particularly of concern in NM, where the 2000 presidential election hung on 344 votes -- and even worse, one couldn't be sure because of the inability to know after the fact how these hypothetical illegal voters actually cast their ballot.

That's serious stuff. Again, you may be asking what's the connection to voter registration systems technology. Well, consider the effect of lack of transparency. Mr. Iglesias' efforts were based on information not readily available to the public, or to his detractors inside the beltway. As a result, there was real angst over ACORN's activities and a possible conspiracy to swing a Presidential election. And that information vacuum was a factor in feeding the conspiracy theorists who may eventually have helped the process of sacking Iglesias.

Now, imagine a world in which there is, in fact, quite readily available information about [a] the entire stream of voter registration requests, [b] source of requests (e.g., individuals, ACORN, Rock the Vote, etc.), [c] county officials' adjudication of those requests, [d] results of adjudication, etc. Suppose that a state could easily generate reports about this stream for officials (e.g., States' A.G. or Federal DoJ,), and even openly publish redacted versions of these reports or even the raw data.

Well, that's what we're building in the TrustTheVote Project. And that transparency is (and should be) what "open government" is about.  If that transparency had been the case in NM a couple years ago, then the information vacuum would not have existed, except as willful refusal to examine readily available information.

And that's where open-source, open-data, operationally transparent, "people's technology" can be the basis for real IT systems that can fill information vacuums and defuse conspiracy theories -- helping to increase the health of public discourse. Yes, it sounds a bit highfalutin, idealistic, so don't take it from me -- let us prove it with real running code and stuff people can see, touch, and try.

-- EJS

Comment

Comment

Identifying the Gold: Does Open Source Help?

A good question re-surfaced for us as we participated in the National Civic Summit recently. The issue was and remains about identifying a "gold build," that is, when there is a particular system/version that is certified for use as a voting system, how should election officials know that the systems that they deployed are systems that are an instance of the certified system. Previously, we provided some answers of how you could answer the question "How do I know that this voting machine is a good one" and provide in the wiki on a more technical treatment of "field validation" of voting system devices. But the  slightly different question that arose recently is: how does open source help?

The simple answer is that  open source techniques do not directly help at all. We could build a completely open system that has exactly the same architectural blockades to field validation as the current vendors' product do. However, the TrustTheVote open source project has some advantages. First, we're working on voting systems, which have sufficiently simple functional requirements (compared to general purpose computing systems) that field validation of voting devices isn't as difficult as in the more general case. *

The second advantage allows us to sidestep many of these complexities, given the relative simplicity of voting devices. We were able to  go back to the drawing board and use an architecture that simplifies the field validation problems, for the very specific and limited class of systems that are voting devices.

Openness itself didn't create these two advantages; but in conducting a public works project, we have the freedom to start fresh and avoid basic architecture pitfalls that can undermine trust. Therefore, the value of working openly is that the benefit of this work -- increased confidence and trust -- is a bit more easily achieved because field validation is fundamentally a systems trust issue, and we address in a way that can be freely assessed by anyone. And that's where the open source approach helps.

-- EJS

* NOTE: for the detail-oriented folks: in general, the twin problems of Trusted Software Distribution and Trusted System Validation are, in their general form, truly hard problems. Feasible approaches to them usually rely on complex use of cryptography, which simply shifts the burden to other hard problems in practical applied cryptography. For example, with "code signing" my Dad's computer can tell him that it thinks he should trust some new software because is it signed as being from his SW vendor (e.g., Microsoft or HP); but he wonders (rightly) why he should trust his computer's judgment in this matter, given the other mistakes that his computer makes. For more on the non-general voting-system-friendly solution, see the TrustTheVote wiki: https://wiki.trustthevote.org/index.php/Field_Validation_of_Voting_Systems

Comment

Comment

Can We Really Detect Flakey Voting Machines?

That's a catchy blog headline, I hope, or at least an important issue. But I've fooled you because while answering the question, I am going to discuss "audit" again. I wrote earlier that one kind of audit is performed by election officials to detect errors in voting machines, or to put it another way, to ensure that election results weren't garbled by the computers used to create them. That sounds like a good thing to detect, and ensure, but how can we understand whether the detection is effective? Today's post is the beginning of an answer to that question. And it's a very relevant question, because we know from last year's experience in Humboldt County CA that malfunctions do occur. In fact, with just the right bad luck in the locales affected, perhaps only half a dozen Humboldt-sized, Humboldt-style glitches would have been required to swing MN's close Coleman-Franken race.  And recall that each county has hundreds of opportunities for such a glitch! Five or ten malfunctions per thousands, across a medium-sized state, may not sound like a lot, but its enough to swing a major contest every few years.

To take a specific example, let's look at the voting method of paper ballots, counted by machine partly  in polling places and partly in a central facility. (Similar issues apply to other voting methods including those using touch-screens or other direct-record devices.) One audit procedure is essentially a hand-count "spot check" or partial "re-do" of the machine count. Precincts are randomly selected to get a set of precincts with enough combined ballots to exceed some threshold percentage of the vote, say 1%. Then each of these precinct's ballots are re-counted, for each contest, and the hand-count results compared to the machine count. There are often small variances -- different interpretations by people and software -- and these are scrutinized and documented to ensure that are in fact borderline interpretation cases or due to some other procedural, non-technical issue. Any substantial variation would be a sign of some potential machine malfunction, and would trigger further hand counts until the rules for the audit process are complete, or a full re-count is triggered by the audit procedure rules.

Fair enough, but in the typical case where 1% of a county's paper ballots have been audited with no errors detected, what do we actually know? How confident can we be that the remaining unaudited ballots were correctly  machine-counted? What if a race is pretty darn close, say 2% margin of error, but not so close as to trigger a recount; if 1% of ballots were audited, what can we expect about the other 99% of ballots, and the chance that machine counting errors might change the election result?

Yes, I started a general question, and answered it with some more specific questions. But at least I didn't bore you with too much more of the A-word. Coming soon, another post that answers the questions remaining from today, by explaining in simple terms what a "risk limiting audit" is, how it is different from the flat-percentage audit discussed today, and, finally, how you can tell for any election you want, whether the election officials were able to test whether election results were garbled by the computers used to create them.

-- EJS

Comment

Comment

A Rose by Any Other Name Would Smell as Sweet ... But What About "Audit"?

Pretty much any time the word "audit" gets used to describe a process of double-checking something, or getting a second pair eyes ... the eyes glaze over. Despite Shakespeare's adage, calling something "audit" is pretty much the kiss of death for interest value. But what I want to do today is talk about the A-word in the context of elections, to separate out several of the confusingly many meanings of the term, and focus on one meaning of the term -- the one where you should be interested if you want to understand how election officials try to ensure that an election result is correct -- including ensuring that the results weren't garbled by the computers used to create them. I hope that's an important enough issue to keep you reading a couple more paragraphs. Here some of the many uses and abuses of the term:

  1. Each county is subject to review of the "election evidence" that includes the county level results, ballot info, and loads of records of how the election was conducted, e.g. poll books, paper tapes from voting machines. The A-word is mis-used to describe these review activities of a state canvassing board, the body that decides to make election results official by "certifying" them. Such reviews rarely include scrutiny of loads of election evidence, but in close races or cases of apparent irregularity, there can be plenty of scrutiny. Canvass board review is part of every election, regardless of the way that voting is performed.
  2. During a re-count, a county or state can sometimes bring in outside professionals to help perform or assess the procedures for conducting a recount. These pros sometimes are employees of big name accounting firms that also do corporate financial audits. A-word creeping in again.
  3. There is a particular auditing practice of counties that use computer-counted paper ballots. A more accurate technical term is "risk limiting audit". Though this term is, over and above the kiss, the full-body embrace of death for interest value, it's very important if you want to understand the potential drawbacks of using machine-counted paper ballots. More on this later.
  4. Lastly, the term audit has been used to describe a process in which independent parties -- not election officials or deputed volunteers, or contractors from audit firms -- can gain access to and use information about what election officials did to ensure correct election results. The best-known recent example is the Humboldt County Transparency Project's work towards enabling members of the public to re-do the county's process of counting the ballots. In it's recent Senate race recount, Minnesota also took some notable steps forward by showing the public the wacky mail-in ballots that they had to decide whether or not to count, and letting the public know what their decisions were.

Now, here is the kind of audit that is the most worth understanding and working towards, in the spirit if full transparency of election operations: something like #4, but with the scope of #1, that is, public access to the greatest legally allowable amount and kind of information that allows us to see in detail what our public servants did in conducting the election - warts and all, because no election is ever done perfectly. Sound like a big job? Sure, but computers are pretty good at storing, tracking, and publishing all kinds of information; so as we progressively computerize elections (yes, it's happening, whether or not you or I like it), those systems can in fact enable transparency at that very detailed level. At least, that's what we believe at TrustTheVote, and we'll try to prove it by making the technology and getting it out there for real use. We'll see!

-- EJS

PS: Coming soon: more on the A-word - really understanding the boring-sounding "risk limiting audit" and whether it actually helps ferret out errant voting machines. Not a rosy story, but more interesting than the A-word I hope!

Comment

NYT Letters: Searching for a Reliable Voting System

I can't resist calling your attention to some very thoughtful letters to the editor of the New York Times, on the topic of voting technology. For starters, the NYT had very straightforward editorial with trust as the keyword How to Trust Electronic Voting that started with the opinion that "Electronic voting machines that do not produce a paper record of every ballot cast cannot be trusted" and which advocated the adoption of the voting method that is variously called "hybrid" or "machine independent" or "optically scanned paper ballots." More recently, the NYT published a letters article Searching for a Reliable Voting System (note the shift from trust to reliability). All five comments are worthy, but span a fascinating range of views that I have to share briefly, especially since some of them are a great tee for a future posting ;-)

  • Mitch Trachtenberg pointed out that paper ballots are helpful, but are much more helpful if the paper ballots are available for independent counting "whether by hand of by computer assisted projects like the Humboldt County Election Transparency Project."
  • Daniel Cannon also pointed out that paper ballots by themselves are not sufficient for trust. Cannon wants measures for trust in the counting technology, including technology transparency (full disclosure from vendors) to address concerns of technical methods for stealing elections.
  • James Stevens, by contrast, says that since the paper ballots are not actually hand counted, we're still inherently trusting the technology, which can be rigged. Stevens calls for hand count, with no technology. (Interestingly, the hybrid scheme provides a half-way point in, where partial hand counting is used to detect flaws in the electronic counts -- which is something that Cannon calls for as well.)
  • Henry Finkelstein points out that paper ballots are easily tampered with. Hand counting created tampering concerns that led to voting machines in the first place -- the mechanical "lever machines" that work fine without computers.
  • John Smith agrees with the need for paper audit trails, but isn't sold on optically scanned paper ballots because of the cost of paper ballots, especially the need to print enough to avoid running out in polling places.

What a range of opinions! And the oddest part is that each of these 5 remarks has merit, despite seeming contradicting one or more of the others -- yet another poignant testament to the inherent squirrelly-ness of U.S. election practices and technology.

And with such poignancy, I will have to save for another day the pleasure of doing a Stevens-vs.-Finkelstein face-off posting, or a posting of counterpoint to Smith explaining why dollars per voter per election for paper ballots isn't so terrible compared to existing alternatives, to say nothing of cheaper near term alternatives.

-- EJS