Poor software design, serious IT management inefficiencies and an untested deployment of a critical application were all major factors in last month’s Election Day problems in Denver, according to a scathing report from an IT consultant. The problems led to hours-long delays for voters looking to cast ballots and raised questions about the overall efficacy of e-voting.
The 32-page report, released Monday, concluded that the main reason for problems was the electronic poll book (ePollBook) software used by the independent Denver Election Commission (DEC) to oversee voting. The e-poll book software — an US$85,000 custom application created by Oakland, Calif.-based Sequoia Voting Systems — included the names, addresses and other information for all registered voters in Denver.
Sequoia was already a voting services vendor to the city and county, and the application was designed to allow poll workers across the Denver area to check off voters as they came in to vote at newly created voting centers. Denver has moved from the old precinct-style polling places to a new “voting center” model where voters can go to any polling place in the area to cast ballots, regardless of where they live. The software was supposed to make it easy for officials at any voting center to check online and make sure a voter had not already voted somewhere else in Denver.
Instead, it led to massive problems on Election Day due to “decidedly subprofessional architecture and construction,” according to the report from consultants Fred Hessler and Matt Smith at Fujitsu Consulting in Greenwood Village, Colo. Fujitsu was hired by Denver shortly after the election to find out what went wrong and help to fix the problems.
“The ePollBook is a poorly designed and fundamentally flawed application that demonstrates little familiarity with basic tenets of Web development,” the report stated. “Due to unnecessary and progressive consumption of system resources, the application’s performance will gradually degrade in a limited-use environment and will be immediately and noticeably hampered with a high number of concurrent users.”
In other words, the more heavily it was used, the slower it worked.
“Moreover, it appears that this application was never stress-tested by the DEC or Sequoia,” other than using it in the spring primary as a test election, the report said. “It is at best naive to deploy enterprise software in an untested state. It is remarkably poor practice to deliberately choose a critical production event (the primary election) to serve as a test cycle.”
The Sequoia application was chosen over a tested ePollBook application already in use by Larimer County, Colo., that has been offered to other Colorado counties for free. The consultants recommend that the DEC either get the Sequoia application repaired or take a new look at the Larimer software to see whether it could be used effectively in Denver. The Larimer application uses a server-resident Microsoft Access front-end accessed via Citrix and an Oracle database on a dedicated server, as well as five application servers for access by election officials.
The voting center delays — with waits in some places of up to three hours — forced an estimated 20,000 voters to abandon their efforts to vote on Election Day, according to the report.
Other problems with the software include Web sessions that would not expire unless a user clicked a specific “exit” button to close the application, tying up system resources, according to the report. The problem, gleaned from user activity logs generated during the Nov. 7 election, was that 90 percent of user sessions that day were not ended using the special button but were closed by users who simply shut the browser. That did not free up resources, causing the system slowdowns.
“In media reports following the election, Sequoia defended this flaw by stating that the DEC had not requested that a session-timeout feature be implemented,” the consultants wrote. “This is a weak and puzzling defense. In any case, session management is a fundamental responsibility that developers of Web applications are expected to fulfill. Describing session management as a special feature that must be requested by the client is not a reasonable position to adopt.”
Also troubling, the consultants said, is that the application and database currently share a server instead of relying on a dedicated database server — something that would have improved performance, security and redundancy.
A spokeswoman for Sequoia, Michelle M. Shafer, declined to comment directly on the consultant’s report in an e-mail response. “While we may disagree with opinions expressed by the author of this report, our focus is on helping Denver solve their problems,” she wrote.
In addition to the software problems, the report stated, IT management within the DEC needs to change so that similar situations don’t occur again.
The three key flaws within the DEC are “generally substandard information technology operations and management,” “dysfunctional communications between the technology function and other leadership,” and “a general and pervasive insufficiency of oversight, due diligence, and quality assurance,” according to the report.
These issues also led to problems with absentee ballots that couldn’t be easily scanned by poll workers and other difficulties with equipment, poll workers and other systems, said the report. “The less-than-rigorous conduct of the ePollBook development project and the ultimate failure of [it] on Election Day, along with … the absentee ballot scanning problem, should be viewed in a broader context of substandard technology management within the DEC,” the report said. “Given the increasing criticality of technology in conducting elections and the sensitivity of personal data in the DEC’s possession, this casual approach to technology cannot be permitted to continue.”
Alton Dillard, a spokesman for the DEC, said the commission “agrees with 99 percent of the report” and will take actions to resolve the problems. “The ePollBook was the chokepoint, but there are some other things that need to be addressed,” he said.
The DEC meets Dec. 19 to decide how to handle next year’s spring primary and off-year fall elections. Three options are under consideration, Dillard said, including the use of mailed ballots for all voters, a return to precinct voting or continuing to use voting centers while fixing or replacing the ePollBook software. Officials want to get everything fixed before the 2008 presidential election, he said.
“Right now, there’s no uniformity among the [election] commissioners on which form to accept,” Dillard said.
Chris Henderson, the chief operating officer for the city of Denver and a spokesman for Mayor John Hickenlooper, said the consultant’s report shows that “clearly the … technology component of the election commission is pretty broken right now. We are dismayed on a lot of levels about the troubled nature of the implementation of the [ePollBook] software. The challenge is the election commission’s business to sort out those questions.”
Henderson said he hopes the DEC looks seriously at the consultants’ other recommendations, including a call for the DEC to take advantage of the IT staff and resources used by the city and county. “I think, clearly, there’s an opportunity for them to benefit from some of the smart people we have working for the city of Denver,” he said.
On a related note, John Gaydeski, the executive director of the DEC, resigned from his post last week in response to the problems stemming from the November election.