2014: Winter Issue

Home – Judge Scheindlin Tells Government Agencies to Conduct Searches Again, Questions Keyword Searches by In-House Teams

Judge Scheindlin Tells Government Agencies to Conduct Searches Again, Questions Keyword Searches by In-House Teams

As long as there have been saws, carpenters have lived by this axiom- "measure twice; cut once." To do the reverse and "save time" by skipping that second measurement too often results in a waste of lumber, re-doing work or watching your deck fall off the back of your house. Apparently the same is true with electronic discovery. 


Simple keyword searching by in-house staff may seem less expensive than more sophisticated efforts, but the risk of replicating work and sanctions looms large.


Now, the debate over the adequacy of simple key-word searching in favor of processes like predictive coding and complex search criteria has been carried to the Freedom of Information Act (FOIA) arena by one of e-discovery's leading judicial commentators, U.S. Judge Shira A. Scheindlin of New York. 


The judge held that reliance on simple keyword searching alone is often not enough to find responsive information, and that document custodians are not necessarily qualified to run the kind of complex searches demanded by FOIA (National Day Laborer Organizing Network, et al. v. United States Immigration and Customs Enforcement Agency, et al., 2012 U.S. Dist. LEXIS 97863 [SDNY, July 13, 2012]).


Given the thorough detail and analysis of the decision, it is certain to be quoted in e-discovery cases that follow. Even its FOIA focus won't likely stop that. "Although this latest opinion is limited to analysis of the reasonability of search efforts by federal government agencies in a FOIA search (where the government has a high burden of proof), we can expect to see this case frequently cited in the broader context of electronic data discovery," wrote Jackson Lewis partner Ralph Losey LLP in a post on the ALM® Law Technology News® site. 


Writing for InsideCounsel, attorneys Judah Lifschitz and Laura Fraher of Shapiro, Lifschitz & Schram, P.L. say this ruling also serves as a warning to those who wish to rely on "in-house personnel's ability to conduct reasonable searches." 


In a paper posted on martindale.com® last month, attorneys Donald E. Taylor and James E. Tonrey of Wilentz, Goldman & Spitzer said the ruling will have an impact on e-discovery in general, calling it a "significant decision that will likely change the landscape of electronic discovery."


"Judge Scheindlin's sweeping references to 'the old way' of doing things and new emerging technology reflects a willingness to move beyond the mere mechanics of electronic discovery toward fine-tuning those mechanics and making them more efficient," the Wilentz attorneys wrote.


A Search for Transparency


The ruling came in a suit brought by the National Day Laborer Organization Network, the Center for Constitutional Rights and the Immigration Justice Clinic of the Cardoza School of Law against the U.S. Immigration and Customs Enforcement Agency, the U.S. Department of Homeland Security, the FBI and the Executive Office for Immigration Review.  


The plaintiffs brought the suit as part of a campaign to end Secure Communities-a federal immigration enforcement program-and bring greater transparency to the nation's immigration debate.


The cost of this litigation-which has spawned several e-discovery decisions from Judge Scheindlin-has soared into the "hundreds of thousands," the judge said. "Transparency is indeed expensive," she wrote, "but it pales in comparison to the cost to a democracy of operating behind a veil of secrecy." 


Plaintiffs are looking for documents relating to records shared by local law enforcement agencies-including fingerprints-with the FBI and Homeland Security.  Documents were requested, but the government said the request was a burden. A Rapid Production List (RPL) was created.  However, the government didn't comply causing the judge to direct the government to produce the RPL documents and information relating to the process by which states may opt out of the program. 


The government conducted searches and produced "tens of thousands of responsive records," the judge said, but the plaintiffs said the searches were inadequate. Each of the agencies argued why their searches-both manual and computer-driven-were adequate and explained their efforts. Judge Scheindlin said some were "extremely rigorous" while others were "woefully inadequate."  For many she said she didn't get enough information to decide.


The plaintiffs claimed the agencies' searches were "not reasonably designed to uncover all responsive records." They said the agencies not only improperly excluded some of the information custodians, they provided custodians with "vague search instructions." Or, the plaintiffs said, the custodians used inadequate search terms or didn't specify the terms at all. The government agencies disagreed and continued to defend their efforts. 


Things Have Changed


"It is impossible to evaluate the adequacy of an electronic search for records without knowing what search terms have been used," Judge Scheindlin wrote. "In earlier times, custodians and searchers were responsible for familiarizing themselves with the scope of a request and then examining documents individually in order to determine if they were responsive. Things have changed. Now custodians can search their entire email archives, which likely constitute the vast majority of their written communications, with a few key strokes. The computer does the searching. But as a result, the precise instructions that custodians give their computers are crucial."


Today, the judge said, a custodian-who never actually looks at the universe of documents they are searching-may miss key documents simply due to a typo in their search request. In the case of the FBI, Judge Scheindlin said, the agency gave little insight into its search other than to say much of it was done by hand and for the rest it provided no search terms.


The government said the judge did not need to know the search terms and combinations to determine adequacy. The agencies said declarations about the searches do not have to "set forth with meticulous documentation the details of an epic search."  The government said it was not clear why records custodians could not be trusted to run effective searches of their records-"a skill that most office workers employ on a daily basis."


Reasonable Specificity


Judge Scheindlin said the search details are critical to determining adequacy. "[C]ustodians cannot 'be trusted to run effective searches,' without providing a detailed description of those searches, because FOIA places a burden on defendants to establish that they have conducted adequate searches; FOIA permits agencies to do so by submitting affidavits that 'contain reasonable specificity of detail rather than merely conclusory statements.'"


         Responding to the government's claim that any office worker can search his own files-the judge held that most custodians are not equipped to run effective searches because "designing legally sufficient electronic searches in the discovery or FOIA contexts is not part of their daily responsibilities."


         "Searching for an answer on GoogleTM (or Westlaw® or Lexis®) is very different from searching for all responsive documents in the FOIA or e-discovery context," Judge Scheindlin added. "Simple keyword searching is often not enough."


         There is a need for "careful thought, quality control, testing and cooperation with opposing counsel in designing search terms or 'keywords' to be used to produce emails or other electronically stored information."     And beyond the use of keyword search, parties can (and frequently should) rely on latent semantic indexing, statistical probability models, and machine learning tools to find responsive documents. "Through iterative learning, these methods (known as 'computer-assisted' or 'predictive' coding) allow humans to teach computers what documents are and are not responsive to a particular FOIA or discovery request and they can significantly increase the effectiveness and efficiency of searches." Judge Scheindlin said.


You Can Work it Out


         "Evaluation of search terms is highly context-specific: the failure to use certain search terms will sometimes be fatal, sometimes unproblematic, and sometimes improper but harmless or at least mitigated," the judge wrote. "Even courts that have carefully considered defendants' search terms have generally not grappled with the research showing that, in many contexts, the use of keywords without testing and refinement (or more sophisticated techniques) will in fact not be reasonably calculated to uncover all responsive material."


While saying she could not really make a determination on search adequacy, and wanting to avoid revisiting old searches, she directed the parties to work it out, directing them to cooperate in designing and performing a "small number of new, targeted searches." Custodians who should have searched their records have to run complete searches. Custodians who conducted searches and didn't provide details also have to conduct "new, fully-documented searches."


Judge Scheindlin directed the parties to "agree on search terms and protocols-and, if necessary, testing to evaluate and refine those terms."





  • Work with the opposing side to narrow the scope of document searches.


  • When using in-house teams make sure they are trained and qualified to perform accurate searches for responsive documents, otherwise look to outside specialists.


  • Provide custodians with clear, specific and detailed criteria to prepare them for their searches.


  • Do not rely on simple keyword searches. At least use, test and adjust combinations of terms and/or Boolean searches, and look to computer-assisted, predictive coding methods for increased accuracy.


  • Log the details of your searches so you can defend with "reasonable specificity" the adequacy of your efforts.


For more information about LexisNexis products and solutions, connect with us through our corporate site