Some say that predictive coding isn’t as useful to plaintiffs as it is to defendants. See, for example, this post on Linkedin.
In my view, what really matters is whether the litigant is producing or receiving the documents. Predictive coding is more useful to a producing party than to the receiving party. And, in a way, predictive coding is actually the opposite of post-production analysis.
Reviewing documents for early data assessment, early case assessment, or production has become a severe pain point for litigants. The exponential growth in our use of unstructured or semi-structured electronic documents in litigation is making it increasingly difficult to comply with ediscovery deadlines. If a party walks into a scheduling meeting without having a reasonable data map and a proposed plan, or if production isn’t done right, the hammer can come down hard.
In contrast, the receiver’s analysis of the documents produced is less constrained by time pressures, and the producing party isn’t too likely to complain that the receiver didn’t adequately review the documents.
Technology has responded by providing predictive coding solutions for litigation, mainly to facilitate production by either side, not post-production analysis. Predictive coding attempts to generalize from a set of human judgments about the discoverability of certain example documents.
Once the documents are produced, however, the question of whether they are discoverable is pretty much moot. In fact, in a way, what the receiving party does with the documents is the opposite of the predictive coding process. In post-production analysis, the litigant isn’t generalizing from examples to a larger set. Instead, it is analyzing the larger set of documents that it has received and trying to zoom in on those few documents that most clearly support its position.
If a particular monthly report to senior management contains a particular admission, that doesn’t mean that all monthly reports in the series contain the same admission. And if they do, it’s merely cumulative. The receiving party wants to find the best example from the pack, and that’s more a matter of nuance and judgment.
The state of the art in ediscovery is that we have technologies that can tell us which documents seem to address a particular subject. That’s the task for the producing party. That’s predictive coding. We don’t have technologies that can tell us which documents contain or support a particular statement. That’s the task for the receiving party. That’s Watson. So until we all have access to Watson, recipients need to generally rely on other search tools, their experience, their knowledge, and their intuition.
great perspective Josh
Thanks Tom. As you and Jeff Parkhurst explained in your recent webinar, predictive coding is one of the many tools that are useful in ediscovery. http://cavolegal.com/whats-the-most-efficient-way-to-review-documentsby-tom-oconnor-and-jeff-parkhurst/
When Stratify was introduced it did perform cluster analysis based on the content of a document collection. The analysis was unguided by the user so the results were often overwhelming in the number of clusters.
It’s like the blind men and the elephant. Unguided clustering may be helpful when you haven’t yet oriented yourself to the collection. In my opinion, once you know what you’re looking for, example-based clustering is more efficient.
I think you should reconsider your premise. In our experience, Predictive Ranking is incredibly useful for reviewing in-bound productions. The reason is that productions are often received shortly before depositions take place. Your goal as the receiving lawyer is to find as many good documents as you can and bring them to the deposition.
Typically the opposition produces many responsive (perhaps) but not important documents, leaving key documents to be found. Using a continuous ranking engine allows the team to review in ranked order and push hot documents to the top of the stack. Typically you don’t have time to review all. The cutoff is the point where you run out of hot documents or time (or both).
For lawyers afraid of the transparency requirements for outbound productions, using TAR for incoming productions is all but a no brainer.
Our experience at least.
Thanks John. I put my position more clearly at the beginning of the post than at the end. I think that, in general, predictive coding is more useful on the producing side than on the receiving side. I agree that it can be useful on the receiving side in many circumstances, as in your example where you’re getting sandbagged by an unworthy opponent.
I also agree that prioritizing documents for review or analysis is essential for either side at every point, and that predictive coding is a good tool for that. I hold the same view in the context of proactive litigation readiness and information governance.
IMHO, the term “TAR” should not be synonymous with predictive coding. I agree with you that the receiving party should use any appropriate technology to assist in review and analysis of the documents. However, I also think that you’d agree that, for example, the technological ability to regress complex linguistic constructs against each other in n-dimensional space can assist review outside of a predicting coding framework (as well as within it). So, for clarity and avoidance of confusion, I’d put all of those advanced technologies, not only predictive coding, under the umbrella of “TAR.”
We call it Predictive Ranking rather than predictive coding because these systems rank documents, they do not code them. I agree that for some the term TAR is broader and could engulf all things people do with computers. However, that is not the meaning most people give it.
Again, I deal with folks who are afraid to use predictive ranking technology for outbound productions because they are concerned about the transparency requirements. None of that exists with regard to incoming productions. Thus, they readily understand the value of that use case.
Both are important uses for predictive ranking. I just wanted to point out the value for incoming productions.
I should say that we don’t follow the TAR 1.0 model, which probably affects our behavior. TAR 1.0 requires subject matter experts, separate training, does not handle rolling collections, and is built around one-time training. All of that makes the process brittle and hard to use.
TAR 2.0 systems are built around continuous learning and continuous ranking. There is no need for random training seeds, SMEs, separate training from review or retraining. You just jump in and go. The system keeps learning and the results keep improving.
I reluctantly have to acknowledge that, at this point, the meaning of “TAR” as generally used is the Grossman-Cormack definition. http://www.edrm.net/resources/glossaries/grossman-cormack. I just wish it were otherwise.
Looking forward to learning more about Catalyst’s system.
I assume you think our definition of TAR is too narrow, and should include visualization, clustering, and other investigative tools. If it were to include such tools, why should it not also include keyword searching? And if it were that broad, what meaning would it have?
Our definition does admit, for example, rule-based methods that to not use machine learning and are therefore not usually labeled “predictive coding.” And it is not clear to me whether or not the continuous active learning method used by Catalyst would be called predictive coding, but it certainly fits our definition of TAR.
(NOTE BY JOSH: To avoid unduly skinny response columns, I’ve limited the response “depth” to five levels. I’m copying Gordon’s comment below and responding there.)
In the trenches, I a can tell you that it is more useful outbound. There are two major problems with its use.
First the technology is akin to voice recognition 10 years ago. Good, but not inspiring confidence.
Second, it suffers in practice from the same flaws as every other technology that is run by lawyers. At the field level there is simply no empirical testing. For a field so young, most of operations people are rigidly hierarchical. At the field level where the work is done, lawyers do not think probabilistically. There is no random sampling, no concept of estimators and nothing Bayesian. Therefore, people with eyeballs on the documents in my experience are almost never asked about the predictive coding results.
So, it is useful, but could be so much more so if it was extended to its limits. Unfortunately, right now, its limitations are compounded by the lawyers that actually use it.
We have come a long way from the voice recognition days for TAR. The first generation of TAR engines did have the kinds of flaws you describe. There is a newer generation built around continuous learning that does not have these limitations.
You don’t have to estimate to measure success. Reviewers see the benefits because suddenly they are tagging far more relevant documents than would occur randomly. When you quickly go from 20 relevant per 100 to 60 and more, you know the systems are working. When they later drop down, you know it is time to stop.
I believe every review will use predictive ranking once more systems move to the next generation of continuous learning systems.
Lee, I hope your experiences in p-discovery were better than in ediscovery, in at least one instance. Your individual contributions as a team member were invaluable. As I think you know, I’m a big believer in teamwork, and it paid off handsomely in the case we worked together.
Yes. P-discovery has been a good experience. I am just frustrated that it could be so much more. Let’s push the tools to the limit and then get better tools.
The project we worked together on was great. It was an example of how to work together to get the most out people and software.
–
Gordon Cormack on October 28, 2014 at 12:17 pm said:
I assume you think our definition of TAR is too narrow, and should include visualization, clustering, and other investigative tools. If it were to include such tools, why should it not also include keyword searching? And if it were that broad, what meaning would it have?
Our definition does admit, for example, rule-based methods that to not use machine learning and are therefore not usually labeled “predictive coding.” And it is not clear to me whether or not the continuous active learning method used by Catalyst would be called predictive coding, but it certainly fits our definition of TAR.
Gordon, your definition appears to be generally consistent with common usage, since most reported decisions and ediscovery authorities use “predictive coding” and “TAR” synonymously, while relatively few include such tools as visualization and clustering in “technology-assisted review.”
Moreover, for the benefit of anyone who might not know how authoritative The Grossman-Cormack Glossary of Technology-Assisted Review is, it has been repeatedly endorsed in publications by The Sedona Conference, and it has been published in the Federal Courts Law Review and adopted by the EDRM organization.
That being said, yes, if I could, I’d change everything so that “technology-assisted review” meant what I think it sounds like. I’d define it to include, and to be limited to, using computers to help us find, categorize, or analyze electronic documents based on data in or about those documents. So yes, I’d include keyword search. (Keyword search is a blunt instrument if it’s the only one in your utility belt, but it’s still pretty sharp if you’re looking for documents about “Chewco.”)
Anyway, here’s where I take off my Andy Rooney/Humpty Dumpty hat and bow to the consensus.
* * *
Based on your post, I’m not entirely clear about the Glossary definition of TAR. In your post, you said that there are rule-based methods that do not use machine learning, and are therefore not usually labeled “predictive coding,” but that nonetheless fall within the definition of TAR.
Under my understanding of the Glossary’s definition, these rules are derived from a set of human judgments made on a small sample of documents and are then used to extrapolate to the remaining documents in the population. How can a TAR method derive rules from a set of human judgments other than by using machine learning algorithms? And if it then uses those rules to extrapolate to the remaining documents, why isn’t that predictive coding?
For the benefit of readers, the Glossary defines “Technology-Assisted Review (TAR)” as:
In a rule-based method, the rules are generally constructed by hand, not using a machine-learning algorithm. The experts who construct the rules are informed by examining a sample of documents, and the effect of the rules on those documents.
One may use analytic tools in formulating the rules, but in the end the rules are used to make automated decisions about documents that were never seen by the authors of the rules.
The rules themselves might (or might not) closely resemble Boolean queries. But they are derived using a quite different process from what we normally call “keyword search” or even “Boolean search.”
H5 is an example of a Vendor in which professional linguists and statisticians construct a rule base. H5 was one of the two methods (along with a continuous active learning method) that Grossman and I compared to human review in our JOLT study (Richmond J. Law & Tech, 2011).
Thank you for the clarification, Gordon.
Here is the link to the JOLT study: http://jolt.richmond.edu/v17i3/article11.pdf
This is an interesting dialogue on several fronts. At a base level, I agree that the terminology and verbiage can become confusing to the lay practitioner and has yet to be standardized across the industry. I am of the schools of thought that Predictive Coding (PC) is an element of Technology Assisted Review (TAR) and is but one tool available to practitioners. That said, PC can be an extremely valuable tool to all concerned parties and for the assessment of produced and received documents. From and education perspective, we are still early into the acceptance of PC, although it is spreading. Many parties who do not have experience with PC, are more willing to engage PC on received productions for their first experience than they are for outgoing productions. My experience has been that once the benefits of PC are experienced, there is no going back and they look for opportunities to leverage it.
I think the bottom line is that as adoption of PC increases, we will (and are) find that clients leverage the technology in new and interesting ways.
With respect to validation and testing of results, I (and our organization) are strong proponents of testing and validation. The truth of the matter is that without taking the appropriate steps to validate your work (this is true whether you use PC or not), it does not really matter how good the technology is, because without validation procedures, defensibility becomes difficult.