Data, Volume, and Defensibility: How Canadian Litigation Teams are Managing the Complexity Crunch

[00:00:09] Mallory Hendry: Thank you for joining us for today’s webinar, Data Volume and Defensibility: How Canadian Legal Teams Are Managing the Complexity Crunch. I’m Mallory Hendry, Content Specialist Manager at Canadian Lawyer. First, a few housekeeping notes. This session is being recorded, and it will be available to all registrants in the coming days. We will also have a question and answer period, so please submit any questions using the Q&A function at the bottom of your screen. And just so you know, if we run out of time to address them during the webinar, our hosts will get back to you separately, so feel free to send the questions as they come to you. Today, we’re pleased to welcome 3 experts from Array Canada. So Array is a global legal services provider that works with law firms and in-house legal teams throughout the eDiscovery process, with offices in Canada, the US, and the UK. With roots in Canadian eDiscovery spanning more than 25 years, Array has Canadian leadership and team members across the country dedicated to supporting Canadian clients from coast to coast to coast. Array works with its clients to collect and process electronic information, analyze and reduce data volumes, and manage document review through to production. That includes helping counsel choose workflows suited to the matter using advanced analytics and AI where appropriate, with experienced eDiscovery professionals and review lawyers supporting the work. So in practical terms, Array brings together the technology and the people to help legal teams make sense of their data and manage the demands of litigation, investigations, and regulatory matters. So our hosts today, we have Jessica Lockett. She’s Vice President of eDiscovery and Review Services. She is an Ontario Barred lawyer with over 16 years of experience in eDiscovery and document review. We also have Abel Kozatsky, sorry, my apologies, he’s their Senior Review Manager at Array, and he is also an Ontario Bar lawyer with a decade of experience designing review workflows and managing complex eDiscovery review projects. And last but not least, we have Joe Evangelista. He is the Manager of eDiscovery Client Services, and he’s one of only a handful of Relativity Master Designated designates in Canada. He brings with him 20 years of eDiscovery experience. Together, these 3 hosts will bring their vast experience to discuss how legal teams can use advanced analytics, technology-assisted review tools, and defensible AI workflows to improve efficiency and accuracy in modern discovery. So welcome, Jessica, Abel, and Joe, and over to you. 

[00:02:55] Jessica Lockett: That’s great. Thank you so much, Mallory. Thank you for the introduction, and thanks to everyone for being here today. Good morning, good afternoon, depending on where you are across the country. So first, I’ll just start with a quick overview of our planned discussion. So first, we’ll unpack the complexity crunch, which is not only related to increased volume of data, you know, that we’ve seen over the years, but to the growing variety and complexity of data types that layers onto that increased volume. Then we’ll look to the planning that really should happen before any substantive review of that data begins, understanding and reducing the data, defining scope, assessing tool eligibility, and designing the actual workflow. So that’s going to lead into a discussion of the advanced— sorry, the technology-assisted review or TAR tools that we have available, including the most recent GenAI-based tools that I’m sure many of you have heard about and maybe even used yourselves. From there, we’ll discuss defensibility and validation metrics, including how to, you know, check the results and support the decisions made through a review process. And then we’ll bring those ideas to life with real examples of TAR tools in action. We’ll finish off with some key considerations for implementing TAR tools in your own matters, And as Mallory mentioned, hopefully we’ll have time for a Q&A session at the end. And again, if not, we will respond to your questions as soon as possible after the presentation. So throughout this discussion, our goal is really to share practical lessons that Canadian legal teams can apply, you know, to their discovery and to their review work. So we’ll start with what we mean when we talk about the complexity crunch. So when we talk about the complexity crunch in eDiscovery, we’re really talking about decisions that come with managing high-volume, complex datasets. So this is answering questions around what data needs human review, which tools we can and should use to help drive efficiency and defensibility of a review, and then how we can defensively support the approach or the approaches So there’s really 4 themes when we’re talking about the complexity crunch and managing that. The first is managing volume and focusing the human review. So every eDiscovery project should really start with a preparation that makes that focus possible. So that’s analyzing and understanding the data and the data types, working to reduce unnecessary volume, and defining which documents, you know, fall not only within the scope but within eligibility The second is choosing the right tool or tools and devising appropriate workflows for the matter. So current advanced tools such as continuous active learning or CAL and the newest generative AI tools, they should really be considered when developing your workflows, including how they might work, you know, both independently and potentially together to meet your review objectives. So I think that’s a really good point. Typically, there’s not, you know, sort of a singular approach or a tool that’s going to get you to your end objective alone. There’s a lot of layering that is usually involved in this process, and it’s really important to consider these tools and their eligibility alongside your review objectives as well as your data characteristics, including, you know, giving consideration to those documents and data types that might not be eligible for use with these tools, and they’re going to require a separate path for review. And the third theme is really around validation and making defensible review decisions. So validation and testing, they tell us about the results at the point of testing, and defensibility is really how we use those results to support decisions within the context of a particular review. And the fourth theme is the role of human judgment, oversight, and accountability, because this does not go away. People define the review criteria, and it gets refined as they learn. People also handle what we call exceptions, which is instances where, you know, there appears to be some substantive conflict between the human reviewer and the tool’s output, and even between 2 human reviewers. And people also document these decisions, you know, that help shape this defensible process. The case studies that we’re going to look at today really aim to show these themes in practice. And the thread connecting them is defensibility. So keeping human oversight and validation connected to the workflow as we manage the complexity crunch. So we’ll start with the groundwork for managing volume and focusing human review. So building on that first theme, you know, the work that really should happen before any real substantive review begins is outlined here. This is where we work to really develop an approach that fits not only the matter, not only the end goals, but also the data itself. So, you know, we’ll focus on data analysis and reduction. We need to understand what we have and where there may be opportunities to reduce duplication and reduce any unnecessary volume. That gives us a clearer picture of the population that we’re working with. But when we talk about, you know, the data population for review, we’re looking at more than just the number of documents, you know, that could potentially be in scope, because a population of data might contain different data sources such as emails, attachments, you know, chat messages, team messages, spreadsheets, scanned PDF images, you know, things like that. Each of those are going to raise different review questions. So, for example, you know, a short chat message, a Teams chat message that says, oh, that works, may only make sense if we see the surrounding conversation within Teams. And so how those messages are captured and processed and presented for review becomes really important. And the range of communication platforms also adds to that complexity. Crunch as well. So, you know, you think about a discussion, and I think this happens to us all, it starts in email and it moves over to Teams, and, you know, then maybe we’re talking on text or WhatsApp. You know, someone might share a link to a document while another person might send a screenshot of that document. You know, so these communication platforms also produce, you know, different kinds of records and export formats, and they can just require very different treatment depending, you know, on what they are, even if they concern the same issue. Same goes for, say, sort of like a recording. You know, we need to consider, is a transcript sufficient to meet the review objective, or does someone actually need to listen to or watch that video or audio for the additional context it might bring? And if so, are there even certain tools out there that can help do that to reduce the human effort involved? So these are just all illustrative of why, you know, file types alone or document numbers alone, they don’t tell us how to effectively approach a review. And that’s really part of this complexity crunch, which is there are more places to look, more data types, more fragmentation, and more preparation required as a result. You know, and alongside that, of course, we need to establish the review scope. So we work closely with counsel to identify which documents fall within the review scope and the basis for setting aside other materials, you know, still date range exclusions or data type exclusions, things like that. So once we’ve analyzed the population and the scope is defined, we then assess its suitability for the tools available. You know, some document— documents or data types may be suitable for advanced review tools, these TAR tools that Joe will be talking about in a moment. And others, you know, may need some additional processing or to go through a different review path entirely. So a document that a particular tool cannot assess might still be in scope for potential relevance, and we need to account for it through a different workflow or tool. And when we bring that analysis together with the review objectives, the approach or approaches we choose should really clearly reflect what we need to accomplish, the types of data in the set, and the capabilities of the tools available. So we may need different workflows, as I mentioned, for different parts of the population, with targeted human review often working in parallel to those processes. So with that groundwork in place, I’ll hand it over to you, Joe. Joe will walk us through some of the main TARD tools available and how they function. 

[00:12:12] Joe Evangelista: Thanks, Jess. So let’s focus in on the tools in eDiscovery. The first point to establish is that AI is not a single product. We typically start with the analytics that many of you already use daily: email threading, near duplicate analysis, language ID, and name normalization. These help with reducing data volumes. They, they’ll help with review prioritization and allow for various QC checks throughout the review. Machine learning and CAL sit between— sit above that layer. Reviewers code documents on a binary relevant or not relevant basis, and the system learns from those decisions and applies that learning across the remaining population. Generative AI approaches things from a different direction. Rather than learning from your coding, it begins with written criteria. You describe what relevance or an issue looks like, and the system applies that prompt across the documents. Then there are hybrid workflows, which in practice describe most of what we do because different parts of a population require different treatment. The point is that each layer adds capability. None of them renders the previous layer obsolete, and none of them removes the need for analysis. These tools do not function without us, and they need direction. They need input. And they, of course, need testing. The technology is most successful with teams that understand how to apply each function so that they work together. So how do you choose? Your decision is driven by the review objective. It’s the, the size and composition of the data, the timelines, the risk profile, privilege considerations, and how the results will need to be defended. So there’s really no one-size-fits-all answer to these questions. CAL and generative AI are not rivals. Increasingly, we’ve actually been using both on the same matter. The technologies, although differ, the governance that applies to all of them is the same. Let’s discuss both after a really quick poll question. The question is going to come up in a moment. There we go. So here’s the question. Approximately what percentage of your document reviews in 2026 use CAL or GenAI tools? So you can, you know, if you, if you haven’t, it’s going to be none, of course. Less than 25%, maybe 25 to 50, or more than 50%. You know, approximate is good. And we’ll just give everyone a minute or so to, to get their answers in. Oh. All right. So it looks like the answer, like more than 50%, is quite low. I imagine the— those totals will start to increase as we As we get through this. So great, thank you for that poll question. So on the next slide, we’re gonna talk about— or where did it go? All right, so CAL. CAL stands for Continuous Active Learning, and the operative word there is continuous. It’s not a model that is trained once and left to run on its own. It’s actually a loop. So reviewers code on a binary, relevant or not relevant basis. The model then reads those decisions, identifies patterns in the document content, and then re-ranks the unreviewed population. So assigning ranks to likely— assigning higher ranks to likely relevant documents. And as reviewers keep coding, the model keeps updating. The practical benefit is that the team reaches its important documents early, rather than treating every document with the same relevance potential. At the appropriate point, the team stops and tests to decide how the remaining population should be handled. 2 things I would add for anyone who has not reviewed using CAL. First, reviewers are not outside the process. Their coding is what drives it. A CAL workflow supported by inconsistent review is going to produce— surprise— an inconsistent model. Second, and this is the most important part and often overlooked, not every record is eligible for assisted review. Some may have very little extracted text, others are in formats the technology simply cannot work with. So these documents are not ignored. They are handled in a separate review stream using, you know, targeted searching, the data isolation using various metadata, and linear review whenever that’s warranted. So, and populations rarely are, are rarely uniform, and defensible workflow will account for what falls outside of that technology. That is a question that you could be— that you could be asked, and you should have a documented answer. So the overall workflow is again a loop. So review, train the CAL model, have it prioritize, and then finally validate. So outside of this, we still need an intentional plan for the data the model cannot assess. So let’s go to another quick poll before we start discussing the GenAI workflow. Right. So has your team used generative AI specifically for document review in 2026? That is the question. So yes, regularly. Maybe one on a few matters, or one or a few matters. Not yet, but exploring it. And, you know, for those that may be Not considering it at the moment. There’s an answer for you there too. And just give everyone a moment to answer. Okay. So a lot of people are still exploring the, the technology, and some have actually used it. So that’s great. And 22% is, you know, of regular use, which is great. I imagine just like Cal, we’re going to start seeing these percentages creep up quite a bit more in the future. Okay, thank you for the poll. And we’ll go to the next slide. So this is the generative AI equivalent, equivalent of the previous workflow, again in generic form before we discuss some examples. So the point to note is the order. 3 of the 5 stages occur before the technology performs any substantive work. These 3 stages need a consistent subject matter expert to succeed during the last 2 steps. First, the prompt. Your draft— you draft written criteria including relevance definition and the issue categories, and then translate them into instructions that the system can apply. This is legal drafting rather than technical work. The same principle applies as when instructing a review team where precise instruction produces the expected results. These prompts are drafted and are refined iteratively. Second is the sample testing. So these written instructions are run against a subset of documents And the output is examined and adjustments are made where, you know, whether— and then the criteria is run again. So third is validation. So the AI coding is compared against coding completed by a senior review lawyer on a statistical sample. So we then investigate the reasons why the review lawyer disagrees with the AI coding and refine the criteria or the prompts until that coding is consistent across the review lawyer and the predicted relevance. The validation criteria, once that’s valid, once it’s consistent, is then applied across the entire eligible population. So finally is the quality control. This is in the form of a targeted post-AI searching, and it’s designed to identify relevant material that may be— may remain in the not relevant population. That step occurs after the technology is finished, and it is frequently where the final defensibility and quality check occurs. Similar to CAL, there is a separate review stream that, that includes ineligible documents and privilege screening. So the 5 stages with people driving 4 of them, That is why it’s properly described as AI-assisted rather than automated. So we’re going to switch over to Abel. He’s going to discuss some of the examples. 

[00:22:00] Abel Kozatsky: So, excuse me, I’m going to discuss 3 case studies, but before I do, I just want to establish a central principle here. And that’s that using generative AI or CAL in itself is not what makes a review defensible. These are just technologies. They’re tools operating within a broader workflow. The defensibility assessment belongs to the workflow as a whole, not to the specific tool. I find it really useful to examine defensibility through 3 lenses, and to be clear, these are practical lenses and not a legal test. The first lens is logical justification. Can we explain what we did, why we selected this workflow, how the technology fit within the review objective, and how we handle documents and exceptions outside of the main process? Now, the second lens is statistical validation. So what does testing tell us about the workflow’s performance, including the relevant material potentially left behind? The, the statistics alone don’t equate to defensibility. What they do is they provide measured evidence about how the process performed. Now, the third lens is case-specific appropriateness. Why is the selected approach and the measured result appropriate for the specific matter, given the legal issues, the size and character of the population, the risks of missed material, and proportionality? The distinction between the second lens, statistical, and the third case-specific lens is really important. Statistics describe performance, but the case context informs whether that performance is sufficient. So the defensibility discussion can be reduced to 3 questions. Does the process make sense? Do the results hold up under testing? And thirdly, is the approach appropriate for this particular matter? Now, this framework applies whether the tool is CAL or generative AI. CAL learns primarily from reviewer coding outcomes. While GenAI applies written criteria that the team develops and refines, the technologies differ, but both sit inside a workflow that can be logically explained, statistically tested, and justified for the particular matter. Okay, so let’s look now more closely at the second lens, statistical validation. These measures help to understand performance, which we can then assess in the context of the matter. So whatever technology you’re using, CALD or generative AI, there’s 3 measures: recall, precision, and illusion that help describe how the workflow performed against human coding on a statistical sample. There are 3 key validation metrics used in evaluating workflows. The first measure is recall. Which answers the completeness question. That is, of all the relevant documents in the tested population, how many did the workflow identify? The second measure is precision, which answers a different question. Of the documents in the workflow identified as relevant by the model, how many were actually relevant? So it describes the composition of the set. Now, the Let’s say the operational significance of precision, it really depends highly on context. I can give you 2 hypothetical examples, which we’ll compare both with 100% recall and 50% precision to show why there’s such a huge difference depending on context. So, in the first example, let’s assume that we have a small, low richness population. Take 5,000 documents with a 5% richness level. So, only 250 of the documents are actually relevant. At 100% recall and 50% precision, the workflow would return 500 documents. All 250 of the relevant documents would return— would be returned, plus 250 false positives. A small team could go through them and pick out the relevant documents, the 250 relevant documents out of those 500, and the other 4,500 documents would never need to be reviewed. So a 50% precision in this case would deliver a substantial review reduction. Now, second example is when you have a large, high richness population. Let’s take 1 million documents for an example at 50% richness. So you have 500,000 documents that are actually relevant. At the same 100% recall rate with 50% precision to capture all 500,000 relevant documents, a generative AI workflow would classify the entire million documents as potentially relevant because it’s at 50% precision. And half of those documents would be false positives. That is, they wouldn’t be relevant. So the entire population would be captured. So there would be zero reduction in the second case. So the same precision score, 50% in both cases, in example 1 and example 2, and you have very different outcomes. So context clearly matters when we talk about precision. Now, the 3rd measure that we look at is illusion, which looks at the population that’s left behind. Of the documents treated as not relevant, what proportion are actually relevant? And that’s measured by sampling the set. Now, the key point that these statistics describe— excuse me, it— I’m sorry. So what’s key here is that the statistics describe performance. They don’t determine sufficiency on their own. Recall, precision, and illusion must be read together with richness, population size, legal issues, risk, and proportionality. There’s no universal threshold that applies to every single review. Now let’s start looking at the case studies. So the first case study is a Canadian complex litigation review that we ran in Relativity Review Center using continuous active learning. We started with a review universe of about 1.79 million documents, and about 962,000 entered assisted review. After removing duplicates, there were about 750,000 unique documents. That were scored through continuous active learning. Reviewers coded continuously. Reviewers, reviewer coding documents continuously informed the model, allowing the system to prioritize documents that were more likely to be relevant. So their efforts weren’t wasted in that way. Reviewers coding the documents, reviewers, they applied the protocol using their legal and matter knowledge and relying on their experience doing many projects previously. And the system learned from the coding outcomes that the reviewers produced. At the selected cutoff, about 65,000 documents were ranked high enough to send to the review team for a closer look. The remaining 686,000 ranked low enough that once we tested them with, with the sample, we found that they didn’t require Eyes-on review. So viewed through the 3 lenses, the workflow had a logical justification. It used continuous reviewer coding to prioritize a very large population and included defined cutoff and had a separate treatment for the non-CAL eligible documents. It was statistically validated. A review team coded a sample. The review team coded sample produced about 2.5% illusion, 61% precision, and 73% recall. And those stats describe the performance of the workflow, and the cutoff was assessed based off its case-specific appropriateness. In this case, Illusion was particularly important because it estimated the review— the relevant material remaining in a much larger set potentially left behind. So the decision was not only based on the percentage, but on the matter’s legal issue, risk, and proportionality. So in terms of defensibility, CAL didn’t independently as a tool understand the legal issues. You have to see it through a framework, looking at the workflow that tested those results and measured the performance in context. That was really the key. Now on to the second case study, which is a generative AI case study. Using Aera for review and relativity. So I’ll present it the same way that I did with the previous example. What came in, how the population was reduced, and what the validation showed. So we started with about 4.9 million documents collected across 20 custodians. Deduplication, search terms, email threading, and further analytics brought the review population to about 750,000 documents before substantive review started. We then deduplicated. The resulting population was then divided into separate workflows. 329,000 roughly unique documents were eligible for review using AIR for review, and another 161,000 and change were not. The 161,000 documents followed a human review path supported by senior review lawyers, targeted analytics, and grouping. So in terms of a contrast, here’s a key difference between the Cal example and this generative AI example. CAL learned from reviewer coding, coding from their coding decisions. AIR, by contrast, required setting out all the review criteria explicitly in writing so the model could apply it directly to each document one at a time. So experienced reviewers and counsel often rely on knowledge absorbed through training, examples, feedback, and familiarity with the subject matter. They make They may correctly code even a document without the protocol fully stating every reason or decision. Some of it’s based on experience. In other words, much of that knowledge is tacit knowledge. In practice, prompt development means drawing that knowledge out of them and turning it into clear written rules that can be applied and tested consistently across an eligible population of documents. In terms of results, the validated prompts alone achieved 98.1% recall. 70.8% precision and 4.8% illusion, which, when we layered a search, a QC search afterwards, after the review, when we analyzed to see what specific documents were being missed by the system, we could further improve the results to 99.5% recall and 1.1% illusion. So those figures, they, like, in any case, those figures describe performance. They don’t independently established that the results were sufficient. So they had to be assessed against the matter’s population, legal risk, legal issues, risk, and proportionality. Now on to the next example, which is the 3rd one, which was an opposing production review. Opposing party production review, sorry. Documents— those are documents produced to our client by the other side. So in this case, the goal was to reduce how much manual review the client and their counsel had to do by setting aside not relevant material and classifying the remaining documents by issues to code. We received 220,985 documents. Analytics deduplication, early case assessment brought that down to about 86,000 documents, and of those, 67,000 1,000 documents or so were unique and were eligible for AIR for review. Council had developed the initial prompts once we had isolated those documents and refined them over several weeks. Measured against a sample that they coded, those prompts reached 76.1% recall, 95.4% precision, and 44.6% elusion. So in other words, like, what was flagged was very accurate, but roughly 1 in 4 relevant documents was still being left behind, and council wanted a higher level of recall for their purposes. So we worked together on a further round of refinement. The work focused on surfacing the tacit reasoning behind council’s coding decisions and turning it into explicit instructions that AI for Review could apply consistently. Council’s coding was the benchmark throughout. After refinement, recall rose from 76.1% to 95.4%, and illusion dropped from 44.6% to 12.5%. Precision dropped a bit from 95.4% to 87.1% to 87.4%. So, in practical, in practical terms, slightly more relevant, non-relevant documents were flagged as relevant, but far fewer relevant documents were missed. which met the stated objective of counsel. So viewed through the 3 lenses, there was a logical justification. The workflow reduced a large inbound production with analytics first, handled ineligible documents separately, and focused counsel’s time on the content most likely to be relevant. There was statistical validation. The results were tested against counsel’s— were tested against counsel’s coding both before and after refinement, so we could show the improvement in the numbers. And the testing was case-specifically appropriate. In this case, recall was the right metric given the goal of the review. And Council assessed the illusion rate against the richness of the population and the size of the excluded set. And obviously the absence of a production obligation since it was an opposing production review, which makes a big difference in terms of the illusion score. of what’s acceptable. So back to, I believe, Joe for the next slide. 

[00:36:19] Joe Evangelista: So we’re just going to talk about the, like, what the examples are actually telling us. So when comparing these 2 examples, the technology in these workflows look different, but in reality the discipline is very similar. So Cal on one side learned from reviewer coding, GenAI applied written criteria. So CAL continuously reprioritized, and GenAI analyzed each document against the instructions until those instructions were validated. So, and both involved exceptions that required some sort of separate review stream, and both relied on statistical validation. Both required experienced reviewers, legal input, documentation, and finally quality control. So when you are asked which is more defensible, the reality is the product label does not really answer that question. So, but the better questions are, you know, was the tool appropriate for the task? Were the instructions clear? Were the results tested? How were the exceptions addressed? And was the process fully documented? All very, very important questions. And, you know, although these tools will help with the efficiency and accuracy of each review, we must understand how and when to use them effectively. So how and when, but there are some things to consider before implementing AI tools. So here are 10 questions to consider, and asking them early on will help keep the project on track and defensible. So starting with the need, what are we trying to accomplish? And, you know, the answer is we want to, one, prioritize the most important data. We want to assign relevance classification and other decisions like issues and privilege, maybe generate a chronology, and most importantly, I think, is quality control everything that makes sense. So what data is included on or prioritized in the review? This is very, very dependent on the matter. So we can certainly— we certainly want to include and prioritize the most amount of content. So most amount of documents. as possible without really causing a lot of duplication, you know. But on the other side of this, the answer to the next question will often be similar from matter to matter. So that question is, what is excluded or handled separately? So of course, similar from matter to matter, you will have system files, you’ve got junk emails, images, spreadsheets, handwritten material and maybe some low-text files that all need to be assessed. And these are examples of exclusions that— exclusions and data we must assess separately. So, and the isolation and strategies for these documents should be discussed quite early in the project. So who defines the appropriate— who defines and approves the criteria? Well, and how are the results being tested? So typically, one of the subject matter experts should define the criteria and test the results. So this helps the technology understand what we actually want. And then once the requirements have been confirmed, we can then, we can then develop the workflow. And so the answers to these questions will be the basis of the entire review workflow discussion. But I do want to point out the last question. Can we explain this workflow clearly to counsel, the client, opposing counsel, or court? And so if the answer to this question is no, the workflow isn’t complete. The question alone has— and this question has— I’ve asked this many times and it’s proven its value more more than once before. So back to Jessica for the key takeaway. 

[00:40:50] Jessica Lockett: Okay, thank you guys so much for that. This went, this went pretty fast. So our key takeaway, I think, you know, we were discussing this when we were developing this presentation for you, and I think, you know, the main takeaway for all of us to express to you is that These tools are not changing. The new capabilities are going to emerge and they’re going to emerge quickly. When continuous active learning came out on the market, we started building that into our processes about a decade ago. And there wasn’t really all that much developed in terms of really changing workflows and bringing in new thoughts to the process until we started getting these GenAI-based tools and they are moving rapidly. So, you know, the key here is that knowing that these tools are going to keep coming, there’ll be new tools that are able to do things we haven’t even conceived of yet today. And so building these successful review processes really means being able to adapt with that technology rather than trying to sort of get it to adapt to your process. You know, so for us, I think that really means, you know, what we try to do is we And we recommend to do is to stay open, you know, to these new tools that are coming in, revisit workflows, you know, learn about these new tools. What problems are they solving? Can they solve a problem that you are experiencing? And, you know, adjust your approaches sort of without losing sight of your review objectives and without losing sight of your objectives of Defensibility and validation and accountability. So, you know, this still needs a discipline. And so when we change a tool or add a new tool or layer on a new tool, because new tools, they’re not going to necessarily mean old ones are going away, I think it’s just layering on, on top of the previous. Mm-hmm. You know, we still need to test the results. We still need to account for exceptions, go through the whole process that Joe and Abel, you know, walked through. It just might be that there is a third tool added on, a fourth tool, a separate, you know, validation step that needs to come in. So, your process needs to be developed with that in mind. You know, you don’t want to spend all of this effort developing out processes that become obsolete in 6 months. You know, so, I think that that’s really the capability that’s worth building is really, you know, knowing how to assess those tools and how to bring them in. is really going to help, you know, sort of help you be successful as you’re bringing these tools in or continuing to bring these tools into your workflows. Because we saw through, Joe, through your questions, you know, that a number of people are actually already bringing these in, which is great. So yeah, so I think that was our, that was our main takeaway for you all today. There’s lots to think about here, but I think we’re going to move over into our sort of question and answer Period. Yes, back to you, Mallory. 

[00:43:59] Mallory Hendry: That’s right. Yes, thank you so much. We’ve had a few questions come in. I’ll go through them here. Uh, someone would like to know, with the tools changing so quickly, as you mentioned, how do you avoid building a process that, you know, becomes outdated, uh, so quickly? 

[00:44:14] Jessica Lockett: Yeah, so I think I’ll just pipe in, um, Joe and Abel. I think it’s, um, you know, just layering on what the key takeaway really, really was, you know, that it’s like being agile and bringing agility into your workflows and sort of, you know, building your processes, really thinking about objectives rather than building them around a specific tool. Mm-hmm. So I think that there’s sort of changing that mindset. I know for some people they have, you know, maybe 1 or 2 tools available to them within their firms. And so the idea might be, oh, let’s build a process around this tool. But I think it’s important to really kind of flip that backwards and say, you know, right now this tool might be the one that we have available to us, but how are we getting to our end objective? Because at the end of the day, you never know where you’re going to be in 2 or 3 years with new tools available to you in your firm or through a service provider or, you know, what have you. So I think it’s keeping the end goal at the forefront. Yeah. When helping out these processes. I don’t know, Abel or Joe, if you have any further comments. 

[00:45:22] Joe Evangelista: Well, I mean, keep keeping the end goal in mind, but then also making sure you stick with all of those, like the items for defensibility and optimizing efficiency within your workflow. So it really is just a matter of being adaptable and, you know, picking up whatever tools you, you have, because why not, right? Use the tools. They’re there for you, and, you know, they’re, they’re gonna improve your— they may improve your results and, you know, increase your efficiency. 

[00:46:00] Jessica Lockett: Thanks, Joe. 

[00:46:01] Mallory Hendry: Okay. Oh, yeah. Thanks, both of you. Someone else would like to know, how do we decide whether to use CAL, generative AI, or a combination of of both? 

[00:46:13] Joe Evangelista: I’ll take that one. So I think you kind of just need to start with what you need to accomplish, and, you know, it also depends on what data you have. So essentially, the needs of the project and the data that’s in scope will typically inform us of what technology to use. But before choosing, we should probably consider, of course, the review objectives, the data characteristics, and, you know, the last thing to really consider is the time constraints and the risk tolerance. So those 2 things, like the time constraints and the risk tolerance, can be quite important in your decision, whether it’s CAL or GenAI. And often, like, the hybrid approach is used here and there, and it’s often chosen when there’s one or more of the, you know, review objectives, data characteristics, time, and risk. When one of those, like, really deviate from what we typically see, we’ll want to try to build a hybrid approach to see if that— or to see if that would help throughout the process. And, you know, One technology can also provide further quality checks against the other. We have seen some workflows use that before, but I mean, the reality is you don’t— 

[00:47:42] Jessica Lockett: Essentially testing the 2 tools against each other as a quality control measure. Yeah, yeah, 

[00:47:46] Joe Evangelista: yeah, yeah. I mean, you know, why not, right? And yeah, or you run one and then the results from that are then used, the other GenAI or CAL is used for the results of the other. So in reality, you just, you don’t have to commit to one tool or one hybrid approach for the entire population. You know, you can see what fits, right? It’s, and you know, if you have any difficulties with deciding, you know, speak up and we can answer those questions. 

[00:48:21] Jessica Lockett: Yeah, I think, can I just add one quick point to that too? Is that, and it kind of goes, Abel, with what you were talking about. In terms of richness affecting things. Mm-hmm. And so I think sometimes we don’t necessarily know for sure, I mean, which one might be the recommended approach until, like Joe said, we do all of that analysis. But then also too understanding, well, how rich is this set and what are you trying to get out of it? Because that might also help us decide and push you towards one being more, you know, quote unquote efficient than the other. So, you know, a low richness, a high volume, low richness set might actually get more from a CAL than would sort of a lower volume, higher rich set. So there’s different— there’s also other variables that we would consider. 

[00:49:16] Joe Evangelista: You know, Jess, with that hybrid approach, your example of huge population, low richness, you can, you can use the— use CAL to isolate the most relevant documents or the most probably relevant documents in there, uh, and then run prompts against it to see. Because, you know, with, with the— with low richness, it is— and it depends on the, the, the review memo— there might be a lot of gray area in your relevance or not relevant decision. And if there is, you may potentially come out with a fairly low precision rate. And if you do, well, then there’s the GenAI tools that you can try and hone those— that smaller subset of rich documents, throw those through the GenAI tool, and then it would likely be able to provide you with better insight into what’s actually relevant or you know, a higher percentage of relevance. 

[00:50:17] Abel Kozatsky: I would add that as you’re going through a review, often, you know, when you’re looking at the data, things come up that you may not have expected at the beginning when you start sampling, or even when Council starts providing new instructions. So it’s very important to keep an eye on changes as you go through the review and to use the tool that’s best appropriate depending on what you’re facing at that point in time in the review. to achieve the ends that, you know, of the review. So that’s, that’s just one thing that I would add. 

[00:50:47] Jessica Lockett: Yeah, it makes sense. It’s like we’re talking about needing to be agile and creating overall workflows, but then also within specific workflows, be agile depending on what comes out in the course of that review. You might need to pivot a little bit or bring on an additional tool or whatnot. 

[00:51:05] Mallory Hendry: Great. Thank you, guys. What should Council ask their provider about validation? 

[00:51:15] Abel Kozatsky: So I would say first, I guess I’ll take this. So first, I would ask these questions at the start of a review, or even when you’re first engaging a potential provider. I think it’s a good idea to have these questions in mind so that you can frame the review correctly from the beginning. It’s always better to frame it correctly from the beginning. So Just in general, I would ask, like, how do you plan to validate the AIR or CAL model, depending on what’s being considered, and sort of ask for a recommendation from your provider of what they think would be the best potential tool to use and how to design the workflow based off the information that you provide. So in terms of the validation, you could ask stuff like, who will code the benchmark sample? Is it going to be counsel? Will it be the team of senior view lawyers? How would disagreements be resolved? And what will happen to documents that the tool can’t assess? So the, like, the AIR ineligible population or the COW ineligible population, how will those be addressed? And what are the workflows for those specific documents? Because those are often quite substantial. So that’s also a very important thing to consider. And then how will the How will the team test the material that’s set aside as not relevant in the end? So in terms of sampling to determine the elution rates so that you can, you know, to validate according to statistics. So those are all important considerations that you should consider asking counsel. Sorry, you should consider asking a provider about validation. Do you guys have anything to add, Jessica or Joe? 

[00:53:00] Jessica Lockett: I know, I totally— I think you’ve laid it all out. I mean, I think one thing additionally too is to talk about when you’re— if you’re using a service provider, talk about their documentation practices. So what are you getting at the end so that you as counsel, you know, have something there with you that if there are questions that come about your process, that you’re armed, you know, with the information. that you might need at that point. So I think talking about documentation practices is important 

[00:53:31] Joe Evangelista: too. I’ll add that, you know, when it comes to validation, I think part of that is not necessarily like how do you validate the, you know, the GenAI workflow or the CAL workflow. It’s how do you validate the entire process, right? Have you used search terms in in your data isolation or your review set isolation? And if that’s the case, did you validate those search terms? Because that’s kind of important too, right? And, you know, or did you scan all of the various data types that were ineligible? And if you did, what did you deem as being all not relevant? And if you did, did you do any sampling? And, you know, how do you validate all of these various approaches to not, not only review set isolation, but also just review in general, right? The validation doesn’t, doesn’t stop when it’s just GenAI or, or CAL. It’s throughout the process. 

[00:54:40] Jessica Lockett: Well, and okay, I have one point to layer on top of that too, is that, is that on it, is that people get really, uh, for, for, for right reasons, I understand, get really want to talk about, okay, what are they at? What were the metrics? And tell me if this can check off all the boxes that this is done correctly and right. But what’s interesting to me is that when these, these tools have come, they’ve brought with it validation where previously, up until these tools, we just trusted, you know, that, that human review was Totally sufficient and never got anything wrong. So I think the fact that these tools are coming in and bringing forth and have brought forth this notion that like we need to, even if this was a fully manual, no tool involved, there still needs to be some validation to the results. So anyway, I just, I’m just thinking out loud. 

[00:55:35] Abel Kozatsky: I agree. You also need to validate the human, like the human Yes. Human-coded documents. I mean, just because someone coded it doesn’t make it right, right? So you need to validate both the models you’re using, generative AI, CAL, whatever it is you’re doing. You know, if you’re doing targeted searches, you’re doing, you know, any kind of mass coding or any individual reviewed documents, that all has to be validated in the end. You typically— it’s a counsel’s, you know, coded sample that they’ll look over something to check the What the rates are. But anyway. 

[00:56:06] Jessica Lockett: Yeah, but that goes to your point about it being, uh, when you’re validating, say you bring GenAI into a process and you’re validating the GenAI results, that’s just one piece of it because you really need to validate the entirety of your process. And like what you’re seeing, and what you did say was that you can’t forget about this. There’s a lot that still requires human review that can’t go through the tool. So that has to be brought in to measuring in that, in that sort of completeness of validation. 

[00:56:32] Abel Kozatsky: So I guess another point about like final validation, I would advise sort of anyone using a provider services is to ask about doing a final validation sample. I think that’s probably really important. Typically, it’s kind of taken for granted, but I think that’s really— if you haven’t considered it, that’s very something very important to consider. 

[00:56:53] Jessica Lockett: That’s a good point. Oh yeah, 

[00:56:53] Joe Evangelista: there’s validation throughout, even during our reviews. Where council will validate us, you know, using a small sample of stuff that we’ve already reviewed. They’ll help with the validation metrics that we calculate. So the precision, recall, illusion, you know, all of that is just going and giving us more and more details on how to decide, is this review a success or do we need to continue on? 

[00:57:23] Jessica Lockett: Can we stop now? Can we stop and produce? 

[00:57:25] Joe Evangelista: Can we stop? 

[00:57:26] Jessica Lockett: Yeah, keep going. Yeah, 

[00:57:27] Joe Evangelista: absolutely. 

[00:57:29] Mallory Hendry: Yeah, those are great points. I think we have time for one last question before we let everyone go. Where does technology-assisted review outperform GenAI? And then where is GenAI creating capabilities technology-assisted review cannot do easily? 

[00:57:47] Joe Evangelista: I could start on that one. 

[00:57:48] Jessica Lockett: Sure, I have some 

[00:57:50] Joe Evangelista: thoughts. Okay, so where does TAR outperform GenAI? I think, you know, it’s, it’s not outperforming. It’s, it’s what is the use case for one or the other? And in terms of CAL being a better use case, I think it’s that large population and low richness. But you’ll only find that out if you run a richness sample on the, on the set of documents. But considering this is a 2-part question, The second part being what is GenAI, what capabilities is GenAI adding over and above what CAL already has? And I think the limitation there with CAL is that it has to work simply based on the extracted text of the documents. GenAI is adding capabilities to analyze audio, video, images. And so that’s an— that’s some very exciting things because a lot of those Those data types used to be ineligible, or, you know, we can’t, we can’t assess them using CAL at that time. But now maybe we can. 

[00:59:01] Jessica Lockett: Well, and not yet. Thank you, Joe. You’ve summed it up perfectly. I think just on top of that too, there are additional features, you know, that some of these GenAI tools have, which is providing document summaries and things like that. So there’s Yeah. That doesn’t go necessarily into the analysis and its performance, but in terms of, you know, deploying them over certain datasets, many of them, if not all of them, will provide those extra features. It will give you a summary. It will tell you, you know, because of this in the document is why we think it goes, you know, responds positively to your prompt. Or so it’s giving an extra analysis that you can use for, for further reasons, you know, sort of down the line as you’re witness prepping and things like that. So just to point that out. 

[00:59:57] Joe Evangelista: Yeah, no, that’s excellent. 

[01:00:00] Abel Kozatsky: Yeah, there’s also limitations with the tool. There’s like size limitations for documents that can be analyzed using generative AI as it currently exists. I don’t know. 2 years from now what it will look like, but right now there are limitations in terms of size of documents and just like the amount of issue codes that you can add using the tool. So those are 2 other things to consider when considering CAL versus generative AI use as well, just to build on what Joe and Jessica are saying. 

[01:00:26] Joe Evangelista: And with whatever the tool brings, we will be adaptable. 

[01:00:30] Abel Kozatsky: Yes. 

[01:00:31] Mallory Hendry: That’s a perfect, perfect note to end on there, everybody. Right on time. Thank you so much to our hosts today, Jessica, Abel, Joe, you know, for sharing your insights. And this was such a wonderful conversation. As mentioned, everyone, a recording will be sent out to you shortly. But in the meantime, if you’d like to learn more, you can visit trustarray.com, and we’ll see you all at the next Canadian Lawyer webinar. Have a great day, everybody. 

[01:00:57] Jessica Lockett: Great, thank you so much. 

[01:00:59] Abel Kozatsky: Thank you. 

[01:00:59] Joe Evangelista: Bye, everyone.