> Defendant (as dictated by AI): The Supreme Court ruled in Johnson v. Smith in 1978...

> Judge: There was no case Johnson v. Smith in 1978.

LLMs hallucinate, and there is absolutely no space for hallucination in a court of law. The legal profession is perhaps the closest one to computer programming, and absolute precision is required, not a just-barely-good-enough statistical machine.

That said, if it's such a catastrophically stupid idea, I'm not really sure why it had to be shot down so harshly: seems like that problem would elegantly solve itself. I assume the real reason it was shot down was out of fear that it would work well. Does anyone else have a better explanation for why there was such a visceral response?

I can't speak for lawyers in general or what everyone's motivations would be, but my initial reaction was that it seemed like a somewhat unethical experiment. I assume the client would have agreed or represented themselves, but even there -- legal advice is tricky because it's advice -- it feels unethical to tell a person to rely on something that is very likely going to give them sub-par legal representation.

Sneaking it into a courtroom without the judge's knowledge feels a lot like a PR stunt, and one that might encourage further legal malpractice in the future.

I assume there are other factors at play, I assume many lawyers felt insulted or threatened, but ignoring that, it's not an experiment I personally would have lauded even as a non-lawyer who wishes the legal industry was, well... less of an industry. The goal of automating parts of the legal industry and improving access to representation is a good goal that I agree with. And maybe there are ways where AI can help with that, sure. I'm optimistic, I guess. But this feels to me like a startup company taking advantage of someone who's in legal trouble for a publicity stunt, not like an ethically run experiment with controls and with efforts made to mitigate harm.

Details have been scarce, so maybe there were other safety measures put in place; I could be wrong. But my understanding was that this was planned to be secret representation where the judge didn't know. And I can't think of any faster way to get into trouble with a judge then pulling something like that. Even if the AI was brilliant, it apparently wasn't brilliant enough to counsel its own developers that running experiments on judges is a bad legal strategy.

Now the question is this. If an AI is doing something illegal like practicing law, how does one sanction an AI?

Edit: found this:

https://jolt.richmond.edu/is-your-artificial-intelligence-gu...

"A person is presumed to be practicing law when engaging in any of the following conduct on behalf of another"

Every state seems to use the word "person" in their rules.

An AI is not a person, and therefore can't be sanctioned for practicing law - my take anyway.

If non-persons can be prosecuted for illegally practicing law, then those non-persons must have the right to get a license. IMHO.

As far as I'm aware, no LLM has reached sentience and started taking on projects of its own volition. So it's easy - you sanction whoever ran the software for an illegal purpose or whoever marketed and sold the software for an illegal purpose.

https://apnews.com/article/sandy-hook-school-shooting-reming...

https://www.gov.ca.gov/2022/07/12/new-california-law-holds-g...

The second link feeling much closer to direct government action.

"Personhood" in a legal sense doesn't necessarily mean a natural person. In this case, the company behind it is a person and is practicing law (so no pro se litigant using the company to generate legal arguments). In addition, if you want something entered into court, you need a (natural person) lawyer to do it, who has a binding ethical duty to supervise the work of his or her subordinates. Blindly dumping AI-generated work product into open court is about as clear-cut an ethical violation as you can find.

To your larger point, law firms would love to automate a bunch of paralegal and associate-level work; I've been involved in some earlier efforts to do things like automated deposition analysis, and there's plenty of precedent in the way the legal profession jumped on shepardizing tools to rapidly cite cases. Increased productivity isn't going to be reflected by partners earning any less, after all.

I'd prefer that the boilerplate actually be reduced instead, but... I don't have any issue with someone using AI to target tasks that are essentially copy-paste operations anyway. I think this was kind of different.

> If an AI is doing something illegal like practicing law, how does one sanction an AI?

IANAL, but AIs don't have legal personhood, so it would be kind of like trying to sanction a hammer. I don't think that the AI was being threatened with legal action over this stunt, DoNotPay was being threatened.

In an instance where an AI just exists and is Open Source and there is no party at fault beyond the person who decides to download and use it, then as long as that person isn't violating court procedure there's probably no one to sanction? It's likely a bad move, but :shrug:.

But this comes into play with stuff like self-driving as well. The law doesn't think of AI as something that's special. If your AI drives you into the side of the wall, it's the same situation as if your back-up camera didn't beep and you backed into another car. Either the manufacturer is at fault because the tool failed, or you're at fault and you didn't have a reasonable expectation that the tool wouldn't fail or you used it improperly. Or maybe nobody's at fault because everyone (both you and the manufacturer) acted reasonably. In all of those cases, the AI doesn't have any more legal rights or masking of liability than your break pads do, it's not treated as a unique entity -- and using an AI doesn't change a manufacturer's liability around advertising.

That gets slightly more complicated with copyright law surrounding AIs, but even there, it's not that AIs are special entities that have their own legal status that can't own copyright, it's that (currently, we'll see if that precedent holds in the future) US courts rule that using an AI is not a sufficiently creative act to generate copyright protections.

Law is different because the bar has a legally enforced monopoly on doing legal work.

DoNotPay was being threatened. But they weren't practicing law - they were just providing legal tools.

My point is that were in uncharted legal territory. Perhaps ask the AI what it thinks ;)

I don't see how this would decrease DoNotPay's liability.

Regardless of how you feel about the bar, I don't think that changes anything about who they would sanction or why. Having a legal monopoly means they're even less likely to go along with a "the AI did it, not me" explanation than a normal market would be.

I mean, no matter what, they're not sanctioning the AI. They don't recognize the AI as a person, they recognize it as a tool that a person/organization is using to perform an action.

Its not and you don’t.

When a legal person (either a natural person or corporation) is doing something illegal like unauthorized practice of law, you sanction that person. The fact that they use an AI as a key tool in their unauthorized law practice is not particularly significant, legally.

I'm going to sit on that particular hill and see what happens. Even if DoNotPay's AI is not ready to do the job, the idea that AI could one day argue the law by focusing on logic and precedent instead of circumstance and interpretation is exceedingly threatening to a lawyer's career. No offense intended to the lawyers out there, of course. Were I in your shoes, I'd feel a bit fidgity over this, too.

other fields like journalism not so much.

That was my initial response too.

Artists, programmers, musicians, teachers are threatened... but shrug and say "that's the future, what can you do". If lawyers feel "threatened" by AI, they get it shot down.

I suddenly have a newfound respect for lawyers :)

Yet if we think about it, we all have exactly the same tools at our disposal - which is just not playing that game. Difference is, while most professions have got used to rolling with whatever "progressive technology" is foisted on us, lawyers have a long tradition of caution and moderating external pressure to "modernise". I'm not sure Microsoft have much influence in the legal field.

This is a thing that lots of people say about unethical businesses, and I'm a little skeptical about it at this point. A couple of objections I have:

- You have a constitutional right to legal representation when accused of a crime by the US government, and while we don't to abandon people who are suffering now because of some theoretical future fix, we also don't want to normalize the idea that constitutional rights only exist when a private market accommodates them. That's explicitly a bad direction for the country to go.

- Saying "well, this works here and now, and people don't have access to anything better" is in my mind only a really effective argument when we know that the thing here and now actually works. But we don't know that this works, which changes a lot about the equation.

- Is sneaking an AI into a courtroom through an earpiece really a cost-effective accessible strategy for poor people? Nothing about this screams "accessibility" to me.

I think summing up the last two points, if the AI was proven to actually work in a court of law, and was an accessible option, then sure, at that point I think the argument would have a lot more weight. It wouldn't be ideal, it would be a bad state for us to be in because your constitutional rights should not depend on an AI. But I could see a strong argument for using the AI in the meantime.

But that doesn't mean that DoNotPay should do unethical things right now to get to that point. The way that your choice is being phrased is begging the question: it assumes that the AI is the only choice other than no representation, that it does work, and that it will produce better outcomes.

But we don't actually know if the AI does work in a court of law, and DoNotPay's decision was to "move fast and break things"; it was to start releasing it into the wild without knowing what would happen. We don't know if asking people to represent themselves with a secret earpiece is a good legal strategy or if it's accessible. We don't know what happens when something goes wrong. We don't know that this actually is a working solution. But they were putting someone's legal outcome on the line anyway.

I think there's a big difference between making an imperfect solution available to poor people because we don't have anything better to offer, and using poor people as experimental fodder to build an imperfect solution that might not work at all. There's a lot of assumption here that using their AI would be better than representing yourself, and I don't know that's true. A judge is not going to pleased with being used as an experiment. And I've been hearing people say that the AI subpoenaed the officer involved in the ticket? That's not a good legal strategy.

The proper way to build a solution like this is to make sure it works before you start using it on people, and I think it's unethical to give someone bad legal advice and to try and justify it because giving that person bad legal advice might allow the company to help other people down the line. A lot of our laws around legal representation are predicated on the idea that legal advice should be solely focused on the good of the client, and not focused on the lawyer's career, or on someone else the lawyer wants to help, or on what the lawyer will be able to do in the future. Based on what we know about the state of the AI today, it doesn't seem like DoNotPay was thinking solely about the good of the person they were advising. We really don't want the legal industry to be an industry that embraces "the ends justify the means."

Run it multiple times with multiple defendants, set up a control group that's receiving remote advice from actual lawyers, mask which group is which to the judges, then ask the judge(s) at the end to rank the cases and see which defendants did best.

That would be a lot more work, but it would also be much higher quality data than what they were trying to do.

And also

> That would be a lot more work, but it would also be much higher quality data

I don’t know much about the field of law, but anecdotally it doesn’t strike me as particularly data driven. So I think, even before introducing any kind of AI, the above would be met with a healthy dose of gatekeeping.

Like the whole sport of referencing prior rulings, based on opinions at a point in time doesn’t seem much different than anecdotes to me.

But I’d love to be proven wrong though.

That said. The consequence of most traffic tickets is increased insurance and a fine. Yes these do have an impact on the accused, but they are the least impactful legal cases, so it would make sense to focus on them as test cases.

I have some past experience working in the courts in my state, and I know there are many judges who are perfectly fine with dismissing minor traffic infractions for no reason other than that they feel like it. If you've got an otherwise clean traffic abstract and sent in a reasonable sounding letter contesting the infraction, these judges probably aren't going to thoroughly read through every word of it and contrast it with what was alleged in the citation. They don't really care about the city making an extra $173 off your parking ticket -- they just want to get through their citation reviews before lunch. Case dismissed.

So I am not surprised at all by the success of DoNotPay for minor traffic infractions. Most traffic courts are heavily strained by heavy case loads. If you give them a reason to throw your case out so they can go home on time, by all means, they will take it.

Someone using it in an actual courtroom would make a boneheadedly dumb argument or refer to a nonexistent precedent or something. Then maybe the judge gets upset and gives them the harshest punishment or contempt of court or they just lose the case. They may or may not ever get a chance to fix it.

A failure mode of jail time and/or massive fines for your customers doesn't sound all that elegant to me. This isn't a thing to show people cat pictures, I don't think move fast and break things is a good strategy.

Not to say that there aren't some entrenched possibly corrupt and self-serving interests here. But that doesn't mean they don't have a point.

That sounds like a horrible judge, maybe this AI can be used to sniff them out and get rid of them.

On the other hand, here's a lawyer who thinks it would not count as legal representation, but hasn't seen the arguments made against it yet. Food for thought.

https://youtu.be/r-gvArVKfUM

Why is the robot not "speech"?

"Speech" would be like publishing a book about self-representation. "Counsel" would be providing advice to a defendant about their specific case. The machine would be in the courtroom advising the defendant on their trial, so that's counsel.

[EDIT] That is, would they be immune from e.g. malpractice if they did this so they "weren't representing" the defendant?

If the book was a generalized "choose your own adventure" where you compose a sensible legal argument from selecting a particular template and filling it in with relevant data - use of the book essentially lets the user find the pre-existing legal advice that is relevant to their situation.

Chatbots as a system are arguably a lot more like the latter than the former - its a tool that someone can use to 'legal advise' themselves.

> Here's how it was supposed to work: The person challenging a speeding ticket would wear smart glasses that both record court proceedings and dictate responses into the defendant's ear from a small speaker.

Also, probably wouldn't matter. The interactive human-ish-like nature might cross the line to being considered as counsel, even if you said it wasn't. See my response to your other comment.

It doesn't really matter if it'd work well or poorly. Lawyers don't want to be replaced, and being a lawyer entails a great ability to be annoying to delay/prevent things you don't want to happen.

One of those is that the lawyer arguing a case is properly credentialed and has been admitted to the bar, and is a professional subject to malpractice standards, who can be held responsible for their performance. An AI spitting out statistically likely responses can't be considered an actual party to the proceedings in that sense.

If a lawyer cites a non-existent precedent, they can make their apologies to the court or be sanctioned. If the AI cites a non-existent precedent, there's literally no way to incorporate that error back into the AI because there's no factual underlying model against which to check the AI's output--unless you had an actual lawyer checking it, in which case, what's the point of the AI?

Someone standing in court, repeating what they hear through an earpiece, is literally committing a fraud on the court by presenting themselves as a credentialled attorney. The stunt of "haha, it was really just chatGPT!" would have had severe legal consequences for everyone involved. The harsh response saved DoNotPay from itself.

IANAL, but I would bet the level of effort to fact check an AI's output would be orders of magnitude lower than researching and building all your own facts.

I used it to generate some ffmpeg commands. I had to verify all the flags myself, but it was like 5 minutes of work compared to probably hours it would have taken me to figure them all out on my own.

Fact-checking nonsensical output would take a lot longer than researching a single body of law, which you can generally do by just looking up a recent case on the matter. You don't need to check every cite; that will have been done for you by the lawyers and judges involved in that case.

But checking every cite in an AI's output: many of those citations won't exist, and for the ones that do, you'll need to closely read all of them to confirm that they say what the AI claims they say, or are even within the ballpark of what the AI claims they say.

Of course there could be a difference between an reading all the cases an AI says are important and actually finding the important cases including ones the AI didn't point you at. However this is not what the bet was about.

Actually fact-checking an AI requires finding and reading all the precedent yourself to verify that the AI has both cited accurately and not missed contradictory precedent that is more relevant (whether newer, from a higher court, or more specifically on-point.)

If it has got an established track record, just as with a human assistant, you can make an informed decision about what corners you can afford to cut on that, but then you aren't really fact-checking it.

OTOH, an AI properly trained on one of the existing human-curated and annotated databases linking case law to issues and tracking which cases apply, overrule, or modify holdings from others might be extremely impressive—but those are likely to be expensive products tied to existing offerings from Westlaw, LexisNexis, etc.

To the cost. If it removed the need for one legal assistant or associate then anything less than the cost of employing said person would be profit. So if it cost < 50k a year you'd be saving. (cost of employing is more than just salary)

Why would this be different with an AI assistant to help you? It's not a binary "do or do not". Just because you have an assistant doesn't mean you don't do anything. Kind of like driver assist can handle some of the load vs full self-driving.

> But checking every cite in an AI's output: many of those citations won't exist, and for the ones that do, you'll need to closely read all of them to confirm that they say what the AI claims they say, or are even within the ballpark of what the AI claims they say.

But you'd have to do this anyway if you did all the research yourself. At least the AI assistant can help give you some good leads so you don't have to start from scratch. A lazy lawyer could skip some verifying, but a good lawyer would still benefit from an AI assistant as was my original bet, just like they would benefit from interns or paralegals, etc. And all those interns and paralegals could still be there, helping verify facts.

No, that's exactly the opposite of what I'm saying. If you did the research yourself, you wouldn't need to verify every cite once you find a relevant source/cite, because previous lawyers would have already validated the citations contained within that source. (A good lawyer should validate at least some of those cites, but frequently that's not necessary unless you're dealing with big stakes.)

And the AI assistant, at least this one and the ones based on ChatGPT, don't provide good leads. They provide crap leads that not only don't exist, but increase the amount of work. And any "AI" based on LLM will never be capable of providing good cites, because they'll never understand what they're reading and/or citing, and they'll miss relevant citations that are not statistically likely (i.e., new case law, or cases with similar facts, or similar law, or otherwise similar contexts that can be applied to the case at hand) that a context-aware AI or living breathing human would find easily.

At best, LLM-based AI might be able to help people with very simple legal situations. But you don't need AI for that. A single decision tree is easier to implement, and it's even easier to verify the domain-specific process and outcomes to make sure you don't get something silly like happened with this "AI".

If it was AI-assisted lawyer, it would be a whole different discussion. Aside from requiring a live feed of interactions to a remote system and other technical details, “lawyers using supportive tools while exercising their own judgement on behalf of their client” isn’t controversial the way marketing an automated system as, or as a substitute for, legal counsel and representation is.

I think making the connection between the predictive output and an underlying representation of reality is the next great step, but until that happens, chatGPT's output is just amazing mimicry of human language.

The title of the article seems misleading.

A techbro who doesn't appear to be a lawyer or has any understanding of the law wants to use AI so people can defend themselves. It doesn't seem like any of this was done with input from any bar associations. Without seeing the emails and "threats", and ignoring the emotional language it sounds like these people were helping him out:

>"In particular, Browder said one state bar official noted that the unauthorized practice of law is a misdemeanor in some states punishable up to six months in county jail."

Were these emails "angry" or just stating very plainly and with forceful language, that if you do this without the AI having the appropriate qualifications, you are most probably going to jail?

It even sounds like Browder didn't really widely publicise the fact that a case defended by an AI was about to happen.

>As word got out, an uneasy buzz began to swirl among various state bar officials, according to Browder. He says angry letters began to pour in.

Really sounds like these letter writers did him a favour.

To avoid the catastrophy that makes it a catastrophically bad idea.

> I assume the real reason it was shot down was out of fear that it would work well. Does anyone else have a better explanation for why there was such a visceral response?

It had already worked badly (subpoenaeing the key adverse witness, who would provide a basically automatic defense win, and one of the most common wins for this kind of case, if they failed to show up.)

We are so close from a new disruptive revolution where a lot of jobs (not just lawyers) will be made obsolete. Possibly similar to inventions like assembly lines, or cars. Such an exciting time to be alive!

1) The legal profession tries to instill a sense of ethics into the lawyers they train, and what DoNotPay was proposing violated that ethics. I don't want to overyhype the legal profession, but many (maybe even most) lawyers really do want to do the right thing, which by their lights means clients get the best representation possible. Which an LLM currently is not, so you get a visceral reaction from people at what they see as discussion of/advocacy of unethical conduct.

2) As a practical matter, it was likely to yield very bad outcomes for all concerned, including professional and legal consequences. Bluntly, DoNotPay was proposing to do something illegal, it really could have resulted in jail time, and possible disbarrment for any lawyers who were involved. For good or ill, judges have immense power to control what goes on in their courtrooms, and the risk of a judge taking offense to this is high. And the higher profile the case (and DoNotPay was offering $1m for a lawyer who would repeat whatever the chatbot said in a Supreme Court case) the higher the stakes. That really could be a career ending mistake for a junior lawyer (plus, almost by definition, anything that reaches the Supreme Court is important; bad representation could throw the result, with potentially terrible consequences for the country).

3) This is not, by any means, the first attempt to try and automate or streamline the provision of basic legal services. And it's a field ripe for such things; simple stuff like many rental agreements, employment contracts, sueing in small claims court, divorce agreements, etc., etc. all seem like they should be able to be generated by filling out a form and pressing a button, not hiring a high priced professional to craft a bespoke document. But over and over, such attempts have failed badly, and lawyers are getting reflexively defensive to anything that looks like that, not because they threaten their jobs, but because they keep being so terrible.

4) And DoNotPay looks a lot like these previous failures. See, eg, https://www.techdirt.com/2023/01/24/the-worlds-first-robot-l... which makes them look less like some cutting edge AI lawyer that will put an army of paralegals out of work, and more like yet another shitty site trying to resell Mechanical Turk at an enormous markup. Now, maybe what they're showing to the public is entirely different than what they want to use in courtrooms, but...it doesn't build confidence.

In many ways I think this is analagous to self driving cars. There is scope for enormous gains here, it's probably inevitable eventually, and it will probably put a lot of delivery drivers, taxi drivers, and truck drivers out of a job when it finally arrives. But right now deciding to hook your new self driving AI up to an unmarked truck and do a secret test in an unnamed US city without warning anyone or getting any licenses or permits is unethical and illegal, and you'd expect a very negative reaction if you announce your plans on Twitter. Especially if your website offers some software that they claim can control an RC car, but when you try it, it keeps steering into walls.

> I assume the real reason it was shot down was out of fear that it would work well

The next lawyer (or AI expert!) I see who thinks it would work well will be the first. There's just an enormous mismatch between the requirements, what chatbots in general seem to be capable of right now, and what DoNotPay in specific seems to be capable of. (Again, see the the Tech Dirt link.)

And again, note that while there's a pretty strong argument that it should be faster, easier, and cheaper to create a rental agreement (or whatever), DoNotPay was making a big deal out of how they wanted to argue in front of the Supreme Court. DoNotPay doesn't seem like they're going to nail the "draft a rental agreement for cheap" any time soon, but maybe they (or someone else) can. But handing a Supreme Court case? Obviously not. Now you might argue (and one certainly hopes!) the whole Supreme Court thing was just an unserious PR stunt, but when a company is making undifferentiated claims that their tech can do something semi-plausible and something entirely impossible, it makes the entire thing look like a scam created by or aimed at people who just don't know any better. Not a good look!

In short: I think the negative reaction is very, very understandable.

If something that's providing input to a formal legal process (which, let's not forget, means false or inaccurate statements have real and potentially prejudicial repercussions), "makes facts up and exaggerates", then there seems to be no reason they should be talking about taking this anywhere near a courthouse.

This feels a lot like "move fast and break things" being applied - where the people silly enough to use this tool and say whatever it came up with would end up with more serious legal issues. It seems like that only stopped when the founder himself was the one facing the serious legal issues - 'good enough for thee, but not for me'...

I think what many are overlooking is that bad inputs to the legal system can jeopardise someone's position in future irretrievably, with little or no recourse (due to his class action/arbitration waiver). Once someone starts down the road of legal action, there's real consequences if you get it wrong - not only through exposure, but also through prejudicing your own position and making it impossible to take a different route, having previously argued something.

[0] https://twitter.com/SemaforComms/status/1618306993902743555

I’ve been using copilot for half a year now and it’s helpful, but often wrong. I carefully verify anything it gives me before using it. I’ve had one bug that made it to production where I think copilot inserted code that I didn’t notice and which slipped past code review. I’ve had countless garbage suggestions I ignored, and a surprising amount of code that seemed reasonable but was subtly broken.

This will still require a human lawyer (and/or intern, depending on the stakes) to check its output carefully. I am not now, nor have I ever been afraid that AI is coming after my job. When it does, we’re dangerously close to general AI and a paradigm shift of such magnitude and consequence that it’s called The Singularity. At which point we may have bigger worries than jobs.

I'm not saying this is good but come on. Humans do that all the time, why aren't we so harsh on humans?

> I am not now, nor have I ever been afraid that AI is coming after my job

I am. This thing amazed me, and even if it won't be able to 100% replace humans (which I doubt), it can make juniors orders of magnitude more productive for example. This will be a complete disruption of the industry and doesn't bode well for salaries.

People who are just skating by may have cause for concern. But those are the people with the most to gain from it, so maybe not even them.

Demand is so high in the business, I have trouble imagining that any tool could make a meaningful impact on that.

It would need to multiply productivity by a huge number, and nothing has that impact. Copilot is barely, optimistically, above 10%. I don't really think I get that much, but just for arguments sake.

I wonder if that is because of the training set, us humans are often wrong or different. If given a room of programmers and asked to implement a Fibonacci algorithm would they all get it right, would they all do it via iteration, recursion, dynamic programming. Co-pilot might not replace you, but it just needs to replace some of those programmers. Then add tools like automated AI reviews or integration tests for example and now you removed another population of tech workers.

I am not sure if that is cause for alarm or the fact that such improvements could be rather beneficial. Some tools will replace people, some will be assistive, and as they improve and other layers are added they will reduce the need for people in areas, improving efficiency and productivity. Robots in manufacturing for example, improve productivity and reduce human labor.

And this leads on to this, these tools are narrow to general and have achieved this in a very short period of time. The cost factors have also massively reduced, if you could pay OpenAI $10 a month to make 1000 mistakes and still deliver code, or a human $120k a year to do the same then which one would you target. AI might not be coming for your job soon but it will be taking away your options to get a job. This is not unique to AI it is the basis for any technological improvement vs labor, yes new ideas and opportunities may come out of this but I don't think they will be equal in volume.

The issue I think is that it's pulling from too many sources. There are plenty of sources that are pretty machine readable that will give it good answers. There's a lot of training that can be eked out from the legal databases that already exist that could make it a lot better. If it takes in too much information from too many sources, it tends to get garbled.

There are also a lot of areas where it will confuse concepts from different areas of law, like mixing up criminal battery with civil battery, but that's not the worst of the problems.

No, this is a common misunderstanding about the way these things work. A LLM is not really pulling from any sources specifically. It has no concept of a source. It has a bunch of weights that were trained to predict the next likely word, and those weights were tuned by feeding in a large amount of text from the internet.

Improving the quality of the sources used to train the weights would likely help, but would not solve the fundamental problem that this isn't actually a lossless knowledge compression algorithm. It's a statistical machine designed to guess the next word. That makes it fundamentally non-deterministic and unsuitable for any task where factual correctness matters (and there's no knowledgeable human in the loop to issue corrections).

check the chat bot on you.com:

https://you.com/search?q=list%20top%205%20best%20selling%20c...

A citation in legalese is very common. A citation that is similar or identical to actual citations, in similar contexts, is therefore an excellent candidate for the completion. A fake citation that looks like a real citation is also a rather good candidate, and will sometimes squeak past the "is this real or fake?" metric used to evaluate generated potential responses.

This may seem like "pulling from a source" but there is no token, semantic information, or even any information in the model about where and when the citation was encountered. There is no identifiable structure or object (so far as anyone can tell anyway) in the model that is a token related to and containing the citation. It just learns to create fake citations so convincingly, that most of the time they're actually real citations.

In the case of citations, you really need a language model & a fact model. The language model then passes over to the fact model, then back to the language model. This means double(+) training.

I suppose the fact model could include things like Wolfram (also discussed on HN).

"In California, for example, the California Supreme Court in the case of Lemon v. Kurtzman (1941) held that a vehicle which did not meet the manufacturer's express warranty was a "lemon" and the manufacturer was liable for damages."

I dont think that case exist, there is a first amendment case Lemon v. Kurtzman, 403 U.S. 602 (1971) though.

I can't find any reference to Kurtzman or 1941 in any of the references. I think the answer is that the AI generating the text, and the code supplying the references are distinct and do not interact.

If you have the training set, and models that summarize text and/or assess similarity, and some basic search engine style tools to reduce or prioritize the problem space, it seems intuitively possible to synthesize probably-credible citations from a draft response without the response being drawn from citations the way a human author would.

Kind of a variant of how plagiarism detection works.

That kind of chatbot has different limitations that would make it unsuitable to be an unsupervised legal advice generator.

It really doesn’t strike me as true that law requires absolute precision. There are many adjacent (both near and far) arguments that can work in law for any given case, since the interpreter is a human. You just need no silly mistakes that shatter credibility, but that’s very different from “get one thing wrong and the system doesn’t work at all or works in wildly unexpected ways.”

Low end law will be one of the first areas to go due to this tech. DoNotPay actually has already been doing this stuff successfully for a while (not in court proceedings themselves though).

I'm not at all opposed to using LLMs in the research and discovery phase. But having a naive defendant with no legal experience parroting an LLM in court is deeply problematic, even if the stakes are low in this particular category of law.

Is it more problematic than this person 1) not being represented, 2) having to pay exorbitant fees to be represented, or 3) having an extremely overworked and disinterested public defender?

I’m not convinced.

The idea that we need to wait to do this stuff until the people whose profession is under threat give “permission” is dismissible on its face and is exactly why we should be doing this is as quickly as possible. For what it’s worth, I mostly agree with you: I’m doubtful the technology is there yet. But that’s a call for each defendant to make in each new case and so long as they’re of sound mind, they should be free to pick whatever legal counsel they want.

You’re leaving off being put in jail for contempt of court, perjuring oneself, making procedural errors that result in an adverse default ruling, racking up fines, et cetera. Bad legal representation is ruinous.

I get that lawyers think their profession is important (it is) and that by and large they’re positive for their clients (they are), but there are a lot of people who simply do not have access to any worthwhile representation. I saw Spanish-speaking kids sent to juvenile detention for very minor charges conducted in English with no interpreter and a completely useless public defender. So in my view that is the alternative for many people, not Pretty Decent Representation.

There are people who can stomach the downside risks to push this tech forward for people who cannot stomach the downside risks of the current reality.

They’d be better off representing themselves than trusting a non-lawyer “lawyer” that subpoenas no-show cops. Not another Andreessen-backed cluster.

Someone with an 80 IQ and an elementary school grasp of English is going to get absolutely dragged by the system with or without a "robot lawyer" if they insist on fighting without competent representation, but they'd probably still stand a better chance of getting off a fine on a technicality if they weren't paying a VC-backed startup to write a letter make sure the cops turned up to testify against them.

They'd also be more likely to not get absolutely dragged if they listened to a human that told them not to bother than a signup flow that encouraged them to purchase further legal representation to pursue the case.

If your goal is to ensure poor folks are pushed towards paying an utterly unaccountable service further fees to escalate legal cases they don't have the knowledge to win so the LPs of Andreesen Horowitz have that bit more growth in their portfolio, I can see how you would think differently.

Invest in a company doing this properly and not pushing half-baked frauds into the world. Supervised learning, mock trials. You’re proposing turning those poor folk into Guinea pigs, gambling with their freedom from afar.

And no, I’m not. I don’t think Defendant Zero (or one, or two, or 100) should be people whose lives would be seriously affected by errors. I’m pretty sure DNP doesn’t either.

The CEO tweeting they fucked up a default judgement [1]. That not only communicates model failure, but also lack of domain expertise and controls at the organisational level.

[1] https://mobile.twitter.com/questauthority/status/16175419211...

What makes you think that? What they are doing is illegal. It's the unlicensed practice of law.

But doing so generates lots of publicity for their online wizards that send out boilerplate legal letters.

The CEO tweeted about the system subpoenaing the traffic cop. If they actually built a system which is so advanced it can handle a court case in real time and yet so ignorant of the basics of fighting a traffic ticket it subpoenas the traffic cop it's... a very odd approach to product management for the flagship product of a legal specialist, and a bit scary to think anyone would use it. Easier to make the mistake of claiming your flagship system does stuff it shouldn't be doing if it's just vaporware and you haven't put too much thought into what it should do

So, DoNotPay is basically just a scam taking money from people without providing anything of value.

But a fun fact is that magistrates generally aren't required to pass the Bar, nor hold a law degree. Most states require extremely basic courses of 40 or so hours of training. I know of a magistrate that has tried numerous times to pass the Bar and has failed. I'm not sure how much competence our system mandates.

And even when they lose the win. Look at the publicity they are getting.

> Defendant (as dictated by AI): The Supreme Court ruled in Johnson v. Smith in 1978...

> Judge: There was no case Johnson v. Smith in 1978. Case closed, here's your fine.

Next time please be more careful picking the lawyer.

(Your hypothetical includes a fine, so it isn't clear whether the offense in your hypothetical is one with, shall we say, enhanced sixth amendment protections under Gideon and progeny, or even one involving a criminal offense rather than a simple civil infraction, but...) in many cases lack of a competent attorney is considered structural error, meaning automatic reversal.

In practice, that means that judges (who are trying to prevent their decisions from being overturned) will gently correct defense counsel and guide them toward competence, something that frustrated me when I was a prosecutor but which the system relies upon.

BREAK BREAK

My jurisdiction (the military justice system) is a bit of an oddball, but I generally (softly) disagree with your last paragraph.

In jurisdictions with good criminal justice systems, most cases don't take that long to review. Possession of marijuana case, the officer stopped you for a broken taillight, smelled or claimed to smell marijuana, asked you if it was okay if he takes a look. You said okay. He finds a tiny amount of a green leafy substance in a small plastic bag. He says, "Hey look, marijuana is not that big a deal but lots of this stuff is laced with other things. This is just marijuana, right?" and you respond "Yes, sir, just marijuana, I don't mess with any real drugs." Prosecution is offering a diversion deal with probation and no permanent criminal record.

The correct answer in that case is to take the deal. We are not going to win by arguing that Raich was wrongly decided or that the officer lied about smelling it (because we're probably talking state court, so Raich doesn't matter, and you consented to the search, so the pretext, even if it was pretextual, also doesn't matter). We also aren't going to win attacking the chain of evidence, because the drug lab results don't matter, because you admitted it's marijuana.

In that case, yes, I'm going to take all of about 4 minutes to strongly advise the client to take the deal.

The rest of my comment I dropped the '/s' for. I think it's wild some people think current LLMs can replace lawyers... The absolute best I would think they could currently do is maybe speed up research for paralegals. I was just imagining expecting a judge to QA a private companies software and thought it was really funny.

I know it sounds scary when stats get thrown around like "Over 90% of defendants plead guilty without a trial!" but that's usually pursuant to a generous deal from an overworked prosecutor's office who absolutely does not want to do a freaking jury trial because you were desperate to get to work and couldn't get a ride.

Obviously, if the underlying reason your license was suspended is extreme (vehicular manslaughter, 3rd+ DUI) then their tune will probably change, but that's an extreme minority of cases. Those are the ones that go to trial.

"All rise!"

Also, the original post is about the traffic ticket. I'm pretty sure if the judge hears a reference to something he had never heard before, he'll be like "huh? wtf?"

Rule 1 of traffic court: If the state's witness (the ticketing officer) doesn't show, defendant gets off. You do not subpoena that witness, thereby ensuring they show up.

If the AI or its handlers cannot be trusted to pregame the court appearance even remotely well then no way in hell should it be trusted with the actual trial.

You want to run this experiment? Great, lets setup a mock court with a real judge and observing lawyers and run through it. But don't waste a real court's time or some poor bastard's money by trying it in the real world first.

A reminder that "Move fast and break things" should never apply where user's life or liberty is at stake.

Would you have a source?

Mike Dunford is a practicing attorney. Embedded tweet is of a non-lawyer who screenshotted Joshua Browser DoNotPay CEO) saying the subpoena had been sent. He has since deleted those tweets as DNP backs away from this plan.

Yes, in this case, it would just be a parking ticket, but the legal system runs on precedent and it's safer to hold a strict line than to create a fuzzy "well, it depends on how much is at stake" line. If we know that ChatGPT is not equipped to give legal advice in the general case, there's no reason to allow a company to sell it as a stand-in for a lawyer.

(I would feel differently about a defendant deciding to use the free ChatGPT in this way, because they would be deliberately ignoring the warnings ChatGPT gives. It's the fact that someone decided to make selling AI legal advice into a business model that makes it troubling.)

But why would the opposing side's lawyers care about this? They presumably want their client to win the lawsuit.

They don't have to help them, but they can't act abusively by, for example, exploiting lack of procedural knowledge.

If they deliberately took advantage of one side using ChatGPT and getting it wrong because the legal foundation of knowledge isn't there for that person, that could be a breach of their duty to the court and result in professional censure or other regulatory consequences.

The prosecution wants to win, but they'd prefer to only have to win once.

If you have to go back to trial, you've already showed your hand, and the defense (who is now competent) can adapt to it.

I'm not sure if this particular case with this particular technology made sense - but I do think we need to encourage AI penetration of the legal profession, in a way that has minimal downside risk. (For defendants and plaintiffs, not lawyers.) It would be hugely beneficial for society if access to good legal advice was made extremely cheap.

These people are being sold snake oil. Cheap snake oil, maybe, but snake oil nonetheless.

You know what an attorney can't do? Disclaim malpractice liability!

It'd be wondrous if the esteemed minds of hackernews could put their brain cycles towards actually applying common sense and other things rather than jerking off to edgy narratives about disruption while completely disregarding the relevant facts to focus on what they find politically juicy ("lawyers are scared it will succeed". It's a tautological narrative you are weaving for yourself that completely skirts past all the principles underlying the legal profession and it's development over hundreds of years.

And the reason lawyers are expensive is because bad legal advice usually costs far more in the long run.

Not really. There are more lawyers than legal jobs. A lot of lawyers are toiling for well under 100k a year. People pay 1500 dollars an hour for some lawyers and 150 an hour for others due to perceived (and actual) quality differences. Adding a bunch more non-lawyers isn't going to impact the demand for the 1500 dollars an hour lawyers.

Legal work is expensive because ANY sort of bespoke professional work is expensive. Imagine if software developers had to customize their work for each customer.

Lawyers make heavy use of automated document sifting in e.g. e-discovery.

Junior lawyers are expensive. Tech that makes them superfluous is a boon to partners. When we toss the village drunk from the bar, it isn’t because we’re scared they’ll drink all the booze.

Like the kid, the experiment is not to add knowledge to the world. Every adult knows touching a hot pan results in a burn. Just like everyone who understands how current LLMs work knows that it will fail at being a lawyer.

Instead the point of such an experiment is to train the experimenter. The kid learns not to touch pans on the stove.

In this case it’s not fair to metaphorically burn desperate legal defendants so that the leaders and investors in an LLM lawyer company learn their lesson. It’s the same reason we don’t let companies “experiment” with using random substances to treat cancer.

But the real question is, could you follow up with a question as it dictating to you live in a courtroom?

I wish you were factually correct here.

We've seen time and time again where courts are mockeries of the ideal because of the people in them and the faults they bring with them.

E.g. see https://www.supremecourt.gov/opinions/21pdf/21-418_i425.pdf and the documented proof in the dissent contradicting the claims in the ruling.

There is no hallucination on the part of the humans involved, only intellectual dishonesty.

We are unique in the universe but not important to its existence. Our automated inference technologies are accurately representing is; I read half a dozen STEM papers last year, even more during lockdown. Comma splices, grammatical errors everywhere. ChatGPT is us in the aggregate.

Even the geniuses of our species are imperfect and hallucinate being better than they are given their accomplishments relative to the laymen.

The court of law is itself an ephemeral hallucination which fails all the time; given the number of people proven innocent, it’s been suggested through analysis up to 25% or more may be incarcerated incorrectly. Drug laws are just one instance of humans hallucinating correct application of courts. YT broke a while back when it AI got hung up on circular logic in a debate about copyright (easily googled).

The burden of proof of “correctness” is on humans to prove their society is not merely a titillating hallucination.

We made computing machines before we had all the abstract semantics to describe it. Do those semantics mean anything to the production of computing machine or are they just a jargon bubble for a minority to memorize and capitalize on relative to those who have no idea wtf they’re talking about.

LOL - you gotta be kidding. In software, we strive for that - by running tests. In the law, there are no tests.

Not saying that absolute precision isn't required. I know lots of cases were an extra comma, a wrong date, or a signature from the wrong person has cost someone tens of millions of dollars. I would argue that AI-based tools could prevent such HUMAN mistakes.

Still you won’t find a useful/sober lawyer who would argue a case in front of a judge based on a made up precedent which never existed in reality.

Making a mistake as you put it, in humans, is quite different than “hallucinations” of an LLM. The practical AI tool that is good for preventing such human mistakes (precisely) doesn’t exist _yet_.

Lawyers like to wax about the interests of justice. Reality is for crap like this, it’s a revenue funnel where there’s a pecking order of people who get breaks. Some small fraction have an actual legal dispute of facts.

I’d argue that you could best serve the interests of justice by bringing the whole process online. You’d eliminate the “friends and family” discount and have a real tribunal process for cases that need it, instead of the cattle call and mockery that these courts are.

NYC actually does a decent job with this by outsourcing the whole thing to DMV and administrative law judges on a tight leash. It’s mostly justice by flowchart. That pushes bullshit down to the police.

There is a ton of people using ChatGPT for programming.... so much so that I wonder if we will have a crisis in skills as people forgo how to write code.

sysadmin circles have tons people celebrating how they will not have to learn powershell now as an example

So it can enhance skills just as much as it can atrophy them.

And I'm okay with some skills atrophying... I hate writing regular expressions, but they're so useful for some situations. It's a shame chatGPT fails so hard at them, otherwise I would be content to never use a regex again.

And judges as a group are some of the most deferential to authority people you can get.

What's so bad about this experiment isn't that they tried to use AI in law, it's that they tried to use AI without a knowledgeable human in the loop.

Even then, before the December update it was sure that there was a T in Sunday

There are other kinds of failures that will be much harder to detect: arguments that sound right but are logically flawed, lost context due to inability to read body language and tone of voice, and lack of a coherent strategy, to name a few.

All of these things could theoretically be solved individually, but each would require new systems to be added which have their own new failure modes. At our current technological level the problem is intractable, even for seemingly simple cases like this one. A defendant is better off defending themselves with their own preparation than they are relying on modern AI in the heat of the moment.

Well, _someone_ doesn't have enough schizophrenia in their life.

A non-lawyer representing themselves in a criminal case would overlook that, make a bad/wrong/misinformed argument, and go to jail.

In other fields, it'll lie to you about the thickness of steel pipe required to transport water at a certain pressure, it'll refer to programming libraries that don't exist, and it'll claim something impossible in one breath and happily explain it as fact in the next.

Give it an inane prompt to better see the difference, like "write me an essay about camels and their value in cybersecurity" which gives us the answer

"The camel is a valuable animal in the cybersecurity field due to its ability to store large amounts of data in its hump. Camels are able to cross vast distances with little water, which makes them ideal for carrying large amounts of data across networks. They are also very sturdy animals, able to withstand harsh conditions and even attacks from cyber criminals. In addition, their long eyelashes protect their eyes from sand and dust, making them perfect for working in dusty environments such as data centers."

Having internet access wouldn’t fix this.

I asked it this:

Are camels important to cyber security?

Answer:

"No, camels are not typically considered to be important for cyber security. Camels are domesticated mammals that are well-adapted to desert environments and are often used as a means of transportation or for their milk, meat, and hides. Cyber security, on the other hand, is the practice of protecting computer systems, networks, and data from unauthorized access, use, disclosure, disruption, modification, or destruction. The two are not related."

Sounds pretty intelligent to me.

You tell me which system is intelligent.

If it cannot deal with such things without being prompted in such a way that the prompter knows the answer already, how could it deal with complex legal situations with actually intelligent adversaries?

We can't even make live fact checking work with humans at the wheel. Legacy code bases are so prolific and terrible we're staring down the barrel of a second major industry crises for parsing dates past 2037, but sure LLM's are totally going to get implemented securely and updated regularly unlike all the other software in the world.

A good bar (pun intended) for the quality of the tech is if it is good enough that a licensed attorney trusts it to give legal advice with their livelihood at stake. If this product doesn’t work for DoNotPay, they can just walk away and do something else, as they are doing anyways here. If it doesn’t work for a lawyer, they’d get sued for malpractice and possibly disbarred, ruining their career. When someone trusts it to that level, have at it.

AI lawyers can help lower those barriers to entry. the court in question is fucking traffic court. please lower the barrier to entry for defense and allow normal not rich folks to get on with their lives.

the costs involved in legal are so ridiculous, our society basically encourages blind folding ourselves to most business ethical standards in hopes that our product demands a high enough margin to then pay for what ever legal fuck ups that cost too much to figure out on the front end.

You're basically trying to cut the argument both ways. Administrative courts are not criminal courts, and an AI would never be allowed near a criminal defense trial for obvious reasons.

We’re talking about traffic tickets that are usually in the hundred dollar range. If anyone is going to court rather than just paying it, it’s because $100 is a non-trivial amount of money to them.

It doesn’t have to be AI lawyers but any change to the system that reduces the total amount of work needed to be done by humans is a win.

For traffic court, perhaps, but these AI tools don’t seem guaranteed to not make this problem worse at broader scales.

These AIs will also be available to large firms, who are equally incentivized to use AI for augmenting argumentation through existing lawyers, but will also be incentivized to train their own walled garden powerful models in a way that poorer clients still would likely not have access to, and which individuals and smaller firms will not have the resources to train themselves.

These kinds of AI models could very easily serve to entrench and raise the cost of a defense by making it so you not only need a lawyer, you need a lawyer-backed-by-a-firm-with-a-LLM to be competitive at trial - making all existing problems even worse.

Even if that becomes the case, this is some how worse than what we have now?

>raise the cost of a defense

Highly doubtful

> you need a lawyer-backed-by-a-firm-with-a-LLM to be competitive at trial - making all existing problems even worse.

First, most things don't go to trial so you're completely missing the cost savings associated with avoiding trial because of these types of AI assisted scenarios.

Second, the cost associated with training models will go down. The cost of a Harvard law degree has ... never gone down.

Someone will have to take the initial risk. ChatGPT like AI may not be "the thing" but for some in this forum to be afraid of AI defense attorney is completely missing the forest through the trees.

Either that, or the client is fully aware they are defending themselves with the help of an AI that's not, and cannot at the moment be, a lawyer. As much as I want to believe AGI is just around the corner, LLMs are not individuals with human-level intelligence.

Thus, effectively, the only thing you could do is a watered down concept of the idea: A lawyer that will parrot the ChatGPT answer, but only if said answer is something they would plausibly argue themselves. They'd have to rewrite or disregard anything ChatGPT told them to say that they don't think is solid argument.

They also run a segment where a non-lawyer takes a bar exam. Recently they've also asked the bar exam question to ChatGPT as well. So far ChatGPT got it wrong every time. For example, it doesn't take into account that multiple answers can be correct, in which case you have to pick the most specific answer available. Leading to a somewhat hilarious scenario where ChatGPT picks an answer and then defends its choice in a way that seems to indicate why the answer it picked is obviously the wrong answer.

*) Of course, Alan Dershowitz is now arguing in court that the seditious horse manure he signed his name under and which is now leading to him being sanctioned or possibly disbarred, is not appropriate because he's old and didn't know what he was doing. It's Dershowitz so who knows, but I'm guessing the court is not going to take his argument seriously. In the odd chance that they do, I guess you can just say whatever you want and not be responsible for it, which... would be weird.

The first wave of ChatGPT stories were all "amazing, wonderful, humanity is basically obsolete". But now that people have had a little time, I'm seeing a ton of examples where people are realizing that ChatGPT sounds like it knows what it's talking about, but actually doesn't know anything.

We know that humans are easily fooled by glib confidence; we can all think of politicians who have succeeded that way. But it sounds like ChatGPT's real innovation is something that produces synthetic bullshit. And here I'm using Frankfurt's definition, "speech intended to persuade without regard for truth": https://en.wikipedia.org/wiki/On_Bullshit

But the hype has been insane, totally untethered from reality. I think it's the instinct to assume that something which can mimic human speech quite well must also have some mechanism for comprehending the meaning of the words it's generating, when it really doesn't.

I wonder what the wider implications are for the legal system. Will there be less qualified human lawyers in the future due to the lack of junior roles that are filled by AI? Will lawyers be allowed to use AI to find different ways of looking at issues?

The situation you describe seems to be more akin to an advanced search or information portal that people can use to guide their self-representation, or even their decision to engage lawyers/discussions with their lawyers (of course, maybe I'm misunderstanding). That stuff has basically always been allowed; nobody is threatening to prosecute Google because pro se litigants use it in their research. There are plenty of websites out there that discuss tenants' rights. There are even template tenancy agreements available online for free.

Also, what were you proposing to use as the knowledge base for your Alexa lawyer? Were you really planning on using ChatGPT or some other general purpose AI? Or would the knowledge base be carefully curated by qualified professionals? And who would create and maintain it, the state? A regulated firm? Or a startup with a name like "DoNotPay"?

I’m not sure the startup’s name is especially germane though. If anything, it seems to fit right in with human lawyers like 1-800-BEAT-DUI.

I doubt it; AI will be a force multiplier from law school on into practice. More value will mean more demand at all experience levels.

AI could be very useful for helping pro-so litigants prepare documents. I imagine with this use case as well as the oral argument use case, judges are also worried about low quality output wasting the court's time.