I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tools, new threats, new solutions.
That's not how complex systems work though? You say that these tools feel "symmetric" for defenders to use, but having both sides use the same tools immediately puts the defenders at a disadvantage in the "asymmetric warfare" context.
The defensive side needs everything to go right, all the time. The offensive side only needs something to go wrong once.
I'm not sure that's the fully right mental model to use. They're not searching randomly with unbounded compute nor selecting from arbitrary strategies in this example. They are both using LLMs and likely the same ones, so will likely uncover overlapping possible solutions. Avoiding that depends on exploring more of the tail of the highly correlated to possibly identical distributions.
It's a subtle difference from what you said in that it's not like everything has to go right in a sequence for the defensive side, defenders just have to hope they committed enough into searching such that the offensive side has a significantly lowered chance of finding solutions they did not. Both the attackers and defenders are attacking a target program and sampling the same distribution for attacks, it's just that the defender is also iterating on patching any found exploits until their budget is exhausted.
That really depends of the offensive class. If that is a single group with some agenda, then that's just everyone spending much resources on creating solution no permanent actor in the game want actually to escalate into, just show they have the tools and skills.
It's probably more worrying as you get script kiddies on steroids which can spawn all around with same mindset as even the dumbest significant geopolitical actor out there.
> These tools feel symmetric for defenders to use as well.
I don't think so. From a pure mathematical standpoint, you'd need better (or equal) results at avg@1 or maj@x, while the attacker needs just pass@x to succeed. That is, the red agent needs to work just once, while the blue agent needs to work all the time. Current agents are much better (20-30%) at pass@x than maj@x.
In real life that's why you sometimes see titles like "teenager hacks into multi-billion dollar company and installs crypto malware".
I do think that you're right in that we'll see improved security stance by using red v. blue agents "in a loop". But I also think that red has a mathematical advantage here.
>> These tools feel symmetric for defenders to use as well.
> I don't think so. From a pure mathematical standpoint, you'd need better (or equal) results at avg@1 or maj@x, while the attacker needs just pass@x to succeed.
Executing remote code is a choice not some sort of force of nature.
Timesharing systems are inherently not safe and way too much effort is put into claiming the stone from Sisyphus.
SaaS and complex centralized software need to go and that is way over due.
I mean, yeah you can have the joy of being right from the heights of the hill you are standing upon. But It seems like you grasp the heart of problem being discussed.
How do we deal with the floods threatening those living in the valleys and slopes?
How do bug bounties change the calculus? Assuming rational white hats who will report every bug which costs fewer LLM tokens than the bounty, on expectation.
For the calculus to change, anyone running an LLM to find bugs would have to be able to find all of the bugs that anyone else running an LLM could ever find.
Correct me if I'm wrong, but I think a better mental model would be something like: Take the union of all bugs found by all white hats, fix all of those, then check if any black hat has found sufficient unfixed bugs to construct an exploit chain?
An LLM Red Team is going to be too expensive most people; an actual infosec company will need to write the prompts, vet them, etc. But you don't need that to find exploits if you're just a human sitting at a console trying things. The hackers still have the massive advantage of 1) time, 2) cost (it will cost them less than the defenders/Red-Team-As-a-SaaS), and 3) they only have to get lucky once.
This + the fact software and hardware has been getting structurally more secure over time. New changes like language safety features, Memory Integrity Enforcement, etc will significantly raise the bar on the difficulty to find exploits.
> These tools feel symmetric for defenders to use as well.
Why? The attackers can run the defending software as well. As such they can test millions of testcases, and if one breaks through the defenses they can make it go live.
Right, that's the same situation as fuzz testing today, which is why I compared it. I feel like you're gesturing towards "Attackers only need to get lucky once, defenders need to do a good job everytime" but a lot of the times when you apply techniques like fuzz testing it doesn't take a lot of effort to get good coverage. I suspect a similar situation will play out with LLM assisted attack generation. For higher value targets based on OSS, there's projects like Google Big Sleep to bring enhanced resources.
Defenders have threat modeling on their side. With access to source code and design docs, configs, infra, actual requirements and ability to redesign / choose the architecture and dependencies for the job, etc - there's a lot that actually gives defending side an advantage.
I'm quite optimistic about AI ultimately making systems more secure and well protected, shifting the overall balance towards the defenders.
Defenders have the added complexity of operating within business constraints like CAB/change control and uptime requirements. Threat actors don’t, so they can move quick and operate at scale.
It's clear, from watching Russia fail to be completely sanctioned that this is not watertight. The question I have is: have these sanctions added a money laundering tax to doing business? How much? What is the cost of enforcing the sanctions vs the added cost and is that worth it?
I don't know if this has been explored, bit I think it's an interesting follow on to "all or nothing" watertight sanctions.
Yes, that's pretty much the main goal with sanctions (and things like export controls): no-one expects them to be impossible to work around, but they should impose some (ideally very large) extra costs and limit the scale. For some things this is more or less built into the regulations with de-minimis rules that tacitly cap the cost multiplier at 10x or so.
On a national scale sanctions aren't there to stop a country from doing things or forcing regime change. They're there to cost enough money to circumvent that it robs them of growth over time to make them into a non-threatening poor backwater over a decades long period.
This is basically what the US did to most of its "makes actual stuff" economy over the past 50yr.
When it comes to banking laws and the like it's not about being watertight. It's about holding enough water that what leaks out is small enough you can crush it with the state jackboot without enough collateral damage to really piss people off and that the cost of circumvention is high enough that you can't make "real money" outside the law at scale.
In theory it prevents failures of the allocator that would allow reading uninitialized memory, which isn't really a thing in Go.
In practice it provides a straightforward path to complying with government crypto certification requirements like FIPS 140 that were written with languages in mind where this is an issue.
If a magazine for parents of severe-peanut-allergy children ended every "may contain undisclosed peanut" recall article with a "Here's our current top 3 brands for child peanut safety: ...", would anyone refer to that as reeking of journalistic bias?
How 'bout if Consumer Reports published a "We Tested 17 kitchen garbage disposals" article, and their 1-paragraph summary of the worst-rated model said "buy one of our 3 top-rated models instead"?
(Yes, I know you're giving a "proper" response. And that very few journalists might say "buy X, Y, or Z instead" about a 900 lbs. gorilla like Cisco. Recall my "Daydream" disclaimer.)
Well the idea behind tokens is that they should be time and authZ limited. In most cases they are not so they degrade to a glorified static password.
Solutions like generating them live with a short lifetime, using solutions like oauth w/ proper scopes, biscuits that limit what they can do in detail, etc, all exist and are rarely used.
How many files do you have? At what scale did you see this being a problem?
I'm a fan of Obsidian, not affiliated with them, but my experience with basic file syncing like syncthing or git is that you should be able to easily get up into the ten's of thousands of files without an issue.
The Obsidian Sync (official) plugin isn't very clever with its syncing, it does one file at a time, seemingly new requests for each file.
I recently had to sync a new Obsidian vault from scratch, a vault with thousands of files, and it took minutes to complete because of this, even though most files are a couple of KB at most.
Easy to fix and low hanging fruit, but still an issue for us with many files in our vaults.
Same here, but the problem is bigger than that. Having to manage a ton of tiny files makes it easier to lose tiny bits of data. What happens if a file changes, gets overwritten and you don't notice? It's much easier to 'version' a single larger file than many hundreds (thousands?) of tiny files containing 4-5 lines at most.
Wandering around my dungeon, doing things, in the first person was such a fun experience as well. It felt like such a novel way to explore what I was doing and look at my creatures "in the face" so to say. In some ways it feels like a precursor to things like minecraft, where you could do some tasks as an imp in the dungeon.
reply