Reputational damage from this is going to be catastrophic. Even if that’s the limit of their liability it’s hard not to see customers leaving en masse.
Ironically some /r/wallstreetbets poster put out an ill-informed “due diligence” post 11 hours ago concerning CrowdStrike being not worth $83 billion and placing puts on the stock.
Everybody took the piss out of them for the post. Now they are quite likely to become very rich.
Not sure what material in their post is ill-informed. Looks like what happened today is exactly what that poster warned of in one of their bullet points.
Yea, everyone is dunking on OP here. But they essentially said that crowdstrike's customers were all vulnerable to something like this. And we saw a similar thing play out only a few years ago with SolarWinds. It's not surprising that this happened. Ofc with making money the timing is the crucial part which is hard to predict.
Is the alternative "mass hacking"? I thought all this software did was check a box on some compliance list. And slow down everyone's work laptop by unnecessarily scanning the same files over and over again.
As someone said earlier in these comments the software is required if you want to operate with government entities. So until that requirement changes it is not going anywhere and continues to print money for the company.
But then, if what you say is true and their software is indeed mandatory in some context, they also have no incentive or motivation to care about the quality of their product, about it bringing actual value or even about it being reliable.
They may just misuse this unique position in the market and squeeze as much profit from it as possible.
The mere fact that there exists such a position in the market is, in my opinion, a problem because it creates an entity which has a guaranteed revenue stream while having no incentive to actually deliver material results.
If the government agencies insist on using this particular product then you're right. If it's a choice between many such products than there should be some competition between them.
From experiencing different AV products at various jobs, they all use kernel level code to do their thing, so any one of them can have this situation happen.
You, the admin, don't get to see what Falcon is doing before it does it.
Your security ppl. have a dashboard that might show them alerts from selected systems if they've configured it, but Crowdstrike central can send commands to agents without any approval whatsoever.
We had a general login/build host at my site that users began having terrible problems using. Configure/compile stuff was breaking all the time. We thought...corrupted source downloads, bad compiler version, faulty RAM...finally, we started running repeated test builds.
Guy from our security org then calls us. He says: "Crowdstrike thinks someone has gotten onto linux host <host>, and has been trying to setup exploits for it and other machines on the network; it's been killing off the suspicious processes but they keep coming back..."
We had to explain to our security that it was a machine where people were expected to be building software, and that perhaps they could explain this to CS.
"No problem; they'll put in an exception for that particular use. Just let us know if you might running anything else unusual that might trigger CS."
TL;DR-please submit a formal whitelist request for every single executable on your linux box so that our corporate-mandate spyware doesn't break everyone's workflow with no warning.
Extremely unlikely. This isn't the first blowup Crowdstrike has had; though it's the worst (IIRC), Crowdstrike is "too big to fail" with tons of enterprise customers who have insane switching costs, even after this nonsense.
Unfortunately for all of us, Crowdstrike will be around for awhile.
Businesses would be crazy to continue with Crowdstrike after this. It's going to cause billions in losses to a huge number of companies. If I was a risk assessment officer at a large company I'd be speed dialling every alternative right now.
A friend of mine who used to work for Crowdstrike tells me they're a hot mess internally and it's amazing they haven't had worse problems than this already.
That sounds like any other companies I have ever worked for: looks great from the outside but a hot mess on the inside.
I have never worked for a company where everything is smooth sailing.
What I noticed is that the smaller the company, the less hot mess they are but at the same time they're also struggling to pay the bill because they don't innovate fast.
As at 4am NY time CRWD has lost $10Bn (~13%) in marketcap. Of course they've tested, but just not enough for this issue (as is often the case).
This is probably several seemingly non consequential issues coming together.
I'm not sure why though, when the system is this important that even successfully tested updates aren't rolled out piecemeal though (or perhaps it has and we're only seeing the result of partial failures around the world)
Testing is never enough. In fact, it won't catch 99% of issues by the virtue of them often testing happy paths only, or that they test what humans can think of, and by no means they are exhaustive.
A robust canarying mechanism is the only way you can limit the blast radius.
Set up A/B testing infra at the binary level so you can ship updates selectively and compare their metrics.
Been doing this for more than 10 years now, it's the ONLY way.
I'm not sure that justifies potentially bricking the devices of hundreds(?) of your clients by shipping untested updates to them. Of course it depends... and would require deeper financial analysis.
> They won't be able to test exhaustively every failure mode that could lead to such issues.
That might be acceptable. My point is that if you are incapable of having even absolutely basic automated tests (that would take a few minutes at most) for extremely impactful software like this starting with something more complex seems like a waste of time (clearly the company is run by incompetent people so they'd just mess it up)
And when it’s more costly for customers to walk back the mistake of adopting your service.
Yeah, I get the impression a lot of SaaS companies operate on this model these days. We just signed with a relatively unknown CI platform, because they were available for support during our evaluation. I wonder how available they’ll be when we have a contract in place…
I'd hope that it's based on calendar months. Wouldn't invoices that span multiple years make things a lot more complicated in terms of accounting in general?
As far as UI/UX is concerned their tvOS app is in a really rough shape. The web UI has a fairly reasonable program guide (edit: for Live TV), while the app doesn't. Also navigation in the app doesn't feel polished at all unfortunately.
I think the Apps are inconsistent, some of the newer Apple ones are great, the computer ones are first tier, but some of the multiple TV OSes are lagging.
I wish there was an open hardware option that was a target for devs to support for projects like Jellyfin but also funded them when you bought it.
The ESP32 does have internal pull-up and pull-down resistors on certain pins. I believe GPIO14 on this board maps to GPIO22 on the ESP32 chip (mismatched IO pin labels is bit of a pet peeve for me!) which does have pull-up and pull-down resistors that can be enabled.
If it were me, I might tie the doorbell switch button from GPIO0 to ground. This pin has a pull-up resistor as it needs to be high to boot normally. So this has the added benefit that you can hold the button in when you first power the device to put it in flash mode, which also eliminates their "Make sure to get one with a “flash/download/io0” button" warning.
If you have an "old fashioned" 24VAC line, you could add a rectifier and buck/step-down power module to drop that to 5VDC. Or just unhook the 24VAC transformer and install a 5VDC one.
It would have been neat to see how much the RGB ring lighting up white helps in no/low light cases. All in all, nice writeup!
Important work. It is a typical quality control architecture in the modern world to have an independent party evaluate whether one can perform an important task or not. Typical examples would be SE vs PE licensing on structural engineering or the various bar exams.
Didder is fantastic. I'm using it to convert images to grayscale for use with Waveshare e-ink displays. I've found no other tool that both lets you specify a custom palette (without which the resulting image looks pretty bad) and is reasonably quick.
For future reference, this works beautifully with their 9.7" display: didder --palette '000000 111111 222222 333333 444444 555555 666666 777777 888888 999999 aaaaaa bbbbbb cccccc dddddd eeeeee ffffff' -g -i - -o - edm --serpentine FloydSteinberg
Image Alchemy [1]
Unfortunately not longer available. But one of the greatest image conversion software I know (including most of the here described algorithm for dithering and some more). And this feature set is from at least 25-30 years ago.
I'm always getting a bit nostalgia when I see these dithered images and remember the time then with the fascination for the opposite direction: 'true color... wow that would be great' ;)
