A better analogy would be "remotely swing a hammer as a service". You can't build something like that and act shocked when a significant fraction of your users use it to harm people instead of driving nails, and you certainly shoulder a large fraction of the responsibility.
I really don't get a lot of this criticism. For example, who is using iceberg with hundreds of concurrent committers, especially at the scale mentioned in the article (10k rows per second)? Using iceberg or any table format over object storage would be insane in that case. But for your typical spark application, you have one main writer (the spark driver) appending or merging a large number of records in > 1 minute microbatches and maybe a handful of maintenance jobs for compaction and retention; Iceberg's concurrency system works fine there.
If you have any use case like one the author describes, maybe use an in-memory cloud database with tiered storage or a plain RDBMS. Iceberg (and similar formats) work great for the use cases for which they're designed.
>who is using iceberg with hundreds of concurrent committers, especially at the scale mentioned in the article (10k rows per second)? Using iceberg or any table format over object storage would be insane in that case
You can achieve 100M database inserts per second with D4M and Accumulo more than a decade ago back in 2014, and object storage is not necessary for that exercise.
Someone need to come up with lakehouse systems based on D4M, it's a long overdue.
D4M is also based on sound mathematics not unlike the venerable SQL [2].
[1] Achieving 100M database inserts per second using Apache Accumulo and D4M (2017 - 46 comments):
> But for your typical spark application, you have one main writer (the spark driver) appending or merging a large number of records...
The multi-writer architecture can't be proven scalable because a single writer doesn't cause it to fall over.
I have caused issues by using 500 concurrent writers on embarrassingly parallel workloads. I have watched people choose sharding schemes to accommodate Iceberg's metadata throughput NOT the natural/logical sharding of the underlying data.
Last I half-knew (so check me), Spark may have done some funky stuff to workaround the Iceberg shortcomings. That is useless if you're not using Spark. If scalability of the architecture requires a funky client in one language and a cooperative backend, we might as well be sticking HDF5 on Lustre. HDF5 on Lustre never fell over for me in the 1000+ embarrassingly parallel concurrent writer use case (massive HPC turbulence restart files with 32K concurrent writers per https://ieeexplore.ieee.org/abstract/document/6799149 )
> Riot and Blizzard both have large flagship franchises that they practically decorate their offices around, statues and all. When I was talking to recruiters in the past, Riot stood out as one of those "don't bother applying if you aren't absolutely shitting passion for our games out of every hole" as culture fit.
I interviewed at Riot when I was trying to move back to Saint Louis. I could not muster up very much enthusiasm for League, and you could just feel the room shift into "just get through this interview ASAP" mode.
It's hard to know how to feel about this complex situation. On one hand:
- The Chinese government clearly has its tendrils in TikTok and uses it to push its oppressive censorship policies around the world
- The security implications of giving China a backdoor into a large percentage of American phones is too concerning to ignore
On the other hand:
- Explicitly targeting specific companies feels wrong. If security and censorship are real concerns here, Congress should be passing laws that apply to all companies, not just TikTok
- Even though there are sound reasons to restrict TikTok, this is still a transparently protectionist policy by the Trump admin and fits right into his nationalistic, totalitarian, xenophobic playbook. A broken clock is still right twice a day, I suppose
- Requiring a sale to an American company is the mirror image of China's forced technology transfers for foreign companies. Two wrongs don't make a right
I think the best scenario is for Congress to preempt Trump with laws that protect American security and free speech interests without executive action. That's easier said than done, though.
At a startup, there's a good chance Fred's time would be better spent knocking out feature work (you only have so much runway, after all). But at a mature tech company, assuming he's good at his job, Fred is likely providing a ton of value by
- Keeping his coworkers sane (barely functional codebases that are difficult to work with have a way of killing morale)
- Having an intimate knowledge of the code base that can inform future feature work
- Paying down tech debt that's generally uninteresting to more feature-oriented coworkers
Having a whole team of Freds is probably not a good idea, but having one in a senior engineer or tech lead role is really useful.
I can't imagine forced remote for all employees would ever become the norm. It feels like likely that companies would move from dedicated offices to leasing co-working spaces for those that want it.
Do they still have to label irradiated food in the EU? IIRC many people avoid it because radiation is scary sounding. I have to imagine that chemically treated meat would be an even tougher sell, regardless of how benign it might actually be.
For a couple of years, the only thing that worked for me was mega-huge doses of PPI's. After several pH studies, endoscopies, and other tests confirmed that I don't have abnormal reflux, food allergies, gluten sensitivity, or anything else, my doctor(s) and I found after years of trial-and-error that a combination of H2 blockers and a low-dose of antidepressants mostly controls my symptoms, which are probably caused by an overly-sensitive gastric nervous system.
FWIW, one of my GI doctors recommended trying alternatives to PPI's, while the other was highly skeptical of these data mining studies and recommended staying the course unless a causal link is established.
It’s so frustrating to often get conflicting advice like this from medical professionals. I understand why it happens but it makes it difficult to make decisions as a patient.
