Hacker Newsnew | past | comments | ask | show | jobs | submit | berkle4455's commentslogin

Don’t use Kafka (or PostgreSQL ) as a job queue. Can we just stop forcing this?

Kafka is an amazing event streaming platform.


Wait til you find out about chatGPT


Dissolve the TSA. Biggest fucking waste of tax payer money and aggregate loss of GDP ever invented.


There was a thread here that touched on this recently. The number of TSA defenders here surprised me.

https://news.ycombinator.com/item?id=37624112


I mean what do you replace it with, return to the prior status quo of private security? I don't think that's even a good idea, and I dislike the TSA as much as everyone else - nor do I think having Local LE do it makes sense, because then you end up with 900 policies.


Do what all the other countries do, only the US manages to make the whole thing incredibly intolerable, with the ruddest staff they could hire.


As far as I know, the US was among the last countries to adopt what other countries did, which was have some sort of federal police perform airport security.

Also, I fly pretty regularly - about one round trip a month - I find the TSA folks pleasant to deal with, and generally courteous - but I also know I'm an actor in a play, and I need to go thru the motions like everyone else.

Functionally, TSA is about 50% theatre, its meant to make Bob and Eileen from Cedar Rapids who fly twice a decade feel safe.


Based on my experience travelling abroad, the security measures are pretty similar. Some of it could be due to international travel, but I don't think that explains departure security.

In the US, there's basically nothing different. I did fly NL to Germany, and back, and it was the same (obviously still international, but within EU). It seems not that many people fly domestic in NL, and there's little reason for relaxed security (who cares if they got stabbed in NL instead of Germany?)

So I don't understand what would be cut. If the freaking dutch are doing most of the same B.S., are we really out of order? Sure we can't have liquids, but isn't this only mildly bizarre and annoying?


I hate the TSA. But one thing to consider is it's effectively a jobs program, just like the US military. For better or worse these are people to whom we'd likely be funneling money in one fashion or another, at least the security theater makes it more palatable to the masses :shrug:


> it's effectively a jobs program

Then build roads, infrastructure, etc. Don't put a "jobs program" in charge of banning citizens from flying ever again, violating citizen's rights and more.


> in charge of banning citizens from flying ever again, violating citizen's rights and more.

You should speak the language people would understand.

"in charge of banning citizens from ever seeing their beloved on the other side of the country, violating citizen's assess and genitals and more"

That people understand


For better or worse these are people to whom we'd likely be funneling money in one fashion or another

Not a fan of TSA, but the people working there seem like normal, employable people. If we got rid of TSA, ~20% could probably stay working security at the Airports, and the other 80% could move on to actual productive jobs, and instead of us funneling money to them they'd be generating money.


Is that true though? Are there really that many jobs hanging around, in the areas in which these people live, paying an equal or better salary, for which these people are qualified? These aren't people who are going to stop working as a person standing around directing traffic and start coding distributed systems.

EDIT: I'm assuming they're paid more than min wage, I'll admit I don't know.


I'm lazy[0], so the first links:

    TSA (Transportation Security Administration) salaries: How much does TSA (Transportation Security Administration) pay?

    Popular Roles
    Transportation Officer
    $43,454 per year
    Transportation Specialist
    $76,443 per year
    Marshal
    $67,643 per year
    
    https://www.indeed.com/cmp/TSA-(transportation-Security-Administration)/salaries
    
Also see https://www.tsacareer.com/tsa-pay-scale/

[0] well, looks I'm a bit less lazier than you, heh


Only thing I know of is that the people who couldn't qualify to be prison guards would go work at the TSA.


Some of them are nice, normal, employable people. Some of them are angry fascists. Some of them are thieves. Some of them are rapists.


Fix potholes? Pick up litter? Clean up subway stations and bus stops? Maintain trails? Education and youth programs? Learning new skills? Building benches, lean-tos, monuments, and other simple structures? Creating new national parks to cover the recent extreme demand? Planting flowers on the side of highways? Spelling out words for aerial photos?

There are many better ways to productively create low skill jobs than hassling and molesting travelers while on a petty power trip.


Stack blocks or some other completely pointless activity: TSA isn't just worthless, by massively slowing down air travel and making it less enjoyable it provides negative value. We would be better off if most of the TSA staff stood out on the tarmac shuffling around pallets all day long for no particular purpose.


I agree with you.

However for large swaths of the country's population people would interpret all of the above as "socialism" and they'd complain. In contrast "defend our people from evil terrorists", either via TSA or military, does not get the same scrutiny.


Sure, that's called culture. It feeds into itself, and is essentially what we're critiquing. The proto-fascist culture where the only uses of government are domination and control shouldn't be encouraged.


Again, 100% agreed. But for someone wondering "why do we pay these people?", this is the answer.


You're trying to make this weird middleground between normative and positive statements.

In the positive sense, obviously a whole lot of suckers bought into nonsense about terrorists hiding under their beds. We know this is the unfortunate realpolitik dynamic. We don't need to infer unspoken motivations about jobs programs, because the straightforward narrative of security theater blanket is still carrying the day.

In the normative sense, it's nonsensical to treat the broken culture of fear/domination as inevitable. That culture is wrong, regardless of how overwhelming it is.


Sure, some of those sound like socialism, but not fixing roads, and I don't think cleaning up transit stations either (but of course, having transit stations could be considered socialism, and making therefore making the stations better could be too).

Hell, making roads nice and making nice parks sounds a lot like 'make america great again' to me. But maybe I'm not conservative enough to be the sounding board.


Again, I 100% agree with this too. And yet I guarantee that it'd be excoriated on Fox News.


> I hate the TSA. But one thing to consider is it’s effectively a jobs program

How was transferring the security function from private industry to the federal government a jobs program? Its an airline-industry subsidy (and it was lobbied for by the industry for exactly that reason in the wake of 9/11), both by making the government responsible for the direct cost of the function and by the government absorbing (or, via sovereign immunity, eliminating) legal liability for both abuses in carrying the function out and failures resulting in harms to travelers and third parties.


When people say "dissolve the TSA", they don't mean: "I want all the security theatre, it's just that I want private industry to do it". They mean: "eliminate the security theatre, thus vastly decreasing the number of people employed to perform security theatre operations". It's a jobs program because all of theatre needs a huge number of bodies to perform the play.


Yes, exactly this. Further, its a jobs program because it allows for an easy vector to add *more* jobs by adding *more* theater and by and large the citizens are willing to accept it. Because who wants to argue against safety?


Arguably the TSA is actually reducing security by breaking locks on gun cases and such.


The existing security function was transferred, but the scope/intensity increased.

What used to be two, maybe three people running a metal detector and an x-ray machine for parcels, is now probably ten people running a lot more equipment, a lot slower.

I'm not convinced we need the intensity of inspection that we have, especially when passenger education in real time was enough to foil the targeting of the 4th plane, and locked cockpit doors and policy changes should foil hijackings without the participation of the pilots. It's a huge cost for a small benefit.


Airport security used to be two people operating a single metal detector.

Now it's dozens of people operating xrays, body scanners, metal detectors, searches, etc.


>How was transferring the security function from private industry to the federal government a jobs program?

By making it much less efficient.


They could clean the streets instead, that'd be a better use of money and everyone's time.


As I've noted elsewhere in the thread, somehow "clean the streets" is considered to be "evil socialism" to half the country but "stand around looking bored by the security line" is not. And that's why it is the way it is.


But the "work" they do at the TSA wastes immense amounts of time and property, as well as causing untold amounts of stress on people who do actually productive things. Putting them to work planting trees or some other useful activity would be far better for everyone.


I would much rather the government paid people money to dig ditches and fill them back in, or even just give them the money for free, rather than paying people to actively make others' lives worse.


Give them jobs fixing roads then.


Also the job still needs to be done, be it TSA or something else.


We could have jobs programs to green our economy, planet and lives, instead of glorified daycare for adults.


& those are the people we force the rest of the populace to interact with? Perhaps not the best idea.


well then we should fund other jobs programs too like gas station attendants & lift button pushers.


Thinking back on the negative experiences I've had with gas station attendants, suddenly the awful stories about the TSA makes a lot more sense.


well any job where you have some degree of security from being fired and in general somebody being forced to pay for outside of market forces will have negative experiences. so thats a separate problem in itself.


I'm all for creating jobs for people who need them, but can they at least do that where they're not as bothersome and won't go through my luggage because I packed a gel sleep mask? every single time :/


When you realize Apache is a hospice organization


It's mind boggling how crypto people still trust hot wallets that run as fucking browser extensions.


The reality is that relying on a cold storage wallet for security is just too much overhead when it's competing against a biometric scanner to authorisar a transaction that can be reversed with a phone call.


I think with any database the solution is simply backups no? backups that preferably aren’t tied to the hosted solution at all.


Actually not quite. Backups (assuming you are doing single node) still needs you to decide on your SLOs - RTO (how long it takes to restore) and RPO (how much data lose you can suffer) numbers. On the instant snazy end you have streaming backups and recovery and then on the other extreme you have backup once in N hours/days and restore taking how ever long it takes to restore (so you have customer outages you need to negotiate.

Now let us involve multi node, (both replication and partitioning of shards). As shards go and up and down ensuring data is in sync etc is a hard consistency problem and needs man years of operational excellence and bug fixing.

So when people think databases - they think of the cool stuff - the database engine that does relational algebra and handles SQL queries. That is (IMO) only 1% of a practical, performant, reliable database (offering).


Maybe if you are gigantic, but there is a long tail of people with <1TB database needs that don’t really need shards and can be well served by a fail over cluster with a master and one or two replicas that can become masters.

These days you don’t really need shards until you hit many terabytes or even more depending on your read and especially write load. NVMe storage is really fast and lots of RAM for caching has become cheap.


So my point was around all things a managed for gives you (eg sharding and replication). Even by the time I had to setup streaming replication and have to worry about wal drifts it is easier to pay a managed provider no?


Also what about the customer that deleted an important thing 6 weeks ago and absolutely needs it recovered? BTW, it's just one tentant in that DB, the other shouldn't be recovered, naturally.


In that case, it’d probably be best to just handle deletions at the application layer (e.g., setting a “deleted_at” timestamp field with scheduled permanent deletions later).

And in terms of data compliance, it’s very important to make sure permanent deletions propagate through your backup systems within a reasonable amount of time - Google Cloud[1], for example, is ~180 days.

[1] https://services.google.com/fh/files/misc/gcp_data_deletion_...


Backups? Do you want to share your idea about how you'd do backups? Especially to a distributed database?

Here are some of the questions you'll have to answer and some options you will have to consider before you go there:

Let's start with the heavy stuff: consistency groups. I.e. groups of bulk storage that underlines your entire infrastructure that ensure that your application and database(s) all recover to the shared state once they crash. To better explain this concept, consider this: you have an application that works with two databases, let's say a document database to store documents uploaded by users (which are later parsed by the application and transformed into records in a relational database). Now, each database provides best consistency guarantees... but they still can fail independently and subsequently recover to different state, where, for example, the document database can be ahead of the relational one (and lose some data). Similar problems face sharded databases.

How geographically far are you going to send your backups? You see, the closer to the working server they are, the higher is the chance you'll lose them together. But, here's the problem: the further away the backups are, the lower is your ability to keep the backup up-to-date with the database, and, subsequently, more data to lose.

Well, backups inherently lose data (for the time between the last backup and the time of the crash). So, if you don't want to lose data at all, you probably want replication rather than backups. And you probably want online replication (but then the distance between the replicas is even more important than in the case with backups).

Also, backups are huge. If you want to ship them outside of the facilities of the storage vendor... that's going to be expensive.

Another point to consider: databases provide consistency guarantees, but does your database provide consistency guarantees you want? Is every relation encoded by using foreign keys, or does the application have some knowledge of how to interpret pieces of data and stitch them together into relationships unknown to your database? Are you sure that every operation that requires atomicity is implemented in a database rather than application (which doesn't enforce atomicity)? What if you stick a backup (recovery point) in a precise moment when your application was doing something that was meant to be atomic, but the application author didn't know how to express in SQL (because in their fear of technology they chose to use Hybernate or SQLAlchemy etc.)? And if you do so, it spoils your backup...


I actually do not understand the point here. And maybe you are not very familiar with the concept of transactions. Backups can only account for committed transactions.

However, we are talking about Postgres, here, not a generic database. PostgreSQL natively provides continuous backup, streaming replication, including synchronous (controlled at transaction level), cascading, and logical. You can easily implement with Postgres, even in Kubernetes with CloudNativePG, architectures with RPO=0 (yes, zero data loss) and low RTO in the same Kubernetes cluster (normally a region), and RPO <= 5 minutes with low RTO across regions. Out of the box, with CloudNativePG, through replica clusters.

We are also now launching native declarative support for Kubernetes Volume Snapshot API in CloudNativePG with the possibility to use incremental/differential backup and recovery to reduce RTO in case of very large databases recovery (like ... dozens of seconds to restore 500GB databases).

So maybe it is time to reconsider some assumptions.


> And maybe you are not very familiar with the concept of transactions.

Hahaha. Really? Try being more subtle maybe? Or maybe try reading what you replied to?


k8s cluster and simple are rarely found together.


Do not host anything of importance on Hetzner. They'll gladly blackhole your server with zero recourse.


They also do it when their system incorrectly flags traffic. They're the Paypal of web hosting. Stay far, far away.


> Companies will very successfully have their contracts enforced

Hetzner turned off all access to my paid server due to a false-positive on their netscan/DDOS (literally it was tailscaled doing a netcheck) protection and equally incompetent technical support staff.

Can I sue them for breach of contract and subsequent damages? I moved all my hosting off Hetzner as a result, but I'm still very disappointed in their actions.


While I didn't read the contract, I'm halfway sure that it says that they can terminate the contract for any reason.


They didn't terminate my contract though, they just nullrouted my server. They even had the audacity at the end of the month to send me a bill.


Ah, that's shitty. I guess in theory you could demand a refund or file a chargeback for an amount based on the services they didn't contractually provide. As you ceased doing business with them I don't think there would have been any drawback to this. IANAL


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: