I run a large scraper farm against several large sites. They're not online shops, and we don't compete with them. But they do have hundreds of thousands of data points that we use to provide reports and analytics for our clients, who also do not compete with the sites.
I absolutely would pay for an API that provides that data. I'd be willing to pay 10x more than the cost of maintaining and running the scrapers.
But the sites being scraped have no interest in that.
Have you tried approaching those sites and asking them to provide an API, pointing out that it would be easier for both of you in the long run? Or are you just assuming they wouldn't do it.
Because right now, I sure wish that the bots - which comprise probably 2/3 of my traffic - are causing me huge headaches and I wish that the people doing it would tell me what the heck they want.
Building and maintaining the scraper is the not cost they would use to measure it internally. It’s the cost to build the API, and support it and perhaps any perverse incentive it creates where even more data flows out to competitors.
For all intents and purposes, this isn't competitive data for them. There aren't really competitors in the space anyway, the barrier to entry is ridiculous. In fact, by law, operators in the industry are required to share this particular data with each other and industry regulators. But they don't share it with outside parties in the aggregate form we need it in. Hence, the scraping.
Alternatively, in a tool that allows multiple channels, create a channel for the specific issue and invite the relevant people into it.
The problem in the essay isn't chat rooms, it's having ONE chat room for all discussions. Branch off into another chat/channel and title that channel for the specific issue being discussed.
It's the gentrification cycle, and it's been going on for decades (perhaps centuries.) It's just now getting to smaller mid-sized cities in the US.
- Some part of $City is a dirty, smelly, claustrophobic place, no one wants to live there so rents are low.
-> Low wage service workers, underserved/oppressed communities, and other "undesirables" move there because they have no where else to go, and form their own community.
-> Artists, actors, students, and others with no money move there because of low rents, and the area become "quirky" and "authentic" (ignoring, of course, the community culture that already lives there.)
-> Patrons of artists and actors like to live among artists and actors so also move to the area (or adjacent surroundings.)
-> Some of those patrons start up businesses to serve the low-rent community (the ubiquitous "coffee shop", bookstore, or gallery.)
-> The area becomes a center for art and creativity, and new amenities draw more people.
-> Middle income people start to move in, bringing demand for services and more amenities, but specifically the ones they are used to from elsewhere, not the unique "quirky" ones that made the area interesting to begin with. (Right around here is the part of the cycle where the city council and police start receiving more complaints than in the past, and usually complaints directed at the community which was there first.)
-> Developers start to take notice, buying land and increasing property values, building cheaply constructed luxury "slums" but charging sky high prices due to desirable location and access to amenities.
-> As property values go up, so does rent, forcing out first the low wage service workers, underserved/oppressed communities, and other "undesirables", and then the artists/actors/students/etc.
-> The area starts to lose what made it "quirky" and "authentic" to begin with, as the coffee shops are replaced with Starbucks, ethnic restaurants are replaced with chain restaurants, and little theaters and galleries can't afford rent anymore. It become sterile and Disney-fied.
And while all this is happening, newcomer residents are demanding that the city & police do something about all the growing homeless and addiction problems brought on by the people who can no longer afford to live there being forced out of their homes.
This doesn't protect someone (or a whole society) from bad actors who know how to exploit the assumption of positive intent. This might work for arguing with Aunt Tillie at Thanksgiving dinner, but it's a loophole actively exploited by people in power whose motives run counter to what's good, in general, for society as a whole. See: tobacco executives, climate change deniers, those who profit off anti-vaxx sentiment, televangelists, etc.
Engaging people and meeting them where they are, with honest discussion, can sometimes turn an ardent detractor of something positive into an ardent supporter of something positive.
"Ardent detractors" aren't the people I'm talking about. I'm talking about bad faith actors who understand exactly what is harmful about their position and are choosing that position anyway, because their incentives are aligned with maintaining the harmful situation. These aren't people you can engage in good faith with precisely because they will use your assumption of good faith against you.
Picking apart someone's belief through explaining their logical fallacies presupposes that they got to their belief through a logical process.
"Reasoning will never make a Man correct an ill Opinion, which by Reasoning he never acquired." -- Jonathan Swift, 1721.
To rephrase in a more modern way: You can't use reason or logic to get to someone to stop believing something they didn't use reason or logic to start believing.
I've had options at 3 companies that all went through some event.
1 - Company acquired after I left. I paid 3k for my shares when I left, made 3k net. Not bad.
2 - Company was self-funded when I left, paid about 2k for shares. When they took external investment years later, I needed to sell my shares, net 6k.
3 - Company I was still working for was acquired. I'm acquihired at a better salary & benefits, plus retention bonus after a year. Exercised my options at 4k to net 55k.
I've had a run of good luck, and, in each case, the cost to exercise my options was one I could afford to lose if things turned. If it comes up again, I'll have to evaluate where I am at that time. Each event is unique, you have to judge your specific circumstances.
The House has shown the ability to pass it, which means that if the Dems can flip the Senate (using the GA election), then the House can pass it again next session and have it go to a more favorable Senate. This was less about getting it through Mitch McConnell's Senate, and more about signaling voters: this is what we can do if it's no longer Mitch McConnell's Senate.
> The House represents the people. The Senate represents the state.
That was the original thinking, but the 17th amendment turned the Senate into just another House of Representatives, but one where each member has a longer term and a hugely outsized influence compared to the number of people they represent.
To be fair, most minority and underserved communities back in the 80s and 90s wanted the Dems to take a tough on crime stance, which is why the Dems shifted that way. For example, the crime bill that everyone loves to rag on Biden for voting for was written in cooperation with the input of Black community leaders and activists at the time. It's only in hindsight at the implementation and consequences of those policies that those communities are starting to shift away from that stance.
That's not the modern Republican platform. Both major parties want stronger central government in some instances, and weaker in others. They just differ in which instances are which.
And they're like that because most people (voters in particular) are like that. Everyone wants stronger government control and regulation over the things they personally dislike, and less control over the things they do like, and they vote for the party that matches those preferences.
Most Republican voters like and dislike the same things, while Democratic voters tend to have much larger variability (outside a core set of things.) It's why the GOP has an easier time hanging on to power than the Democrats do.
I absolutely would pay for an API that provides that data. I'd be willing to pay 10x more than the cost of maintaining and running the scrapers.
But the sites being scraped have no interest in that.