I work as a Data Engineer and in my country Azure is pretty big, and as a consequence their Data Factory service has become a common choice for enterprises. It's a GUI based ETL tool, architects prefer it since it is a managed cloud service and supposedly is easy to use.
In practice you lose all the benefits of abstraction, unit testing, proper CI/CD, etc. I haven't met an engineer that likes the service. Some projects have resorted to writing code generation tools, so that they can take config files and programmatically generate the JSON serialization of the pipelines that you're supposed to develop by clicking and dragging.
While visual representation of ETLs can be of great help understanding the data flow, engineers tend to eventually start using commands - either in the VSCode, or the Cisco iOS, or local shell. Applications subject to scripted automation and having command line tend to be well respected - a good example is AutoCAD which had a prompt from day one, which is like many years ago. This prompt still stays and is used by architects and alike.
This graph-based visual programming somehow fails to deliver on speed of development. Mouse has 2 buttons, the keyboard approx. 100. Not to mention that LLMs work on the language/code level, and are expected to stay so for a while. We dont' have universal means to express things visually. Except for the graph notation of edgex/virtices. But then there is no universal knowledge, people dont usually disambiguate between sequence diagram, bpmn and state diagram. these are all graphs, right, but not the same semantically.
I'd rather go for a standardized ETL langauge a-la-markdown, and only then get to appreciate the GUI.
>In practice you lose all the benefits of abstraction, unit testing, proper CI/CD, etc.
Why? We are pretty deep into the ecosystem.
Abstraction -> the only thing data factory does not allow you is to reference a previous activity as a variable, which makes sense if you don't want to let your customer blow up your product. Parametrize all you want.
Unit testing -> test all you want each activity, pipeline, flow, resume it from where it broke. Clone the entire thing into a test data factory, then deploy that once ready.
CI/CD -> the first step it nags you about is setting up CI/CD. If you want to get fancy, you setup a dev environment and deploy that to production after testing and sign-off.
Abstracting ETL only works when you remember or have the same people on staff that abstracted that ETL process. Data factory 'could' be visual but does not let you pull the same level of non-sense that SSIS would.
For example, we call data factory via API, the pipeline is fully abstracted, it does one thing, but it's inputs and outputs are controlled by the request.
once those custom codegen tools become established and popular, someone will have the idea that if they slap a GUI on the front it'll be much easier for non-engineers in the business to pick up and use.
I think the comparison was Snowflake vs Databricks SQL. Databricks SQL is a PaaS service just like Snowflake. Also, it uses their Photon engine, which is a proprietary engine written in C++. It is not Spark.
I'm aware that Databricks is a PaaS service, but what Databricks runs under the hood is Spark (with a few proprietary extensions). So your jobs/queries do require some tuning just like with OS Spark.
Spark has had SQL engines (SparkSQL/Hive on Spark) for a long time. Photon is just a new, faster one. Photon tasks also run on Spark executors only, so it's not independent of Spark[1]. Also, while it's proprietary now, I wouldn't be surprised if Databricks open-sources it in the future, like they did with Delta Lake.
It's definitely a long term thing. Some guys from my university founded a company around foldable containers in 2008 (https://4foldcontainers.com/). It's still not common.
I've had this experience in Germany and it turned out that the waitress gets a cut of the turnover, at least for drinks. So they have an incentive to keep the glasses full. I'm not sure if this is common in Germany.
I started reading "The Master and Margarita" this year after seeing this title pop up on HN so often. I could not finish it, I found it rather boring. I'm curious, am I the only one? What does everyone find so great about this book?
I read it earlier this year and also can't understand why it's so highly recommended. If I were living in the Soviet Union during the time it was written it would probably be the greatest book I had ever read, but as someone just looking for something to read nowadays, I mean, so, so many other better choices.
That is an incredibly limited view. How many amazing books written in "add country" during "add time" which are classics and amazing books. Sure, there is always an opportunity cost, but what does it have to do where you are living and the time it was written? Does it mean you can only read books written after the new millennia?
I also read it because it was mentioned on HN and I think it is boring too. Everything that is happening seems quite random and not very interesting. I feel either something got lost in translation or I lack some knowledge about Russian culture and don't get most references, even though I enjoyed a few other Russian classics. Otherwise I'm couldn't explain why it seems so popular here. It really doesn't help that the German translation which I read seems really theatrical and has really contrived dialog.
I also found it hard to follow and the story not particularly interesting, at least not enough of a reward for struggling through it. I finished it a few pages a night over ~3 months, which was a good activity to help me sleep.
I read the translation by Richard Pevear and Larissa Volokhonsky. Could be that it simply whooshed me, maybe I'll try again older and wiser and find myself enjoying it.
My girlfriend works as a GP. One of her colleagues was on a visit and had to call an ambulance. She was able to do so by reaching out to the police via Facebook.. (not sure how the address was communicated and what the privacy impact is of that)
Because PG's essays are intellectually uninteresting and often times mistaken (which is a nice way of putting incorrect). If the essay's didn't have his name on it, they'd be harshly critiqued when they get past around these parts. If you're looking for academic perspectives, especially around the subjects of rhetoric and debate, there is absolutely nothing worth reading in those essays.
Quite frankly I find it annoying whenever I see someone linking to them. It's junk food for your mind, with some Silicon Valley sprinkles on top.
I mean look at this gem in the linked essay:
>So although not knowing how you know things may seem part of being human, it may not be.
