It also reminded me how much I wish there was an actual specification for BEAM, finding out the details of the how the bytecodes work is an arduous task compared to the JVM where everything is explicitly stated. IMHO both VMs are excellent in their own right, the HotSpot JIT is incredible, but I still can't deny that I find the beam process model on concurrency more elegant than the JVM one, though in practice I've only toyed in erlang so I have no real-world grounds of comparison there. Does anyone by chance?
Wouldn't it be more correct to speak of the EVM as opposed to ERTS here? ERTS defines things like the boot protocol, distribution protocol, term serialization, port drivers, NIFs and BIFs. Though when talking about scheduling semantics, that's on the EVM level (to differentiate it from BEAM in particular).
I was working with Erlang for about 2 years in a multi-language environment, and we based our networking messaging infrastructure on top of the Erlang protocol. It's still there, but we're migrating from it.
One major thing that turned me off of the language for our particular project was that there was no way to have a large read-only data structure in RAM and sic a bunch of parallel threads to analyze or search it. It is hackable if you mash your data structure into a large byte binary, since large bins live on a shared heap, but it's a pain to deal with non-native representation for your data.
It's very obvious that Erlang is optimized for a particular usage model, and even when you're dealing with a number of functional parallel processes, it might still clash with what you're doing.
That's not the type of interface you'd want to build algorithms on which are constantly and deeply traversing data object links, if any type of performance is required.
Same thing as with ETS or SQL.
Implementing traversal-heavy algorithms across a large in-RAM data network does not require external tools, and is not helped either in performance or simplicity by decoupling direct pointer-based linkages into hash keys, and speaking to external interfaces. At least not for the work we were doing.
If you need a multi-gigabyte, read-only, in-memory data structure with support for ad-hoc querying and concurrent reads... you probably should just use a SQL database.
The content is quite interesting, but this paper could use some serious editing. There are typos throughout, and the tone is overly colloquial for an academic context.
The paper could also use an evaluation section - e.g. implementing a solution to a well known problem like the Dining Philosophers in both languages and comparing both the code and runtime characteristics.
I strongly disagree in regards to the colloquial aspect you mentioned. Academic papers have no reason to be pompous or high level and can benefit from increased visibility by presenting the findings in a more colloquial manner.
What's wrong with colloquial style? Perhaps the usual stuffy style of academic writing is a trapping to avoid, not something to imitate and perpetuate.
The report mentions Quasar (I'm its main author) and simply says, "instrumentation has many challenges that add complexity to the program instead of removing it." -- without mentioning why. I believe this is false. Quasar removes just as much complexity as Erlang does (Erlang's enforcement of immutability is orthogonal; indeed, if you use Quasar with Clojure you get that, too), but it is true that the instrumentation is not entirely transparent -- yet. This doesn't add any complexity to the program (the code is exactly the same as it would be in Erlang), but it does add some difficulty to the compilation process, as all blocking methods must be annotated. This is an annoyance, but it will probably go away in Java 9, as we are working with Oracle to make a minor change to the JVM (no, unfortunately not building continuations into the JVM just yet) that will make the bytecode instrumentation process fully automatic and completely transparent.
Fibers implemented with bytecode instrumentation also have some (small) added overhead (which is why we'd like them to be built directly into the JVM), but this makes little difference in practice: HotSpot's compiler is so good that with any added real work, that overhead is becomes negligent, and the compilation quality means that overall performance exceeds anything that can be achieved by Erlang VMs.
Also, the report says: "the JVM has a single heap and sharing state between concurrent parties is done via locks that guarantee mutual exclusion to a specific memory area. This means that the burden of guaranteeing correct access to shared variables lies on the programmer, who must guard the critical sections by locks". This is grossly inaccurate. While it is true that the JVM has a shared heap, this means that it can allow programs to share mutable state among threads -- not that it necessarily does so. The JVM leaves the concurrency model up to the language implemented on top of it (just as the hardware and OS support a shared heap, but various languages may choose not to expose that as a visible abstraction to program code). E.g. Clojure only allows shared mutable state if it enforces transactional modifications. Erlang also allows this kind of shared state via ETS; the difference is that ETS must be programmed in C, whereas on the JVM you can write the shared data structure in a JVM language. This also means that on the JVM, objects stored in such a concurrent data structures are handled by the GC, whereas in Erlang (IIRC) ETS cause some issues with GC (EDIT: in fact, ETS data is not garbage collected at all).
I believe that the JVM is a strict superset of any Erlang VM. In particular, HotSpot (the OpenJDK's JVM) is so well implemented, that the main difference -- even when running programs that behave just like Erlang program -- is a huge boost in performance, and never needing to use C to achieve either good performance or some behavior that is unsupported by Erlang semantics.
Maybe I should make the argument as to why ETS is not garbage collected. It is a pragmatic solution to an often occurring problem in software.
You have a lot of data.
You rarely change that data.
You still have to walk over it when you garbage collect.
ETS allows you to store Erlang terms into a tuple space outside the heap of any process. This means you avoid garbage collecting, and you can have 120 gigabyte of RSS, but only have to collect 400 megabytes of those. If you look at how many "mmap" implementations there are for GC'ed languages, and if you have ever reached for one, you know what I'm talking about.
Take the fastest webserver written for Java and for Erlang. Now run static requests around 2048 bytes, 10k connections, 3 req/s on each connection for 5 minutes. Measure the 99.9th percentile latency.
Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. This is not desirable in a soft-realtime system.
I'd definitely not order one as the strict superset of the other.
The TechEmpower Benchmarks measures the maximal number of requests per second you can churn through a server. They don't care for the number of connections, but simply pick the best result from trying different connection counts.
Also, they don't really measure latency correct I think, because they make the mistake of "coordinated omission" (Look it up, Gil Tene, of Zing/Azul fame is its main discoverer).
I was very careful in my wording. You need 10.000 connections, each running 3 req/s for 30.000 req/s. So we are not trying to maximize the system. We are trying to saturate it over a long period with a constant load. And you need to use wrk2 (also from Gil Tene) because otherwise the load generator and the System-under-test ends up coordinating among themselves, hiding the fact that some of the 10.000 connections are starved to get more req/s through on a subset.
The Java frameworks are very very good at handling 512 connections and throwing as many requests through those as possible. What they fail at, is 10.000 connections. What usually happens in bad load generators is that they allow the Java code to starve, say 9500 of those connections while getting all the good numbers on the last 500. wrk2 doesn't allow that to happen, and starvation is counted against the SUT.
As for the source, you would have to wait. I have the raw data, but I have not yet written the blog post. I think I've given you all the information you need to carry out the test if you are so inclined here.
I deleted my comment because I realize the benchmarks don't cover a 5 minute test. It's unproven whether the Java implementations need to in fact GC over that time. I'm skeptical though -- they probably reuse all their objects, and I think it would be interesting to prove that.
Also, the benchmarks do in fact test all the way up to 16,384 simultaneous connections.[1]
I bet if you published a test case that proves Java GC impacting the max latency over 5 minutes, that would get the attention of TechEmpower and make a case for extending the test past 15 seconds.
> It's unproven whether the Java implementations need to in fact GC over that time. I'm skeptical though -- they probably reuse all their objects
Are you saying that it's common for java web-frameworks to automatically reuse objects and that they do not cause any GC?
> Also, the benchmarks do in fact test all the way up to 16,384 simultaneous connections.
Read jlouis comment once more. It's not that they don't try many connections at once, it's that they take the best result that they get. Big difference.
EDIT:
Looking at the code for the Java Netty/JSON test shows that it does not reuse objects. The biggest takeaway from these tests is that it's hard to do performance testing...
> Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so.
Not even close (certainly not when using Quasar). Whether or not you have large GC pauses depends on how you use the heap. If you only allocate objects that live for the duration of the request (which can be enforced by your choice of JVM language) you get the same GC behavior as Erlang (only more general), and a much, much, much, better computation performance, due to HotSpot having one of the world's most advanced optimizing compilers.
The TechEmpower Web Framework Benchmarks score Java 5X faster than Erlang at a simple JSON test. (There's an even simpler plaintext test but no one was implemented it for Erlang.)
However it's worth nothing the test only runs for 15 seconds. It might be an interesting addition to run it for 10 minutes and measure 99.9% latency, as jlouis proposes, and prove one of you correct.
It also shows that Erlang is 15X faster than Java (depending on what framework you compare) and that the Java Netty/JSON test has 8X more code than the Erlang Cowboy/JSON counterpart ;)
>It also shows that Erlang is 15X faster than Java
That's nonsense. jlouis challenged us to "Take the fastest webserver written for Java and for Erlang", not the slowest ones. And that's exactly what the TechEmpower benchmarks do and they give Java a ~500% advantage.
As for codesize, if you don't like netty then the very next ranked result is servlet3-cass which delivers nearly the same performance -- again, approx 5X faster than the fastest Erlang impl -- in just a few lines of code. Surely you were aware of this when you wrote that comment.
There was a smiley in the end you know... My point was that these "tests" really shows nothing. It's a very specific scenario and if you read the comment from jlouis you would know that his scenario was very different and would yield a different result.
You're trying to dismiss the benchmark because you don't like the result. But actually the tests do show max latency with high concurrency which is the question at hand, and AFAICS the only factor not covered is a minutes-long test that might surface any GC impacts. So they are an excellent base for comparing performance.
Again if you or anyone can prove a significant GC impact serving up text or Json definitely hit up the techempower folks and make a case for extending the test length. It might help improve Erlang's standing which right now utterly sucks in comparison to Java performance (probably because of the maturity of the JVM).
HotSpot doesn't matter that much in this test. You end up spending 80% of the time in the kernel and when I tested, the Erlang code only spent a fraction of it's time in the emulator loop. In other words, the speed comes from everything but the optimizing compiler. It is from the runtimes.
Sure I'll just fork over $8,000 and get started... The beauty of BEAM is that I don't have to know or worry about projects like Zing, BEAM "just works." (Another mark against Zing just because you brought it up -- where's the source code?)
It's not that you don't have to know or worry about groundbreaking projects like Zing, but that they don't exist. The resources put into advancing the Erlang ecosystem are a tiny fraction of those put into the JVM ecosystem, and it shows.
You say BEAM "just works", but it "just works" if you can put up with its non-stellar performance or happen to play directly to its strengths. Any technology just works if you happen to need exactly what it provides. Java and the JVM, however, like the Linux kernel, are a huge project with vast resources, wide reach and very wide applicability from small embedded devices to mainframes. So, naturally, there are a lot of things to choose from, but you are guaranteed to find one that suits your need -- no matter what it is.
Too many Erlang projects start out thinking Erlang is good enough, and then find that they need to write more and more of their code in C. This just cannot happen with the JVM. Not that it's appropriate for everything (it's bad for command-line utilities and Java SE is inappropriate for constrained environments), but it's much harder to miss. You don't need to know or worry about Zing unless you need what it provides (worst-case latency of under 50us); similarly for other tens of thousands of JVM libraries and tools. All you need to know is that you can find a high-quality implementation of whatever you might need down the line.
As for Zing's source code: Zing is proprietary, but as the JVM ecosystem is so big, there's a similar open-source attempt by Red Hat and intended to go into OpenJDK called Shenandoah[1]. Its code is here: http://icedtea.classpath.org/hg/jdk9-shenandoah/hotspot/
I'm aware of Zing and I'm pretty sure it would fare way better in this kind of problem. However, I'm inclined to guess that every time you sink 1 hour of work into BEAM, there is somewhere between 5-15 hours sunk into OpenJDK and other VMs in Java space. It wins many competitions by brute force alone.
Furthermore, Erlang has some things against it in the speed department:
It is forced functional.
It is dynamically typed (A tracing JIT can somewhat alleviate this problem).
But I think the BEAM architecture is pretty sound. It has a lot of things in common with a micro kernel, architecturally. And while micro kernels are not popular currently, they see a lot of use in the embedded space, on sattelite's and so on.
Regarding the JVM as a strict superset, there are a few features I'm not sure fit that model.
Isolated garbage collection. Limiting GC to a single process/thread. It's my understanding that this is only possible with individual heaps, prohibiting shared memory. I see this as a polarizing tradeoff (with pros and cons). I don't think the JVM does this, and therefore isn't a superset.
Preemptive scheduling and soft real time. It's my understanding that Quasar bridges this gap by looking for natural points in execution to insert a pause/continue (oversimplified I'm sure). What are the guarantees? Erlang provides fairly strong guarantees about fair scheduling, making it very suitable for soft real-time systems. Would Quasar, for instance, pause a tight for loop if execution was running long? Or does it only pause on blocking operations like IO?
> It's my understanding that this is only possible with individual heaps, prohibiting shared memory.
Yes, but the only advantage is a simpler GC algorithm. HotSpot's GCs are so advanced (let alone HotSpot descendents like Zing) that they give you similar behavior even without isolation, and they include collection of shared data structures, too.
> Would Quasar, for instance, pause a tight for loop if execution was running long? Or does it only pause on blocking operations like IO?
Quasar does full preemptive scheduling but not time-sliced based preemption. This is not because it can't -- as a matter of fact, early versions of Quasar did time-sliced based preemption, but it turned out to provide no benefit whatsoever. In fact, Erlang's behavior is a limitation. The reason is that Erlang has only one type of thread -- the process, or the user-mode thread -- and therefore has to handle any type of thread, including those that are computation heavy. Thing is, work-stealing is a great scheduling mechanism for transaction-serving threads (that block very often), but not so good for computational threads. Quasar lets you choose: fibers for transactions stuff, and plain threads for long-running computations, both are abstracted into what we call a "strand".
I'm skeptical that Hotspots GC can provide the same behavior, but I'm not familiar enough to challenge that claim. My skepticism arises from most JVM benchmarks around latency (for example, a webserver) having a significant standard deviation, which I had always attributed to an overly broad GC.
I have a chip on my shoulder about limited concurrency abstractions. I've been burned too many times by Akka and thread pools and Scala Parallel collections etc.. I've always appreciated the fact that while I won't get the most performance out of BEAM, there's very little I can do wrong that will break my application.
I'm going to have to spend some time with Quasar. If your claims hold up, then Clojure plus Quasar seems like fantastic platform.
The JDK has some of the best implementations of lock-free data structures anywhere, and they are actually all implemented in Java (the JVM exposes direct access to memory fences and CAS). You really never have to drop down to C unless you want to interface with some OS-specific code.
The JVM is simply much more general-purpose. Just like your machine and OS themselves can support Erlang, so can the JVM.
In Armstrong Thesis there are explicit assertion about why thread-based JVM is a flawed design.
My point was that there is a big difference between a VM for a functional and imperative language (immutability makes it and GC a lot simpler). JVM's selling points, that it is "general" and is "already here" are strong, but in my opinion, its praise is more religion-like repetition of dogmas and marketing memes.
I would like to assert that Erlang designers have made better design decisions (to avoid threading and mutation whatsoever) so Erlang's VM is much simpler, more stable (JVM crashes often, wastes memory - memory used vs. memory served ratio, and GC behavior tend to become unpredictable for long-running process) and good-enough.
So is this a sort of Greenspun's tenth rule situation? Any sufficiently advanced distributed systems contains a buggy implementation of Erlang?
More seriously, is that what you are suggesting? By rewriting (or heavily reconfiguring) the GC, adding a scheduler, etc you end up with all the benefits of Erlang while maintaining the multi-paradigm platform of the JVM? Sounds amazing. Give it to me with out of the box sane configuration and I'm sold.
It sounds like it's not quite the case, especially around distributed systems. But perhaps that's just the intended design because clustering machines is sort of out of style in the age of RESTful microservices.
> Any sufficiently advanced distributed systems contains a buggy implementation of Erlang?
Why won't you say that about machine code? The JVM simply operates at a lower level than Erlang, hence it can be used to implement Erlang, and, it turns out that doing so might gives better results than current Erlang VMs implemented in C, because there are man-decades (or centuries) of shared functionality that's already in the JVM.
> By rewriting (or heavily reconfiguring) the GC, adding a scheduler, etc
You don't need to rewrite or heavily configure the GC (besides, which one? There are plenty of JVM GCs) -- just to use it as Erlang does, or one that is pauseless no matter how you use it (two implementations, one commercial and one open-source under development at Red Hat). Also, you don't need to add a scheduler. One of the world's best implementations of a work-stealing scheduler, also with about a decade of effort behind it by Doug Lea, is packaged with the JDK. See here[1] to get a feel for the effort behind it.
> It sounds like it's not quite the case, especially around distributed systems.
Erlang doesn't add anything not already found in high-quality, battle-tested libraries like JGroups[2] when it comes to clustering (actually, JGroups is more powerful than Erlang's clustering). The JVM ecosystem is so huge that it's very hard to come up with something (other than syntax) that doesn't already have a solid implementation out there.
I'd really like to retrofit Erjang to use Quasar rather than Kilim (Quasar is just better maintained, more battle-tested and probably faster).
I really believe that running Erlang on the JVM is a win for everyone. The Erlang community simply doesn't have a tenth of the resources that are put into OpenJDK[3]. Why not enjoy the benefits?
Greenspun's tenth rule is tongue-in-cheek and usually followed by something like "...including Erlang itself".
I get it. I see the huge benefits in a ground-up design for Erlang in the JVM. (Said design should definitely use existing GC, scheduler, etc.).
However, while what you are describing is technically the JVM, it's sort of a Ship of Theseus (with the GC replaced, with static binary manipulation by Quasar, etc.). This isn't a bad thing, but it's not what I'd normally associate with the JVM and its ecosystem.
I've got a bit of a "chip on my shoulder" regarding what I see as a pretty terrible experience with Akka. I feel strongly that preemption makes a huge difference in usability. With Akka, there are too many caveats relating to things you can and cannot do. Rather than simplifying my life, Akka gives me a whole bunch of additional ways I can break my application. While it could be said that "being a professional engineer" means solidly understanding my tools, I don't feel that is a scalable way to build development teams.
I'd love an out-of-the-box "erlang mode" switch (or some sort of launcher) that would give me these safe behaviors. Something along the lines of Percona for MySQL.
Reach out to me directly with email (in profile). This has been a fascinating discussion and I'm very interested in learning more.
But you don't need to replace the GC. HotSpot alone comes with a choice of 3 (or 4) GCs, and a fifth will join them. As of Java 9, HotSpot will support pluggable JITs (written in Java!). Quasar's bytecode manipulation is common practice among a large number of JVM tools, including one of the most popular Maven plugins, many of the most popular JVM profilers, and Spring. It isn't some exotic technique.
And don't forget -- the JVM is a spec with many implementations (including some for hard-realtime systems), although I'm talking of OpenJDK's HotSpot here.
I totally agree about preemption (and Akka, obviously). Blocking a thread (userspace or kernel) is the essential abstraction of imperative programming (or, more accurately, suspending a continuation) of that is the essence of imperative programming (even functional imperative programming, like Erlang and Clojure).
> I believe that the JVM is a strict superset of any Erlang VM.
I don't get how JVM is a strict superset of Erlang VM when it doesn't have pre-emption.
If BEAM set have preemptive scheduler as an element and JVM, IIRC, does not have such feature, then JVM is not a super set let alone a strict super set.
Quasar does preemptive scheduling, and even used to have time-slice based preemption. The latter was taken out as it proved to provide no benefit whatsoever, because processes that benefit from time-slice preemption are better off using the kernel's scheduler anyway, and the JVM gives them that option. Erlang has to do it because it only exposes a single scheduler. That's is still a strict subset of the JVM's capabilities.
Yep. Both Thread and Fiber are abstracted into a Strand. So Strand.currentStrand returns the current fiber/thread, Strand.sleep sleeps etc.. Locks and channels work with strands, too, so that both fibers and threads can synchronize on the same data, and even actors are assigned to a strand, so that an actor can run in either a fiber or a thread.
Then, you pick the strand implementation (lightweight or heavyweight) -- heavyweight for computation-heavy code like video encoding, lightweight for anything that blocks often -- and if you pick a fiber you can further select a particular scheduler, with the default being work-stealing (like in Erlang or Go), but you can plug your own (and control such stuff as CPU pinning for certain schedulers).
whereas in Erlang (IIRC) ETS cause some issues with GC
There isn't any at all for ETS:
Note that there is no automatic garbage collection for
tables. Even if there are no references to a table from
any process, it will not automatically be destroyed
unless the owner process terminates. It can be destroyed
explicitly by using delete/1. The default owner is the
process that created the table. Table ownership can be
transferred at process termination by using the heir
option or explicitly by calling give_away/3.
Of course, you'll seldom be using ETS directly, but rather through Mnesia.
I've built dozens of Erlang apps and used ETS in every one. I've used Mnesia once, and it was not a pleasant experience. Most Erlang dev teams use ETS directly. It's possible to use Mnesia successfully but I see it far less often than ETS these days.
Hi! I'm the author of the paper referenced by this post and let me start off by saying that you're right in that the paper didn't do justice to Quasar. I kindly apologies for this. I actually did want to revise that section before submitting, but as usual, the deadline came first and that section remained in my opinion one of the most "hand-wavy" ones.
Mostly the comment about "adding complexity", comes from my own biased experience of working with bytecode instrumentation. I work on the JRebel team @ZeroTurnaround and as you can imagine, we have to do quite a bit of instrumentation to get this nice reloading behaviour. Though as Murphy's law states, if something can go wrong it will and if something goes wrong after instrumentation then debugging it will not be a pleasant task. Which of course doesn't mean that it can't work nicely eventually, proven by our large number of happy customers.
Quasar's own documentation states that "If you forget to mark a method as suspendable ... you will encounter some strange errors. These will usually take the form of non-sensical ClassCastExceptions, NullPointerExceptions, or SuspendExecution being thrown". I think forgetting something is a very human thing to do and when you've got a large codebase then debugging such exceptions is what I meant by "adding complexity". But again this was only a speculation.
Regarding the other comment about shared mutable state, then of course it isn't necessarily required and I did write in the summary that "Java and the JVM provide enough tools to retrofit any concurrency model out there, but retrofitting anything won’t be the same as taking it into the initial design", which I do think still holds and I believe is one of the reasons why using libraries like Quasar won't be a trouble free experience, at least not until it's somewhat built into the JVM.
Though I won't even try to claim to know the ultimate truth and I'm always grateful to have someone correct me when I'm wrong. I appreciate your efforts in trying to make the JVM a better/more versatile platform with Quasar. I think that any such bold undertaking will only benefit the ecosystem in the long run.
Quasar's instrumentation is much less intrusive than JRebel's: no fields are added and no state changes tracked. There are only minor additions to the actual execution bytecode (again, no class-layout changes) that capture the stack. Those errors mentioned in the documentation (we should change that) are now automatically analyzed and tell you exactly where you've forgotten to annotate a method. Finally, with the changes in Java 9, instrumentation will become completely transparent, and require no manual annotation on the part of the user whatsoever.
> Java and the JVM provide enough tools to retrofit any concurrency model out there, but retrofitting anything won’t be the same as taking it into the initial design
The concurrency model requires no retrofitting. It is simply a strict superset of Erlang's. The computer and OS also support a full shared-memory concurrency model, yet it can be restricted -- not retrofitted -- to run languages like Erlang or Rust, with a more restricted model. Same goes for the JVM. It has a general-purpose shared heap, but any language may restrict its use. No retrofitting is required. Quasar doesn't impose any further restrictions (that's not its job -- simply to provide fibers), but a language like Clojure certainly does. Clojure is no less safer than Erlang. The implication is that an Erlang running on the JVM requires no C code to implement something like ETS, but the underlying JVM semantics are no more foreign to Erlang than the underlying machine semantics, and vice-versa: Erlang is no more foreign to the JVM than to the hardware. Erlang simply places restrictions on their use, and they both provide lower-level abstractions.
It reads like it was written by an undergraduate for independent study credit, or as a lit review for a future non-doctoral thesis. Certainly no result here, just a survey.
It also reminded me how much I wish there was an actual specification for BEAM, finding out the details of the how the bytecodes work is an arduous task compared to the JVM where everything is explicitly stated. IMHO both VMs are excellent in their own right, the HotSpot JIT is incredible, but I still can't deny that I find the beam process model on concurrency more elegant than the JVM one, though in practice I've only toyed in erlang so I have no real-world grounds of comparison there. Does anyone by chance?