Hacker Newsnew | past | comments | ask | show | jobs | submit | elierotenberg's commentslogin

For this kind of tasks I usually create a private Firefox extension which gives me access to extended browser capabilities and the ability to lift some security-related restrictions. I run it in a sandboxed browser, much like I would do with something like Selenium or Puppeter, but I have much more options to hand-tune the automation.


It reminds me of a lightning talk I gave at ReactConf some time ago :)

Basically forward-shuffling the component tree to produce HTML that will be really hard to match using CSS Selectors usually found in adblockers and other DOM-targeting scripts.

Code: https://github.com/elierotenberg/react-armor

Video: https://www.youtube.com/watch?v=n63XXHY4A7I


It seems like monotonicity (or a subset, stronger monotinicity, still covering many practical use cases) can be statically enforced in an algrebraic type system. With some generics magic you can encode monotonicity as a type assertion on (optionally asynchronous) function calls mapping an underlying transport/RPC mechanism (REST over HTTP, RPC over Websocket, custom protocol over TCP/domain socket...).

In TypeScript for example you can strongly type your communication protocol (web requests/reponses, websockets, parent/worker ipc, etc) and get some guarantees at compile time. For example, you can guarantee that your state updates over the wire are eventually consistent.

The same goes of course for other algebraic type systems by encoding monotonicity in the form of static type assertions (Flow, Caml, Haskell...).


Well, since you just need to restrict the operations one can do with data, even a Java or C++ style type system is enough.

What is best done with another language (or a complex library on a very malleable language) is permitting non-monotonic operations with a clear boundary between them and some code style that makes them undesirable.


Interesting idea. Can you elaborate on the TypeScript comment? How would you encode something like this in TS?


Is React Resolver related to React Nexus[1]?

[1] https://github.com/elierotenberg/react-nexus


Hey Elie! Not directly, no. It looks like both of our projects originated from "real world apps" (as you say) with SEO requirements and, in my case in particular, ensuring mobile users get a response ASAP before React progressively enhances the page.

It looks like we've landed on about the same architectural solution given React's synchronous nature.

I'll have to check it out in more detail as, to be honest, I was more aware of the Nexus Flux library.


One really useful pattern is React higher-order components, eg. https://github.com/elierotenberg/react-nexus#componentgetnex...

The pattern is

  function decorate(params) {
    return (Component) => class extends React.Component {
      ... // some code using params
      render() {
        return <Component {...this.props} />
      }
    };
  }
my2c.


I haven't seen that usage for higher-order components before. This is the pattern I've been following:

    function decorate (Component) {
      let HigherOrderComponent = React.createClass({
        render () {
          return <Component {...this.props} />
        }
      })
      
      return HigherOrderComponent
    }
as explained here: https://medium.com/@dan_abramov/mixins-are-dead-long-live-hi...


They are the same I think, only one has extra parameters. At least in Python, @fn(...args) evaluates the function first and uses the result as a decorator. The guy who wrote the blog post you mention actually uses the parent's pattern in react-dnd.


Oh, you're absolutely right, 10M/s is a mistake, which I've corrected.

Few remarks though:

- it's still way below the number of actions we're talking about here (~100k/s)

- since redis is only used as a generic MQ and not as a store, it can be sharded at the app level without the pain usually associated with redis clustering

- I've deployed a similar (but less performant) design for the player of a gaming website, which is in use in production for more than a year, and works like a charm (we're talking ~5-50k users per channel on a daily basis). This is definitely a "second-system" pattern, but I try to avoid the associated pitfalls :)

I'd be genuinely interested by your feedback!


Have you ever used redis MQ at scale? I'm guessing no; the redis server is not your bottleneck. The fact that every server proc has to parse every message puts a hard ceiling on the amount of traffic you can handle. Intelligent routing is, I believe, the answer here. I've spoken with antirez on this and he agrees that at the scales you're talking about, redis MQ doesn't cut it.


Feedback?

How about just not making wild claims about byzantine fantasy designs that you never tested under any kind of load.

There has been a lot of research in messaging architectures, some of the best message brokers are free. As it happens, none of them have any resemblance to your proposed design.

RabbitMQ has been benchmarked[1] to 1 million messages/sec on 30 servers and works very well for many people.

Why not start with that?

[1] http://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-mi...


This benchmark is indeed very interesting.

I think I may have failed to express my point, though. I'm not building a message queue, as it is certainly a very hard problem that has been engineered for years by people way smarter than me :) I'm merely leveraging the goodness of their implementations (in my case redis, but RabbitMQ is also an option I've considered explicitly in my post).

The chat is a contrived example to show that even under high load, full-scale flux over the wire is a reasonable option. As for "any kind of serious load", well, maybe my example fails to meet the requirements, but unless I'm building Facebook, I think I've faced something serious enough to be able to think about my next step.


If you're building a large scale chat service you are implicitly also building a message queue.

And as for the high load you haven't actually experienced high load until you put this into production with a million users.

To make that clearer: you can design a system for any number of users, the only relevant question is how it held up in practice and as long as you haven't had a million concurrent users you just don't know (and probably it won't).


I'm not building a message queue

That may be the kernel of the problem here; you built a subset of a message queue without realising it.

RabbitMQ has a websocket plugin[1]. Just make your javascript connect directly to a RabbitMQ cluster and you have a solid, scalable foundation - almost for free.

[1] http://www.rabbitmq.com/blog/2012/05/14/introducing-rabbitmq...


I was under the impression that the design was indeed never tested, but at least the author had some real life experience of building a moderately sized chat service.

I can understand why people like you are pissed by this kind of blog post which reads a bit too much like an ad, but i think it's still good that people are trying to reinvent the wheel with completely new technologies, because sometimes it leads to surprising results.

Maybe the OP should add some warnings in the blog, saying that it's an highly experimental design that people shouldn't try to use for their own projects at the moment...


Actually this whole stack is designed with SSR in mind, see the related package react-nexus (which I will soon blog about, too!) :)


Unless you need to handle a very small number of concurrent connections, using 1 process per connection seems to be a huge overhead, although I can think of some use cases.

However, I can imagine a similar tool doing multi/demultiplexing, eg the handler process would take input text lines prepended with a connection identifier (eg. "$socketID $message") and output using a similar formatting. Pretty much like websocketd but with multiplexing and unixy/pipe-friendly (eg. you can apply grep/awk/etc before and after).

How would this fit compared to websocketd?


Indeed, there is no free lunch.

At is stands, this is only really workable for low traffic (so it doesn't eat memory) where connections do not come and go frequently (so it doesn't eat process management CPU).

Once you start doing multiplexing for the sake of making this more reasonable in terms of resource usage, the simplicity benefits kind of fall away as you move closer to a full concurrent web framework.

I guess it really depends what you're tuning for, what your use case is, and how much hardware budget you have to throw at the problem.


CGI fell out of favor for this reason, but WebSockets have a different runtime profile: instead of having to deal with 10K shortlived requests per second, WebSocket endpoints have much fewer but longer lived connections. This is why the CGI model actually works well on WebSockets.


BTW, there is a VM for Dart that is experimenting with different concurrent modes to provide an alternative to async programming: https://github.com/dart-lang/fletch

You can read its short wiki for some clues: https://github.com/dart-lang/fletch/wiki/Processes-and-Isola...

I like Fletch's idea very much. Imagine not having to worry about Async all the time.

Not sure how everything is implemented in Fletch, but I think I heard that in Fletch they are able to share 1 thread per many processes if need be. And they have been trying hard to save memory while implementing those features.

If you want to run some tests to compare with, I created some small samples using different Dart implementations and NodeJS here: https://github.com/jpedrosa/arpoador/tree/master/direct_test...

Fletch also supports a kind of Coroutine: https://github.com/dart-lang/fletch/wiki/Coroutines-and-Thre...


> Imagine not having to worry about Async all the time.

I'm nitpicking, because Fletch truly sounds very cool indeed, but when I use Elixir, Erlang, or Go, I never worry about async either. From that wiki page, I can't really see what the difference with the Erlang VM is.

(that's a good thing, the Erlang VM is awesome, and being able to write server code on an Erlang-like VM and still share code with the browser sounds like the thing that could make me adopt Dart)


About wow many connections for an average machine with 8GB ram would be deemed 'OK' with this tool?


I was just playing around with this tonight using PHP and each process was about 5Mb of RAM. I'd imagine if you wrote your server code in, say C instead, then the memory footprint would be much smaller.

There's also a limit to the number of processes allowed as well. On my OSX laptop with 16Gb RAM for example the default limit is 709 (kinda strange number???). The command

ulimit -a

will tell you the value of "max user processes" for your machine.


FYI I built simple count and greeter versions in Nim and they used around 350k. Some napkin math theorizes that's over 10k concurrents on an 4GB VPS for say a simple chat service backed by redis. I'm not sure how well websocketd will hold up at that point though...

Per process could mean easier scaling too. Like round robin connections balanced between multiple app servers backed by a beefy shared redis for example. I've never really understood how best to scale websocket services, but this could make it easier.


Thanks for that experiment!


The problem here of course is that a CGI-like approach does a fork plus execve for each request, which does not give a large benefit of sharing.

If you have a simple forking socket-based server, Linux (I assume that OS X is not any different) the amount of memory per process is much lower because it uses copy-on-write pages for forks and it's largely the same process.


That must be an OSX oddity. I just checked my Ubuntu laptop with 3GB RAM: 23967 processes, and a Debian server that I happened to be logged in to, with 0.5 GB: 127118 processes.


Of course, with 3GB you could only get 600 connections at 5MB a pop.


There is closed ticket (github.com/joewalnes/websocket) where we were battling this out, you're always welcome to join with good ideas, that would be appreciated.


It's just so easy to write a web-socket server these days in Go and other languages it might not be worth the trouble. I can only see something like this being useful in odd-ball use cases; of which none really come to mind right now but I'm sure I'd recognize it when I was faced with it.


Sounds basically like the difference between CGI and FastCGI.


Thanks for your feedback. Several other comments point in the same direction: automatic (or assisted) handling of optimistic updates. I think this is actually relatively easy, since there are clear hooks in the Nexus Flux API to implement dispatch-time and update-time behaviour, but I will definitely provide an example, or even better, an implementation of this that just works out of the box.


This is a really cool implementation! I'm wondering if it wouldn't be possible to handle optimistic updates by submitting a request to the server and then submitting an optimistic response from the client to itself. Then, when the server comes back, you hook into the already present optimistic response and just confirm or deny it based on the response from the server. That way you keep as little state on the client as possible.


Awesome! Excited to see how this goes, especially with a bigger example!


Its shouldJS. An assertion lib with a fun chainable DSL. Basically whatever.should.predicate desugars to assert(predicate(whatever)). See https://github.com/shouldjs/should.js :)


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: