> SQL streaming engines really seem to be having a moment.
I definitely agree! In the past few years, a bunch of folks (including myself) who had been working with Flink/Spark Streaming/KSQL/etc. at large companies decided that the time was right for a new generation of streaming systems and started companies to do that. For myself, seeing how much users struggled to build pipelines on Flink at Lyft inspired me to build Arroyo.
I think it's really exciting after ~5 years of relative stagnation.
> As someone who is less familiar with all the players in the space, how should I think about Arroyo vs. streaming databases like Materialize or caching tools like Readyset?
There are no hard lines (and internally all of these systems look fairly similar) but the products and use cases are pretty different.
To give my gloss:
* Readyset is a very clever cache for your OLTP database that lets you push it into more analytical territory with reasonable performance, but still focused mostly on product use cases; the stream processing system is internal and not exposed to users
* Materialize is designed to provide OLAP materialized views on top of your OLTP database by reading postgres/mysql changefeeds. It gives you up-to-date results for analytical queries without needing to replicate your postgres to snowflake and repeatedly query it.
* Arroyo is a modern Flink, designed for more traditional stream processing use cases. This includes real-time analytics, but is more focused on operational and product use cases like alerting, real-time ML, automated remediation, and streaming ETL.
Also, Arroyo is the only one of these that is fully open source (apache 2) and designed for self hosting.
I had a similar experience. I'm a great coder with good communication and leadership skills. In 2009 I co-founded a hip startup in the TV space. I had experience as a senior / architect prior to this, but this was my first time as CTO, let alone any management position.
I LOVED the first year. I was solving difficult problems, working productively, coding like crazy, learning a lot about business and startups. But it was just me and a contractor or two.
In the second year we were growing and I started to recruit. I didn't really like finding people, but I loved creating a great culture and productive environment, so it wasn't bad. I found some great people to hire and continued to be the lead dev. I started to experiment with process and getting teams to run themselves.
By year three we had 12 devs, and multiple teams. I was spending a lot of time cleaning up after unwise decisions and trying to implement process to keep us productive. Our apps and team got big. I loved being in flow and coding so much, I would drop the ball on management tasks to finish features I was shipping. I was involved in too many things: product decisions, partner deals, traveling, and our customer development research (we were floundering and making the wrong product at this point).
I hated it. Every time I procrastinated something I got more stressed and depressed. I tried changing my role to lead dev, but it was a mess and I left.
After wandering around and experimenting with different things for years, I think I've figured some things out. I'm contracting with multiple long-term clients. The skills and network I earned doing my startup have been great for finding work. I only take projects where I can have a high level of ownership and creativity (no team to deal with). I have been able to work with fun technology across multiple disciplines (in the last year: React, Haskell, Elm, iOS, Swift, Clojure, among others).
I do mess around with my own ideas sometimes, but I realized that what I really love is to create products, and I don't like management, or selling things I've created. So I'm most effective creating products with/for other people who are taking the business risk.
Not sure if there's anything in there you can use, but good luck! Figure out what you love to do, keep experimenting. I hated being a CTO too. There's no shame in figuring out it's not right for you, but it IS hard to allow yourself to go down in status. For me it was this big ego thing holding me back. Don't make that mistake :)
I definitely agree! In the past few years, a bunch of folks (including myself) who had been working with Flink/Spark Streaming/KSQL/etc. at large companies decided that the time was right for a new generation of streaming systems and started companies to do that. For myself, seeing how much users struggled to build pipelines on Flink at Lyft inspired me to build Arroyo.
I think it's really exciting after ~5 years of relative stagnation.
> As someone who is less familiar with all the players in the space, how should I think about Arroyo vs. streaming databases like Materialize or caching tools like Readyset?
There are no hard lines (and internally all of these systems look fairly similar) but the products and use cases are pretty different.
To give my gloss:
* Readyset is a very clever cache for your OLTP database that lets you push it into more analytical territory with reasonable performance, but still focused mostly on product use cases; the stream processing system is internal and not exposed to users
* Materialize is designed to provide OLAP materialized views on top of your OLTP database by reading postgres/mysql changefeeds. It gives you up-to-date results for analytical queries without needing to replicate your postgres to snowflake and repeatedly query it.
* Arroyo is a modern Flink, designed for more traditional stream processing use cases. This includes real-time analytics, but is more focused on operational and product use cases like alerting, real-time ML, automated remediation, and streaming ETL.
Also, Arroyo is the only one of these that is fully open source (apache 2) and designed for self hosting.