With how many nodes? Does it scale linearly with nodes? (1Gb/s by itself doesn't... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		paulsutter on March 19, 2018 \| parent \| context \| favorite \| on: Breaking the trillion-rows-per-second barrier with... With how many nodes? Does it scale linearly with nodes? (1Gb/s by itself doesn't help me estimate the scale of the project), Thanks!

nikita on March 19, 2018 [–]

Data ingest scales linearly assuming the source is scalable (S3, HDFS, or Kafka). There are other things that matter: how wide is the table, what data types, etc. We achieve 1GB/s on a 16 nodes cluster for some combination of the above. What is your target?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact