MoEs have a lot of technical complexity and aren't well supported in the open so... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		fnbr 20 days ago \| parent \| context \| favorite \| on: Olmo 3: Charting a path through the model flow to ... MoEs have a lot of technical complexity and aren't well supported in the open source world. We plan to release a MoE soon(ish). I do think that MoEs are clearly the future. I think we will release more MoEs moving forward once we have the tech in place to do so efficiently. For all use cases except local usage, I think that MoEs are clearly superior to dense models.

trebligdivad 20 days ago [–]

Even local, MoE are just so much faster, and they let you pick a large/less quantized model and still get a useful speed.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact