TurboQuant model weight compression support added to Llamacpp

3 comments

• kgeist 1 hour ago
  I think the headline is misleading. It's some random fork of llama.cpp, I can't find evidence that TurboQuant was actually added to llama.cpp proper.

  The only legit PR I can find is this [0] and it's still open.

  There's currently a lot of rejected vibe-coded PRs: [1] (violation of AI policy).

  The OP's PR says it was generated with Claude Code so it has a very low chance of getting merged upstream.

  [0] https://github.com/ggml-org/llama.cpp/pull/21089

  [1] https://github.com/ggml-org/llama.cpp/pulls?q=Turboquant+is%...
• jsilence 1 hour ago
  Great news! Expecting this to get implemented in all the major inference runners pretty fast. See also: https://news.ycombinator.com/item?id=47637422
• lastdong 2 hours ago
  Cuda support added. Also see https://news.ycombinator.com/item?id=47562135#47635952