WebLLM: high-performance in-browser LLM inference engine

(github.com)

25 points | by saikatsg 2 hours ago

3 comments

TekMol 58 minutes ago
This seems to be the demo:
https://chat.webllm.ai/
I am getting:
```
    WebGPUNotAvailableError: WebGPU is not supported in
    your current environment, but it is necessary to
    run the WebLLM engine.
```
On both, FireFox and Chromium on Linux.
[-]
- toto007 51 minutes ago
  You can enable WebGPU support in Google Chrome by turning on hardware acceleration and activating the WebGPU flag. It Works.
refulgentis 1 hour ago
Project is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.
[-]
- skybrian 1 hour ago
  What did you switch to?
  [-]
  - seamossfet 1 hour ago
    We use the ONNX runtime for small models in the browser https://github.com/microsoft/onnxruntime
    [-]
    - sroussey 56 minutes ago
      And ONNX is what Transformers.js uses as well, at least for the moment.
  - theanonymousone 58 minutes ago
    Transformers.js supports a lot of models since v4
nnevatie 11 minutes ago
[flagged]