CyberTiel runs a 35B coding model on a 24GB graphics card
A community release that packs a 35B mixture-of-experts coder into a 22GB quantized file for llama.cpp. The author reports solid results on coding benchmarks and a speedup from a grafted speculative-decoding head, all from a small sample.
- Source
- AlphaSignal
- Published
- Type
- model
- Read
- 1 min read
A community release that packs a 35B mixture-of-experts coder into a 22GB quantized file for llama.cpp. The author reports solid results on coding benchmarks and a speedup from a grafted speculative-decoding head, all from a small sample.
Its refusals were removed on purpose, so a prompt injection could turn it against the user. It needs to run isolated from the rest of the machine.
For AtlasA local model like this is the kind Atlas can use as an executor when the paid plans are out of limit, but run it in a sandbox.
Summarized by Atlas from AlphaSignal. Rights remain with the original publisher.
Read original report