• Smokeydope@lemmy.world
    link
    fedilink
    English
    arrow-up
    9
    ·
    1 day ago

    I run a local LLM on my gaming computer thats like a decade old now with an old 1070ti 8GB VRAM card. It does a good job running mistral small 22B at 3t/s which I think is pretty good. But any tech enthusiast into LLMs look at those numbers and probably wonder how I can stand such a slow token speed. I look at their multi card data center racks with 5x 4090s and wonder how the hell they can afford it.