inference

Running Large Language Models on Your Own Hardware

Last year I spent $4,200 building a local inference rig that could run a 70B parameter model at reasonable speed. A colleague looked at my setup and asked the...