Quick Takeaways
- Running large language models (LLMs) locally on Apple Silicon is surprisingly cost-effective; a 120B parameter model can be 5x cheaper per token than a quarter-sized model due to efficient streaming and quantization techniques.
- Model efficiency is driven by data movement: models activating fewer parameters per token (like mixture-of-experts) consume less power and generate tokens faster, regardless of total parameter count.
- In real-world traffic, the cost hierarchy remains consistent: mid-sized dense models are the most expensive, while well-quantized mixture-of-experts models are the most economical.
- The key takeaway: choose the smallest, fastest model that meets your quality needs; measuring throughput and energy consumption is essential to optimize costs, especially when running models locally.
Understanding the Costs of Running Local Large Language Models
Running a large language model (LLM) on your own hardware isn’t usually costly beyond the initial investment. For example, electricity costs are minimal, often just a few cents per thousand tokens generated. The real expense depends on how efficiently the model processes data. Recent measurements show that the largest models aren’t always the most expensive to run. Instead, models with better throughput—how quickly they generate tokens—and less power consumption tend to cost less overall. This means that smaller, optimized models can be more affordable than larger, less efficient ones. When choosing a model to run locally, it’s important to focus on these performance metrics over size alone.
How the Cost Is Measured and What Affects It
The key to understanding power costs is to measure how much electricity a model uses during operation. A simple device can record the energy consumption from the wall while the model runs. By comparing this with the model’s output speed and the electricity rate, we determine the cost per token. For example, a model that draws less power and generates tokens faster will be cheaper to operate. In tests, a 27-billion-parameter dense model used more power and produced fewer tokens per second, making it more expensive. Conversely, an optimized mixture-of-experts (MoE) model with 120 billion parameters ran faster and consumed less power, resulting in lower costs. This demonstrates that the design and functioning of the model greatly influence its operational expenses.
The Practical Implications and Which Models to Choose
For those considering running an LLM at home, the takeaway is to choose models based on efficiency and throughput, not just size. Larger models aren’t always better if they are inefficient. For example, a smaller, well-quantized MoE model can cost much less and work faster than a bigger dense model. Additionally, real-world traffic conditions—like variable requests and idle times—can affect costs. Measurements show that, even with these real-world conditions, the same trend remains: optimized models cost less per token. This highlights that the right choice depends on the workload and quality needs. Moreover, since memory limits often restrict model size, ensuring your hardware can support the model is crucial. Ultimately, measuring your own setup helps identify the most cost-effective and performant options for running large language models locally.
Continue Your Tech Journey
Explore the future of technology with our detailed insights on Artificial Intelligence.
Explore past and present digital transformations on the Internet Archive.
AITechV1
