Making Low-Bit LLMs Actually Fast in Parallel via Vector Table Lookup
Vec-LUT turns repetitive scalar table lookups into contiguous vector reads, accelerating parallel ternary LLM inference on x86 and ARM CPUs by up to 4.2×.
Tagged
Edge AI
All writing filed under this topic.
How Far Are We from a "GPT in Our Pocket"?
A look at the practicalities of deploying large language models on edge devices.