Making Low-Bit LLMs Actually Fast in Parallel via Vector Table Lookup
Vec-LUT turns repetitive scalar table lookups into contiguous vector reads, accelerating parallel ternary LLM inference on x86 and ARM CPUs by up to 4.2×.
Tagged
Quantization
All writing filed under this topic.
Enhancing GPTQv2 Format Support in vLLM: Analysis and Implementation
An analysis of GPTQv2 format limitations in vLLM and the CUDA kernel changes needed for low-bit asymmetric quantization inference.