Enhancing GPTQv2 Format Support in vLLM: Analysis and Implementation
An analysis of GPTQv2 format limitations in vLLM and the CUDA kernel changes needed for low-bit asymmetric quantization inference.
Tagged
LLM
All writing filed under this topic.
Tagged
All writing filed under this topic.
An analysis of GPTQv2 format limitations in vLLM and the CUDA kernel changes needed for low-bit asymmetric quantization inference.