How Far Are We from a "GPT in Our Pocket"?

WeChat version (in Chinese): 「口袋里的 GPT」，离我们还有多远？

Getting to Know GPT

On November 30, 2022, OpenAI officially released ChatGPT.

On December 5, just five days after its release, ChatGPT passed one million users.

In China, ChatGPT first drew attention in tech circles in December 2022. By February 2023, it had broken decisively into the mainstream and become the talk of the town. More recently, Humane and GPTs have brought the subject back into the spotlight. In 2023, even parents who had never cared about AI were asking: What is “GPT”?

GPT stands for Generative Pre-Trained Transformer. More specifically, the Transformer is an attention-based neural network architecture first proposed in 2017¹, while GPT refers to a family of models developed by OpenAI on top of that architecture. These models are pretrained on text at scale and can generate natural-language text. The much-discussed term “large model” now mainly refers to large-scale generative models built on the Transformer architecture, with GPT as their best-known example.

Over the past year, models beyond GPT have sprung up one after another, including Meta’s open-source LLaMA and the many models built upon it. Their overall architectures remain similar to GPT’s, but they differ in model hyperparameters, training data, and training methods. Besides “large models,” they are also commonly called large language models (LLMs) or foundation models (FMs), with each term emphasizing something slightly different. For convenience, this article refers to them collectively as large models or LLMs.

A GPT in Your Pocket

Today’s large models already perform at a level close to, or even beyond, humans on tasks such as conversation, writing, and programming. This has led many people to feel that artificial general intelligence (AGI) is one step closer, and large models have indeed shown the potential to reshape how we work. The imagination runs free: scenes once found only in science fiction suddenly seem within reach, creating a distinct sense that “the future is already here.”

One especially compelling idea is to give everyone a customized large model that serves as a personal assistant they can carry anywhere. It might live in a smart device in your pocket, a wearable such as a pair of glasses, or some even cooler form. This “GPT in your pocket” could handle documents, organize your schedule, and even interact with other people—or their intelligent assistants—on your behalf.

Reality, however, always falls short of the ideal. Beyond the limitations of today’s models and algorithms, privacy, security, and network latency are crucial concerns. A deeply personalized assistant woven into a user’s daily life would inevitably need to process a great deal of private data. Relying on cloud-hosted large-model services therefore poses serious risks to personal privacy. At the same time, the enormous volume of inference requests generated by future users would place still more pressure on network transmission, making it difficult to meet the requirements of latency-sensitive applications. Deploying large models directly on edge devices could help address these issues, but it also brings a distinct set of challenges.

Challenges in Deploying Large Models on Edge Devices

Limited Resources

The first challenge of deploying large models on edge devices is that their limited hardware resources struggle to meet the models’ demands. The remarkable capabilities of large models rest on their enormous parameter counts. For example, classic CNNs in computer vision, such as ResNet, generally have between 10M and 60M parameters, whereas GPT-3 has 175B—three to four orders of magnitude more. As a result, the memory needed merely to load model weights jumps from roughly 100 MB to hundreds of GB, to say nothing of storing intermediate data during computation.

Even a “small” model better suited to edge deployment, such as LLaMA 7B, has three orders of magnitude more parameters than lightweight CNNs such as MobileNet. At half precision, its full set of parameters alone requires at least 14 GB of memory, already far beyond the capacity of many midrange devices on the market. And beyond forward inference, fine-tuning the model locally on user data would demand still more memory and compute.

Inefficient Computation

Edge deployment of large models is primarily about inference, and the models’ inference efficiency presents another major challenge. Even if we manage to compress a model and provide matching hardware resources, thereby making deployment feasible, we still have to address the practicality problems created by latency and power consumption. After all, no one wants a digital assistant that produces text at a snail’s pace, one word every few seconds, while becoming too hot to touch (> <). A major reason large-model inference remains inefficient is its low ratio of computation to memory access, which prevents full use of the available compute resources.

This computation-to-memory-access ratio corresponds to operational intensity in the well-known Roofline Model². It is defined as the average number of computational operations performed per byte of memory accessed. On a given hardware platform, the lower a workload’s computation-to-memory-access ratio, the more memory traffic each operation requires on average. Once the hardware’s memory bandwidth is saturated, reducing the amount of computation or adding more compute capacity no longer produces any real performance gain. The system is then said to be memory-bound; the opposite case is compute-bound. Typical large-model generation workloads on edge devices tend to be memory-bound.

This low computation-to-memory-access ratio follows from the nature of edge LLM workloads. Because of constraints imposed by both the hardware and the application setting, an edge LLM usually does not handle highly concurrent requests. Most of the time, it handles only one request, meaning batch size = 1. The text being processed is also relatively short. During generation, therefore, the model’s computation consists mainly of matrix-vector multiplication and “tall-and-skinny” matrix multiplication. Its weights participate in relatively few operations each time they are loaded, so loading those weights—that is, accessing memory—becomes the system bottleneck.

Existing Approaches

Existing work addressing the two major problems described above—limited resources and inefficient computation—can be divided into two broad categories: model compression and inference optimization. In real deployments, these two families of techniques usually need to be combined for the best results.

Model Compression

The most direct way to reduce a model’s memory and compute overhead is to make the model smaller. Common compression methods for large models include quantization, pruning, distillation, and low-rank factorization. This section offers a brief introduction to each; interested readers are encouraged to explore them further.

Quantization^{3, 4, 5, 6} represents model parameters with fewer bits, thereby reducing model size. Depending on whether the model is trained during quantization, methods can be further divided into post-training quantization (PTQ) and quantization-aware training (QAT). Low-bit quantization, such as 4-bit and 3-bit quantization, remains an important research direction.

Pruning^{7, 8, 9} removes a portion of a model’s less important weights, reducing both storage and computational overhead. Pruning methods can be further divided into structured and unstructured approaches; structured pruning is generally friendlier to hardware execution.

Distillation uses an existing, effective teacher model—with many parameters and high accuracy—to guide the training of a lightweight student model—with fewer parameters and lower accuracy—so that the smaller model produces outputs close to those of the larger one. Distillation is often used together with quantization and pruning.

Low-rank factorization approximates the original weight matrix as the product of two low-rank matrices, reducing both parameter count and computation. LoRA¹⁰ (Low-Rank Adaptation) and its variants use low-rank factorization to make model fine-tuning more efficient.

Inference Optimization

Beyond model compression, there is also considerable room to optimize the computation involved in edge LLM inference. Existing work has proposed the following methods to address latency, memory use, and related problems.

The KV cache is a time-for-memory optimization widely used in current inference frameworks. Its central idea is to save the tensors that would otherwise be recomputed during each iteration of attention—namely K and V—and update them incrementally as the sequence is generated. This avoids redundant computation. As the sequence grows, however, the KV cache also becomes substantially larger, so appropriate memory-management strategies are required^{11, 12}.

Speculative sampling^{13, 14, 15} can improve throughput in small-batch settings. A lightweight draft model first generates, or “guesses,” a set of tokens, which are then evaluated and checked by the target model. Multiple tokens produced in one pass by the draft model can be verified in parallel with a single forward pass of the target model, increasing decoding throughput.

One important direction in operator optimization is optimizing attention computation. Compared with an FFN dominated by dense linear layers, attention includes matrix multiplications whose dimensions vary with sequence length, as well as operations such as Softmax. Its memory-access patterns and parallel execution therefore require specialized optimization. Representative recent work includes Flash-Decoding¹⁶ and FlashDecoding++¹⁷.

Stream loading loads only a portion of the weights into main memory—or GPU memory—at a time for inference, reducing overall memory usage. By carefully designing its GPU–CPU offloading strategy, FlexGen¹⁸ achieves high-throughput, single-GPU LLM inference. This approach is not suitable, however, when low latency matters more than high throughput, because loading LLM weights carries too much overhead.

Summary and Outlook

This article has introduced the current challenges and existing techniques for deploying large models on edge devices, covering several directions in both model compression and system optimization. My hope is that it gives interested readers a quick way to fill in the necessary background and catch up with recent progress.

Although many existing techniques have been discussed above, an important problem remains: how can we jointly optimize model compression and inference systems to further reduce the hardware requirements and power consumption of large-model inference, while improving efficiency on resource-constrained devices? If the subject interests you, I would be glad to discuss it further—and please feel free to point out anything I could have explained better (0.0).