The H100 is NVIDIA's flagship AI accelerator — 80 billion transistors on an 814 mm² die, packed with 132 Streaming Multiprocessors, 528 fourth-generation Tensor Cores, and 80 GB of HBM3 memory feeding them. For roughly three years, it was the chip that trained nearly every large language model in production.
To picture what's actually inside an H100, don't imagine a graphics card. Imagine a specialized city.
Each block in the city is a Streaming Multiprocessor (SM) — a self-contained compute unit with its own scheduler, its own local memory, its own crew of arithmetic units. The full H100 die contains 144 of these blocks. The commercial SXM5 version enables 132 of them (NVIDIA disables a few to improve manufacturing yield — more on that below). Each SM contains 128 general-purpose CUDA cores and 4 fourth-generation Tensor Cores.
Multiply that out: ~16,900 CUDA cores and 528 Tensor Cores on a single chip. All operating in parallel, coordinated by hardware schedulers that shift work around thousands of times per second.
The GH100 die measures 814 mm² — roughly the size of a large postage stamp — and packs 80 billion transistors. Built on TSMC's 4N process (a custom 4nm-class node developed specifically for NVIDIA), this is one of the largest production GPU dies ever made. Making dies this big is genuinely hard: the bigger the die, the more likely a manufacturing defect ruins one, which is why NVIDIA disables a handful of SMs and sells slightly-imperfect chips as PCIe versions with 114 SMs enabled instead of 132.
This process of turning imperfect chips into cheaper products is called binning, and it's how the entire semiconductor industry manages the reality that no manufacturing process is perfect at this scale.
Sitting right next to the compute die are six stacks of HBM3 memory, connected by microscopic wires through a silicon interposer. Total: 80 GB per GPU (or 94 GB on the H100 NVL variant), with a total bandwidth of 3.35 TB/s. For comparison, high-end desktop DDR5 tops out around 100 GB/s — the H100 has roughly 33× more memory bandwidth than the fastest gaming PC.
This isn't luxury. It's necessity. Large language model inference is memory-bound, not compute-bound — the GPU spends most of its time waiting for weight data to load from HBM before it can do the math. The compute engines are so fast that they'd sit idle without this kind of memory throughput. This is why memory bandwidth, not raw TFLOPS, is what people who actually deploy LLMs pay attention to.
The H100's headline feature is the Transformer Engine — a hardware capability paired with software that automatically switches between FP16 and FP8 precision on the fly, depending on what each layer of a neural network actually needs.
Higher precision means more accurate math but slower and more memory-hungry. Lower precision (FP8) means faster and cheaper but only works if the numbers don't lose meaningful accuracy. The Transformer Engine watches the numerical range of each tensor as the model runs, and picks the precision that gives the best speed without losing quality. On transformer workloads (i.e., basically all modern LLMs), this delivers roughly 3-4× the throughput of the previous-generation A100.
Combined with the fourth-generation Tensor Cores, the H100 delivers up to 3,958 TFLOPS of FP8 compute. That's the number that let OpenAI, Anthropic, Google, and Meta train foundation models on timelines measured in weeks instead of years.
One H100 is often too much GPU for a single workload. NVIDIA's Multi-Instance GPU (MIG) feature lets you slice a single H100 into up to 7 fully isolated smaller GPUs — each with its own dedicated slice of Tensor Cores, memory, and cache. Each instance appears to the operating system as a separate device. Cloud providers use this to rent the same physical H100 to 7 different customers simultaneously.
The H100 comes in three form factors, and knowing which matters:
From late 2022 through 2024, the H100 was essentially the only chip you could realistically train a frontier model on at scale. Every major lab — OpenAI, Anthropic, Google DeepMind, Meta AI, Mistral, DeepSeek — built their systems around fleets of H100s. NVIDIA reportedly shipped over 2 million H100s in 2024 alone, at roughly $30,000–$40,000 each, generating the revenue surge that briefly made NVIDIA the most valuable company on Earth.
The successor, Blackwell (B200), started shipping in 2024–2025 and is now the current flagship. The H200 (same compute die, more memory) fills the gap. But the H100 remains the reference point — the chip that made the current AI era economically possible.