A GPU is a chip built to do the same math on thousands of numbers at once. That's it. The rest is just clever engineering to feed that math efficiently.
When you run a large language model, train a neural net, or render a game frame, a GPU is grinding through millions of matrix operations in parallel. But what does its silicon actually look like?
The fundamental building block. An NVIDIA H100 has 132 SMs; a consumer RTX 4090 has 128. Each SM contains a handful of things:
GPUs use SIMT — Single Instruction, Multiple Threads. A warp (group of 32 threads) all execute the same instruction at the same time, but on different data. This is why GPUs are fast on regular math like matrix multiplies and slow on branchy code — if the 32 threads take different branches, the GPU has to serialize them.
An H100 has ~3 TB/s of memory bandwidth (compared to a CPU's ~100 GB/s). The reason is simple: you can only do arithmetic as fast as you can feed it operands. Modern GPUs are constantly starving for data — they can do trillions of ops per second, but only if the numbers arrive on time.
This is why HBM (high-bandwidth memory) sits right next to the GPU die on the same package. The distance from memory to compute is measured in millimeters, not centimeters.
A GPU is a giant array of tiny math units, stacked next to a giant pool of very fast memory, orchestrated to keep the math units busy every clock cycle.