experts
small specialist parts inside a large AI model
Some large AI models are built as many small specialists, called experts. For each word, only a few of them are used. Because most experts sit idle at any moment, they do not all need the fastest memory, which lets such a model be split across a graphics card, ordinary memory and the processor.
Think of it as
A hospital with hundreds of specialists: each patient sees only two or three of them, so not every doctor needs to be on the ward.