mixture of experts
A model built from many specialist parts, of which only a few work on each word.
A mixture-of-experts model contains many smaller specialist networks called experts. For each piece of text, a routing step picks only a few of them to do the work. The rest stay idle. This means the model holds a great deal of knowledge but uses only a fraction of its size at any moment.
Think of it as
Like a hospital with dozens of specialists on staff: each patient sees only the two or three relevant doctors, not everyone.
Ideas that use this term