**** POLYGON SEMINAR**** Permutation groups and related algebras have proved to be powerful tools for understanding the counting and correlators of gauge invariant operators in 1-Matrix and multi-matrix models. Mathematical structures such as Belyi maps underlying the mixing of trace structures have been uncovered and finite N effects have been encoded using Young diagram data. These results have found applications in studies of BPS, near-BPS and non-BPS operators in N=4 SYM and quiver gauge theories. I will review some of this work and describe some open problems.