Kreimer discovered a Hopf algebra structure underlying the combinatorics of renormalization in perturbative quantum field theory. Later, Connes and Kreimer explored the link to non-commutative geometry via a Hopf algebra of rooted trees and described a Hopf algebra of Feynman graphs. After reviewing these developments in some detail we show in this talk how to organize the combinatorics of renormalization in terms of unipotent triangular matrix representations. A simple decomposition of such matrices is used to characterize the process of renormalization. We thereby recover a matrix (anti-)representation of the Birkhoff decomposition of Connes and Kreimer.