rt.representation theory - Why the Killing form?

You might find Thomas Hawkin's book "Emergence of the Theory of Lie Groups" an interesting place to look for first-principles explanations of Lie-theory facts. He explains how Killing, Cartan, and Weyl first came up with the structure theory for semi-simple Lie algebras. (See Section 6.2, in particular, for a detailed discussion of Cartan's contributions to Killing-style structure theory---including his introduction and use of the Killing form.)

According to Hawkins, one of Killing's insights in his structure theory for a Lie algebra $mathfrak{g}$ was to consider the characteristic polynomial
$$
{rm det} (t I - {rm ad}(X)) = t^n -psi_1(X)t^{n-1} + psi_2(X)t^{n-2} + cdots + (-1)^npsi_n(X)
$$
as a function of $X$. (The start of the structure theory was to consider those $X$---regular elements---such that the eigenvalue $0$ has minimal multiplicity.) In general the coefficients $psi_i(X)$ are polynomial functions on $mathfrak{g}$ that are invariants for the adjoint action of $mathfrak{g}$ on itself.

Consider, in particular, a simple Lie algebra $mathfrak{g}$. Killing observed that the coefficient $psi_1(X)$, which is linear a linear functional on $X$, must vanish identically, since its kernel is an ideal. Cartan considered the coefficient $psi_2(X)$, which is a quadratic form on $X$. (The value $psi_2(X)$ is essentially the sum of the squares of the eigenvalues---roots---of $X$, since $psi_1(X)=0$.) The bilinear form associated to this quadratic form is the usual Killing form. The invariance of $psi_2$ under the adjoint action translates to the "associativity" property of the Killing form. Cartan observed (in essence) that for $mathfrak{g}$ simple, the kernel of the the associated bilinear form is either $0$ or $mathfrak{g}$ (by invariance + simplicity), and he managed to prove that the kernel is always $0$, starting his repair of the faults in Killing's structure theory. One can look at Hawkins' book for the details of the story, stripped of modern efficiencies.

It is tempting to think that the simpler structure theory of finite-dimensional associative algebras (where non-degeneracy of the trace form also characterizes semi-simplicity) may have inspired Cartan. It seems (again according to Hawkins) that Molien introduced this form for associative algebras (as a bilinear form---as opposed to Cartan's quadratic form) independently in the same year (1893) that Cartan published his thesis.