Although this fact was not very well known during his lifetime, Towles was African-American. His parents were both black, with his father's family being from Alabama and Arkansas, and his mother's from Tennessee. Tom's aunts, his father's sisters, were Lois Towles, a prominent concert pianist, and Dorothea Church, who was the first black model to find fame in Europe.