Elder was born one of five children in Georgia, but his family moved to New Jersey when he was a baby. He was orphaned by the age of 10 and went to live with an aunt and uncle.
During service in the Army he came under the influence of Robert Hayden of Fisk University, and after leaving the army he moved to Harlem, where he became a professional writer and actor. He also had jobs as a numbers runner (like his uncle), poker dealer, waiter and dock worker, and was active in the civil rights movement from about 1950 on.
The first play he acted in was A Raisin in the Sun in 1959. His own plays for the theater are: A Hysterical Turtle in a Rabbit's Race, Ceremonies in Dark Old Men, Charades on East Fourth Street, Seven Comes Up Seven Comes Down, Splendid Mummer and King. He also wrote for television (NYPD for ABC, an adaptation of his own Ceremonies in Dark Old Men, and McCloud) and for the screen (Sounder, Melinda, The Terrible Veil, A Woman Called Moses, Sounder Part Two and Bustin' Loose).