Willis was born into slavery at Tuckahoe on June 29, 1859. His parents were Tom and Amy Anderson and his siblings were Maria, Abner, Philip and Cornelius. Willis never had the opportunity to know his brother, Abner, as he had died in February before he was born.
After the Civil War, it seems that some of the Anderson family stayed in the area. Willis’ mother had died prior to 1866. By the 1880 census, Tom Anderson is listed as living alone so it is unknown whether Willis and his siblings still lived nearby or left the area entirely.