James was born into slavery at Tuckahoe sometime around 1861. His mother was listed as Henrietta Boyd. His siblings were Mary Eliza and Sallie Boyd. It was unclear who the father was for any of these children.
As the Allen accounts do not go beyond 1859, little information is known about James at this time. But it is assumed that he lived with the rest of the Boyd family in the South Cabin, which still stands along Plantation Street.