Joseph Vann died 17 years before the Civil War began.
After being evicted from his father's mansion home "Diamond Hill" in 1834, Joseph moved his large family (he had two wives) and business operations to Tennessee. He established a large plantation on the Tennessee River near the mouth of Ooltewah Creek that became the center of a settlement called Vann's Town (later the site of Harrison, Tennessee).
In 1837 prior to the main Cherokee Removal, he transported a few hundred Cherokee men, women, children, slaves and horses aboard a flotilla of flat boats to Webber's Falls on the Arkansas River in Indian Territory. There Vann constructed a replica of his lost Georgia mansion. This building was later destroyed during the American Civil War.
Some of these slaves served as crew members of Vann's steamboat, a namesake of his favorite race horse "Lucy Walker." On October 23, 1844, the steamboat Lucy Walker departed Louisville, Kentucky, bound for New Orleans. Below New Albany, the vessel blew up when one or more boilers blew up, killing the majority of the passengers and among them the owner and captain.