John Henley Bowen built the house in 1813 as a townhome that sat on one acre of land. Bowen was a prominent lawyer and was the first person from Sumner County to serve in the United States Congress. After his death in 1822, the house was sold to William M. Blackmore in 1835. Blackmore sold it to William P. Rowles, who finally sold it to William Trousdale in 1836 for $2,500. The house is considered a late Federal style house, a period that loosely ended in 1815. The house features Flemish-style brick walls on top of a limestone foundation. The exterior brickwork is unique in its semicircular openings above the windows, which were later filled in to accommodate traditional window panes. The arched walkway and the annex were added to the house in the 1840s to serve as housing for the Trousdale sons as they became old enough for their own space. The house was equipped with electricity and running water in the late 1890s.

After 1900, the house was used as the first library in Gallatin, as well as a meeting place for Confederate veterans. It was later used as a USO dance hall during World War II as well as a community center with tennis courts, civic clubs, and social receptions. Many aspects of the house have been redone through the 1950s-1970s such as the floral wallpaper and light fixtures.