The Silver Spur was built in 1855 by Frank Reusch as a stagecoach stop on the historic Watertown Plank Road and also housed a blacksmith and livery stable. Two years later, a mercantile was added and it became the first post office in Elm Grove. Known as the Elm Grove Hotel, rooms were rented upstairs and horses were stabled on the basement level. The Elm Grove Hotel remained in the Reusch family, operating as Reusch's Tavern in the fifties, until it was sold in 1963 and became the Elm Grove Inn.

The building is the second oldest restaurant in the State of Wisconsin, and the tradition continues.