The Locusts
This was the home where John Jay grew up. In 1745, Peter Jay, his father, bought it -- and the rest of the 400-acre property -- from the Budd family amid fears of the smallpox outbreak in New York City (several of John's brothers were affected by this, including his brother, Peter). For decades, this was the seat of the family. However, it had been badly damaged during the Revolution (and there's archeological evidence to suggest that it might have been occupied by the British).
The home got its name "The Locusts" from James Fenimore Cooper who also featured it in his popular novel, The Spy. After the death of his brother Peter, John deeded the property to his son Peter Augustus. When he deemed the house couldn't handle a makeover, Peter Augustus was demolished it in 1837 to make way for the current mansion. However, the builders reused shutters, nails, and timbers from The Locusts in the present Jay mansion. The veranda also has the same dimensions as the original. Also, note the location of the original gardens in their present spot.