In 1829 the Richmond County Poor Farm was established as a home for lower class citizens, but when the Staten Island became a borough of New York City (and therefore New York's responsibility) in 1898, the state took over the property and renamed it to the New York City Farm Colony in 1902. Dutch Colonial buildings were mostly built between 1904 and 1916. The institution eventually merged with Sea View Hospital, located across the street and had been built solely for the purpose of controlling tuburculosis to become Sea View Farms.

Essentially the colony became a place for the city of New York to ship desperately poor people out of sight and put them to work in exchange for a place to stay. In 1924 ownership of the site was then transferred to the Homes for Dependents agency, which in turn removed the requirement that all residents work on the property. Most of the work up until this point had involved farming many types of food, which was used to feed not only the residents but also other city institutions.

Until the 1930s most of the Colony's residents were elderly and the population was known to balloon to as many as 2,000 residents, but when the Social Security System was put into effect the population began a steady decline. The last residents of the Farm Colony were moved out in 1975, and the buildings have been left to rot and decay since then.

The land of the New York Farm Colony has been a constant debate since its closing. 25 acres were given to the Department of Parks and Recreation, the remaining 70 was designated an official city landmark in 1985. Most buildings are still standing but have fallen to extreme disrepair and vandalism.