
Originally the site of Lee Farm, the Manor House was built for Thomas Lucas around 1770. The farm had been purchased by William Coleman, Lucas’ uncle, for his nephew. Coleman was an agent for several plantation owners in the Leeward Islands. He also owned a plantation in Antigua and was a slave owner. Thomas Lucas was Treasurer and later President of Guys Hospital and had several joint business ventures with his uncle as well as owning a plantation in Tobago. Some people have also suggested that he owned a fleet of ships which may have been involved in transporting slaves from Africa to the Caribbean.
Lucas died in 1784 and the house was let to the Call family, but the Lucas family sold the Lee estate in the 1790s and it was bought by Sir Francis Baring in 1796, who also owned land and slaves in Jamacia. Francis died in 1810 and his son Thomas lived in the House, although he spent much more time at their estate at Stratton Park in Hampshire. Thomas Baring would eventually become an opponent of slavery.

The Barings let the House to Frederick Perkins, who had inherited a share in a Brewery, on the south bank of the Thames, from his father around 1805. Frederick doesn’t seem to have been very interested in the Brewery but drew enough income from it to subsidise his passion for collecting books.
The Baring family reoccupied the house and it became the London home of Sir Francis Baring, 1st Baron Northbrook, Chancellor of the Exchequer from 1839-41. In the 1970s I attended a school (no longer there) which stood adjacent to Manor House Gardens, which was named after Thomas Baring, his son, 1st Earl of Northbrook who made the grant of land for a school adjacent to the gardens and there was also a local pub a few roads away called ‘The Lord Northbrook’.

Around 1860, the Baring family let the house to the Farnall family. Harry Farnall was a government board inspector and became Metropolitan inspector for the poor law board. Working with Florence Nightingale, he instituted an inquiry into the quality of nursing provided by workhouse infirmaries. However, he was also accused of turning a blind eye to abuses in some workhouses.
The Farnalls moved out in the 1870s and it is not clear what happened next, but by 1881, the house was home to a military ‘crammer’ school, preparing young men for a career in the army. In 1898, the Earl of Northbrook sold the land to the LCC. The house, after refurbishment, became a library and the gardens were opened as a public park in 1902. In 1986 ownership was transferred to the London Borough of Lewisham and in 1997 the grounds received a heritage lottery grant to renovate the park and this work was completed in June 2000. In 2008 a lottery grant was obtained to restore the Manor House.
When I was at school in Lee, the Library also housed the local history centre before this moved to a new building in Lewisham. A library and a pre-school centre now occupy the Manor House.
My thanks to runner500.wordpress.com for excellent information on the history of the Manor House.