Planning permission has been refused for the Founder's Place development next to Archbishop's Park.
Canterbury House, Stangate and the Holy Trinity Centre seen from the London Eye
"I cannot see anywhere else such a dense development" said committee chair Brian Palmer. "Living in the lower flats will be like sitting in the Grand Canyon."
He feared that there was a problem of trying to squeeze as much as possible on to the site and expressed concern over its bulk, massing and relationship with Archbishop's Park.
Cllr Toren Smith added that approval would mean a loss of any extension to the park.
Cllr Jeremy Baker spoke of the oppressive nature of the plan and added his concern about conditions in lower flats where the view would be of a blank wall nine metres away.
"Too much is being crammed on to the site" added Cllr Baker. "This is a step too far."
Earlier the applicant, Guy's & St Thomas' Charity, had announced last minute concessions including a park view in the new development for all existing tenants in a block which would be demolished.
Council Leader Peter Truesdale suggested that the concession allowing tenants to choose where they were relocated should be put in writing and warned the committee that no decision should be made unless members were convinced that they had all the facts before them.
Martin McCool, chairman of the Canterbury and Stangate Tenants' Association, speaking for the 36 families who would be displaced by the scheme, said that the plan would mean a third move for some tenants. He called for "appropriate" consultation.
Kate Hoey MP described the Charity's consultation as "shoddy" and expressed concern about such a development in a conservation area. She reminded members that a new inspector's report supported the retention of the Holy Trinity school buildings threatened by the plan.
Jeremy Hart, a nearby resident, and Sarah O'Connell, a Friends of Archbishop's Park committee member, both called for the unique history of the site to be recognised along with its relationship to nearby Lambeth Palace. The Charity claimed that the Church Commissioners had been involved in consultation although none was recorded in agenda papers before the committee.
All Liberal Democrat committee members opposed the scheme along with Labour councillor Toren Smith. Ruth Ling, Labour member for Clapham Common, was the sole supporter.
For the latest local news and events direct to your inbox every Monday, you need our weekly email newsletter SE1 Direct.
7,000+ locals read it every week. Can you afford to miss out?
Read the latest issue before signing up