Plans for 72 new affordable homes on Harper Road were given the green light by Southwark Council's planning committee on Tuesday night.
Family Mosaic housing association wants to demolish the disused low-rise block of 20 homes in front of Symington House and replace it with a new development rising to seven storeys.
The scheme, designed by dRMM architects, has 22 shared ownership homes and 50 flats for social rent.
The site came to national prominence when it was used as the setting for Roger Hiorns' Turner Prize-shortlisted 'Seizure' installation.
Hiorns filled an empty flat with boiling hot copper sulphate solution which was later drained to leave a cave of blue crystals.
It is more than four years since dRMM was selected as the architect for the site.
The planning committee heard from Debbie Walsh and John-Paul Maytum of the Lawson Estate Tenants' & Residents' Association who outlined their objections to the proposal which they believe will lead to segregation between the existing community and new residents.
The development will include a new garden to be bordered by a fence.
Mr Maytum also criticised the consultation Family Mosaic had carried out with neighbours: "We have been talked at a lot but not talked with," he said.
He added: "We want the site developed but this is undeniably really not the right plan."
Angela Wood of Family Mosaic told the committee that the homes would be offered first to former Heygate Estate residents who had indicated that they wished to exercise their right to return to the Elephant & Castle.
"We will be good local neighbours and our record in Southwark shows this," she added.
She explained that the planning application needed to be decided urgently because the funding for the development from the Homes & Communities Agency depended on contracts being signed during November.
One particular point of contention is the gated garden. Ms Wood said that no decision had yet been made as to whether the gate will be locked.
"We oppose this because we only have one opportunity to get this scheme right," said Cllr McNally. He said that an opportunity had been missed to create a shared green space rather than a gated courtyard development.
He added: "Haste to get the HCA money is not a good reason for riding roughshod over local residents."
Committee member Cllr Robin Crookshank-Hilton questioned the use of "Teletubbies-colour" green glazed bricks. Design and conservation officer Michael Tsoukaris stressed that there is a fine Victorian tradition of using glazed bricks in the courtyards of London mansion blocks.
The committee resolved to grant planning permission for the development.
For the latest local news and events direct to your inbox every Monday, you need our weekly email newsletter SE1 Direct.
7,000+ locals read it every week. Can you afford to miss out?
Read the latest issue before signing up