Proposals for a 23-storey building next to the railway in Black Prince Road have been rejected by Lambeth's planning applications committee.
The site is currently occupied by Parliament House, a 7-storey 1960s office block.
In its objection to the planning application Westminster City Council described the proposed building as "an incongruous feature" which would have an adverse impact on the setting of the World Heritage Site and the view down Whitehall from Trafalgar Square to the Houses of Parliament.
Westminster City Council also warned that if the development were approved it would set a precedent for further "encroachments on the skyline" above the World Heritage Site.
Government design watchdog CABE praised the tower as a "striking piece of architecture" but Lambeth councillors were less enamoured of the scheme.
"I should like to start with the good points," said committee member Cllr Brian Palmer when asked to comment. "It's a quirky design. We are told that there is the use of high quality materials. Now I am struggling.
"Not even the architect's best friend would describe it as an elegant building. The quirkiness extends to very odd angles that could make one queasy looking at it. They seem to bend around in various directions that would be apparent without drugs or alcoholic beverages."
Vauxhall Gardens Community Centre members claimed that the building would impinge on the architecture of the former Royal Doulton building opposite.
English Heritage expressed concern on the impact on the setting of the former London Fire Brigade headquarters on Albert Embankment. Some of the thirty local residents who had objected addressed the meeting expressing fears about noise, wind and loss of light.
The proposal included commercial floor space and residential units. Two committee members suggested that the level of affordable housing was too low.
The site in Black Prince Road is just yards away from an even larger development designed by Foster & Partners which received backing from Lambeth Council last year.