I've walked past there almost twice daily for the past 4 years (with small children) and this is the first time I've ever seen or heard of any trouble. I certainly haven't been aware of any ominous vibes either. But as with all places in any area you just need to take care and be self-aware.
I've seen no "trouble", but I have seen groups of half a dozen or so teenage boys arguing in a rather aggressive way on Crosby Row a few times, outside what I think is some sort of youth club, and I've also seen someone let a firework off in the street there. Nothing more than that.