There's usually a reason that tracks will post signage saying, "no drifting." Chances are it's probably because drifting, especially if you're not really all that good at it, means you're going to be swinging around all over the road, driving slower than anyone wants to, and blocking everyone from passing.
Also, if a track has rules, follow them; it doesn't matter if you don't like them. Not your track, not your rules to set.
LuckyDuckRacing posted a video outlining how dumb most people look when they drift.
Not to say that drifting is bad, it's just all about location.