Remove Rajan as RBI governor, Swamy urges Modi