My previous character as a Noghri could beat any brawler 4 times a day without any problems. My current character can just about beat a brawler once. There is also a very marked difference in the amount of damage done during combats.
Ironically it seems to me that the most important attribute for brawling is Strength,Dexterity and Vigor. It seems unarmed combat and dodge seem to play very little into the equation.