You must continue brawling with him (which is what you accomplished by talking to him) until one of you reaches 1 HP. At this point, the brawler will either state that he bested you or that he was no match for you. The loser gets 1 XP, the winner gets a random amount between 1 and 5 XP.
After each match, the brawler's HP will automatically regen to 100% and you may brawl him again (for a total of 4 matches per 24 hours - if I recall correctly). However, you only regain HP at the normal regen-rate.