In the fifth installment to the series, melee fights have not changed too much, in comparison with Grand Theft Auto IV. You will learn about the mechanics of fighting, for the first time, towards the end of the fifth main mission, but there is nothing to stop you from learning it on your own.
You should start each fight by locking onto the opponent. You can throw slow or fast attacks. The former carry a greater probability of reaching the target, but they do not inflict much damage. Strong attacks, ion the other hand, may be unsuccessful but, if they reach the target, they are capable of inflicting more damage, and may bring the enemy down in extreme cases. It is just as important to perform dodges correctly. It is worth noting here that you can either restrict yourself to performing the dodge itself, or counter the enemy right afterwards.
If you plan on winning melee fights on a regular basis, I recommend that you, first of all, increase the strength bar for your current character. You can ensure yourself with an easy way to success by playing as Trevor because he is capable of inflicting much more serious damage in melee fights after activating the fury mode (his special skill).