If you want to take the enemy ship, you have to check how many people he has on the board. Compare that number with your, and decide. If number of your soldier outnumber his, then you can start boarding operation. If not, better use grape shots first (his crew will be decimated and it will be easier to take control over that vessel).
If your opponent will try to run away, destroy his sails and mats - he will be much slower.
Your people have to throw ropes to the enemy ship. You cannot do anything at that moment (only defend yourself).
During the fight is good to check condition of the enemy vessel. Just move your cursor on the ship and the special pop-up will be shown. You can read there if the enemy is ready to surrender (if it will be true stop the boarding operation). Sometimes hostile vessels can surrender without boarding operation -later, you can simply conscript them to your fleet.
After the fight you can take look at some statistics. All captured units will be shown there and you can decide to conscript them to your fleet or not. If you'll choose the first option you will have to pay them. In the second case you can get some prize money for the captured vessel.