Start by switching to Will, who'll eliminate the pirates on the roof. After the fight, destroy the barricade with a powder keg. Then head through the city up to the fort.
Again, throwing axes will come in handy here - the ropes of the drawbridge can be cut with them. Inside the fort there's a large number of pirates so it is best to thin their ranks with Jack's grog bombs. After the battle, turn the crank to free the denizens of Fort Charles and go up to the upper part of the fort using the lift.
Before you kill all foes within, find the bit of damaged wall to the left and blow it up with a powder keg to find another Map Piece.
Immediately after defeating the enemy on land, several boats with reinforcements from the nearby ship will come. Sink as many boats as possible with the cannons - each is worth a whopping 500 gold pieces.