This page of the Disgaea RPG guide explains what to do when you want to give up and end the fight.
If you think that the opponent is too strong for your party or you just want to finish the fight, go to the Menu. Its icon is located in the upper left corner.
When you open a new window, the game will tell you what you will lose by leaving the fight. These will be collected HL coins and chests with items dropped by defeated enemies.
Click on Give Up to exit the fight.
Then click on Quit Battle or choose Retry if you want to repeat the battle.