As soon as you have completed 2-3 journeys, you can return to the city. There is still plenty of time to complete royal quests. As of now, you have already completed 3/5 (for each dungeon you have completed) and you can focus on your kingdom now.
Start by spending one day with each character. Not only will you complete a royal quest this way, but also gain access to a number of new skills. Merchants in the city also unlock their shops and you will be able to buy from them. It is also a good idea to spend some additional days to build your friendship with Aliss and Buzz - this will provide you with additional bonuses in gaining their sympathy through conversations.
After that, upgrade the bridge to the second level of experience and start fishing. You will spend several days on doing that and it will definitely profit you - you will obtain many valuable items and will complete another royal quest. You can also upgrade the courtyard to the second level, thanks to which you will be able to take 6 characters on journeys (soon, more characters will join you).
In the meantime, you should have gained access to a new party member - Levant, a powerful warrior in a huge armor. He makes an excellent companion and is the best tank in the game. He has a number of excellent skills and he is highly valuable in battle. He can replace Griffith - although much depends on the battle itself and its objectives.
Also, remember to distribute new skills among your characters on a regular basis. At this point, your party should be at the 3rd experience level and you can pick several new abilities.
The first chpter should soon end. You have completed five royal quests, but it is only the beginning of your adventure!