Send your horse back and enter Aim Mode (Q button). Aim at the dormant shadowbombs shown on the above screen and shoot them (by pressing the middle mouse button). The explosion will unlock a passage to the western part of the map.
Before heading forth, I recommend opening the world map and taking a few moments to plan the journey. You'll need to pass through the Shadow Gorge, going west, a then turn south towards the Fjord. The final road leads west between gorges.
Unfortunately, the journey to Drenchfort won't go without unpleasant surprises. After passing through the [Shadow Gorge], south of the Shattered Forge, you'll be forced to stop at the spot shown on the above screen.
Depending on preferences, you can either stay mounted, eliminating monsters while riding (left mouse button), or send it away and fight on the ground. Be cautious especially of the Prowlers. They are as much agile as the recently encountered Stalker, but luckily much less resistant.
Apart from Prowlers, take care of the Stingers as well. The best weapon to kill them with is the pistol (you don't have to aim manually, but simply keep pressing the middle mouse button until you run out of bullets). Try to destroy the nest shown on the above screen as quickly as you can, best with your most powerful attacks. Destroying the nest will prevent other Stingers from appearing in the area.
Only then see to the blocked passage, which in this instance is a lowered grate. Find dormant shadowbombs left of the grate and aim at them with your pistol (the above screen). The explosion will unlock a way to the upper ledge. Climb up and look around for a Boatman Coin.
Enter the building and prepare for a rather difficult battle. At first, you'll be attacked by Stingers, but then Prowlers will also appear. While fighting, notice shadowbombs visible in the distance; it's a good idea to detonate them as soon as some monsters come close. You won't kill all of them, though, and sooner or later you'll have to go back to a regular fight. Dodge frequently, use special attacks and be careful not to fall into the large pit.
Afterwards, use the newly unlocked passage, finding another Boatman Coin. You can now jump down and resume the journey, summoning Despair for assistance. Head south and after reaching [The Fjord], turn south-east, and finally east. Watch out for other Prowlers and Stalkers; if you have a low-level character it's best to just go around them from a large distance.
Be especially careful in the last stage of the journey, because among the two paths leading west you have take the lower one, as seen on the above screen (use the map for help if needed). After a while you'll reach your destination, which is [Drenchfort].