Activating the quest
As you're around Riften, visit the city harbour. Apart from a couple guards and dock workers, you will also come across a certain Argonian - From Deepest Fathoms. Approach her and the frightened woman will ask you for help in a certain case. She apparently has stolen a cursed item - the Lexicon. Nightmares will keep tormenting her until she gets rid of it.
If you agree to take on this mission, you will receive a task of reaching Avanchnzel and taking the Lexicon there. The dwarven keep is well guarded by Automatons, so you should equip your best weapon and stock up on potions.
New main objective: Take the Lexicon to Avanchnzel
The dwarven keep can be found west of Riften, at the bottom of the mountain (screen below). It's best to get there from the Orc camp of Lorgashbur, though if you haven't yet been there, you can start at Riften and go round the lake.
After reaching the bottom of the mountain, keep going up the slope and afterwards follow the dwarven path. Soon enough you should reach the keep entrance (screen above). After getting inside, you will witness the ghosts of the previous treasure seekers - apparently the keep became their eternal prison. What's interesting, one of the ghosts is From-Deepest-Fathoms, from whom you received this quest. Has her soul been trapped here and her body left to roam Skyrim?
Keep following the path made by the ghosts, going forward all the time (don't turn left at the first intersection - it's the way back). On your way you will come across a couple Dwarven Spiders and Dwarven Spheres - try to eliminating them from a distance, as they are rather problematic in direct combat (the spiders explode after dying). Keep going onwards, fighting enemies on your way and eventually you will reach the Avanchnzel Animoncultory.
Defeat another Spider and Sphere inside and turn left. Two ghosts will appear and try to open the door, though none of them has the required skills. If you're good at lockpicking, you can enter the room and collect the treasures found inside.
As you explore the further part of the dungeon, you will reach a thieves camp where you will once again see the spirits of the thieves. Apparently one of them wanted to back out, but the others talked him into continuing. You also shouldn't back out. Look out for the door on the fight side of the corridor (screen above), as it's protected with a trap - run straight into the room after opening the it. Inside the small room you will find lots of treasures and the Racial Phylogeny skillbook which will boost your Restoration.
Head out of the room and turn right, where you will come across a Dwemer storage room. On the desk on the left you will find some precious stones, which are however guarded by a Dwarven Spider. Defeat it and move onwards, fighting with more Spiders and Spheres and clearing the rooms from whatever you consider useful. Afterwards enter the room where two Centurions are lying on the assembly table. Use the fact that you're on higher ground and kill all the Dwarven Spiders below. Afterwards head down onto the lower level. Luckily the Centurions weren't finished, so you can roam the room freely - use this chance to steal everything that's possible. Afterwards head out of the room using the passage on the left and go pass one of the thieves' body. Watch another conversation and enter Avanchnzel Boilery.
Cross the room and kill the Dwarven Sphere that will cross your way. Afterwards move to the room with a trap (screen above). As the deadly blade is moving back and forth through the room, you need to move between the horizontal obstacles so that it doesn't hit you. If you're fast enough, you should quickly get to the lower level, where the body of yet another adventurer lies.
Eventually you should reach a room in which the last of the thieves died (of course not counting From-Deepest-Fathoms). Above her body you will find an inactive centurion and a second one, this time fully operable, who you will need to fight. Throughout the battle, try to keep the enemy at distance and shoot at him - he's rather slow, so you should be able to do it. Defeat it and afterwards search its body, the body of his friend and the thief. Put the Lexicon onto the pedestal in the middle of the room. The quest will come to an end and you will have to get out of the dungeon.
Head out of the room using the passage behind the Lexicon altar and unlock the door. You will return to the main part of Avanchnzel, to the corridor which you ignored previously. Kill the Spider and Sphere and head out of the dungeon. You can do it by heading through the Balcony (corridor on the right) or following the path you used to get here (path on the left). Regardless of which you choose, you will quickly return to the surface.
In return for completing the quest you receive the very useful Ancient Knowledge, thanks to which your Blacksmithing increases 15% faster and any Dwarven armor you wear is made 25% stronger. Apparently the Lexicon creators know how to reward assistance.