You have to bridge the gap between bits and voltages. I don't remember the titles of my books, so I will include keywords (You're probably past steps 1&2? Working backwards from #4 would also work).
1. Break down assembly language even further and look into OP codes as well as the FDOES (Fetch-Decode-Operands-Execute-Store) cycle. Think clocks and busses. [microprocessor architecture, bus architecture, instruction set, instruction architecture]
2. Move further into details of how ALU and memory are implemented: how flip-flops are used to store state, and how ALU's adder circuits, etc. can be implemented using NAND gates. Know what a 7401 is. [digital circuit design, half adder, full adder, flip-flop, register]
3. Then at a lower level, study how NAND gates themselves are implemented using transistors. Know about BJTs and FETs. [transistor electronics, electronic circuit analysis and design, BJT, FET]
4. You can be happy at the transistor level, but to solder things that actually work (and at the same time, know what you're doing), you have to study electric circuits and power electronics [electrical engineering, power electronics, ohm's law, thevenin, kirchoff's circuit laws]. Know how to read the color bands on resistors and appreciate the cheeky mnemonics for BBROYGBVGW :)
5. If you want to grind your own sand to make your chips and transistors, you may want to look up material science
*Be careful not to inhale the lead fumes, lest you suffer brain damage :)
Now if someone could recommend books for each stage...
(It's hard to recommend self-learning hardware, because I was taught hardware and am self-learning Computer Science.)