You can make transistors with some...hazardous components, but you may want to start with relays or tubes and resistors (which could, theoretically, just be varying lengths/guages of wire) and build NAND gates. It would be big, hot, noisy, slow, and a huge waste of your time, but there were computers before semiconductors.

For the curious: here's how you make your own tubes from glass, metal wire and metal sheets: http://m.youtube.com/watch?v=EzyXMEpq4qw.

In a practical apocalypse, you might be able to recover transistors from garbage dumps, too.

(http://www.youtube.com/watch?v=w_znRopGtbE) for transistors, but it feels like cheating.

Relays are doable from scratch (mining and refining copper; winding coils, etc etc) but it's going to be pretty hard work and you're going to have a very slow computer at the end of it.

I was planing on going the relay/vaccume tube route.