I used to be quite successful at lucid dreaming when I was younger and had more time on my hands. One of the most enjoyable experiences was when in one dream I found myself atop a very high multi-level office style building, seemingly around 20-30 stories from my vantage point. I knew I was dreaming, and jumped, knowing full well that I wouldn't die. It felt awesome. I landed at the bottom amid a large pile of cardboard boxes. Perhaps the boxes were my minds way of making my survival of the fall vaguely plausible.
I also had a number of dreams where I consciously decided within my dream that I wanted to wake up and did.
My strategy for reaching this kind of lucid dreaming was to use a reminder of some kind. I had read that some studies were using a kind of apparatus attached to ones head, like glasses that detected when you had entered REM sleep by the movement of your eyes and turned on and off soft lights pointed at your closed eyes to alert you that you were dreaming.
I had nothing so sophisticated, so I went with a simple radio next to my bed, playing softly as I went to sleep. I would at times hear the radio in my dreams and it would alert me that I was in fact dreaming.
It's a fascinating area of self exploration that I would recommend to anyone who has the time.
I've been able to do this in the past. It has always happened by chance.
These experiences often left me in a state of well being.
But I've also experience (quite often) what could be called a lucid nightmare: I understand that I'm dreaming and I decide to wake up. I shake myself, I slowly open my eyes and I start to see the room I'm in. Only to discover that I am still dreaming. At this point, fear kicks in and now I really want to wake up. But the cycle repeats three or four times before I'm relly awake, and each time I'm increasingly more scared of not being able to wake up anymore.
From what you say, I think that you could have experienced what it's known as "sleep paralysis", I remember reading about the subject when going over the same problem in my sleep. Maybe you should take a look on this matter, so you know exactly what you are dealing with, thus not being afraid of this condition.
I have had a lucid dream where I am flying above a farmland. I fall many times but still manage to fly for a few moments. I was showing it to somebody I don't remember. As for the "recursive dream", I did have it after watching Waking Life http://us.vdc.imdb.com/title/tt0243017/
The best success I've had with lucid dreaming is doing it directly from wake, Feynman style. The cue is when you notice yourself going into sleep paralysis. It only happens to me while sleeping on my back. It feels like an incredibly intense energy going through your body. If you're not aware of what it is, it can really feel like you're going to do if you don't snap yourself out of it. You won't die, you feel this way every night while your senses are disconnecting, you just normally sleep through it. Just relax, let the energy flow through you, and float away.
It can of course be tough to fall asleep on a stimulant, but some chocolate can make it easier to stay awake as you are falling asleep. 3 or 4 squares of 70% dark chocolate works very well for me.
I have been trying this for a long time but from a spiritual side. I have been somewhat successful at it. I say somewhat simply because lucid dreaming is not the main goal.
One of the tricks that I like is, don't laugh, to pull a finger. In a dream, this one will stretch without any difficulty. I has never failed me but once. Once I gain lucidity, I usually jump in order to fly. And yes, it is an amazing experience. By the way, from my experience, consciousness in a dream is not binary (ie., you have it or you don't). I have observed that the degree of consciousness varies depending on several factors. Some depend on your health, mentally and physically. But other factors seem to be more important, at least for me, but these are of spiritual nature which I won't discuss here.
I believe that one of the main problems on why is so difficult to become conscious in a dream is the possibility that the mind is another entity than yourself or something like that. I remember reading something along those lines when searching for meditation. I believe that Buddhism teaches something like that but I really don't remember. I was able to experience this disconnect once in a dream. I distinctively remember looking at an event which I knew was not possible, but at the same time I had a thought that said that it was possible.
Having experienced this, I know try to learn more about it. No long ago there was an post here on HN about research regarding how cutting between scenes, in a movie for example, affects memory- I think. Sadly I don't have the link. I find this very intriguing since many of my dreams are like movies where viewpoints and scenes change without me questioning it. I must admit that I have watched a lot of TV in my life and I am constantly trying to limit the amount or stop it completely. I would throw the TV out if it was up to me but it's not.
All I need to do to have a lucid dream is simply go to bed very determined to have a lucid dream. Easy, but I have no plans to do it again, because it leaves me feeling mentally exhausted like I've been working instead of sleeping.
I had a lucid dream when I was much younger then couple years ago I tried a couple methods and imediately when I realized I'm dreaming, i'd wake up. Experimentation also led to sleep paralysis on couple occasions.
I've been able to do this to some degree for most of my life. It comes and goes. Lately I think this is connected to the temperature of the room I'm sleeping in, as it gets colder my dreams get more complex, chaotic, and lucid... but I actually try to avoid that, because it also makes my sleep less effective; I'd rather sleep well than have funny lucid dreams, which should give you a sense of how valuable they are.
It's sort of cute and fun, but I do sometimes wonder if this idea that it's some sort of life changing thing is either A: some people are wired differently enough that it happens for them, but this has little-to-no applicability for the rest of us or B: it's sort of become a New Age-style touchstone of faith that this is some sort of way of overcoming your mind or something and the claims that it's some sort of magical adventure are more wishes than reality, fed by the same sort of feedback loops that run most other New Age sorts of ideas.
Perhaps both.
I do admit to a little fascination by the parameters of what is controllable and what is not, rather more than the dream. This gets perilously close to the "who cares?" sort of dream discussion, but, hey, you're the one reading the comments on HN for an article entitled "Can you teach yourself to control your dreams?", so don't come crying to me about it. I have noticed:
* It is indeed sort of possible to completely rewrite the dream to do something like "win the superbowl", but the effort of the will involved is often enough to just flat wake you up. If you are thinking of getting into lucid dreaming I'd recommend less obtrusive changes. Flying's a good start.
* One common observation is that dreams involve your body image, which is in fact a real sense (related to proprioception but not the same). Your body image is surprisingly fuzzy in some ways; as the article mentions, you might not quite have five fingers. Another one I've found is that I must NOT look in a mirror for any period of time. Seeing your own facial image is distressing enough, then it starts... distorting. It's the same sort of thing as http://growabrain.typepad.com/photos/uncategorized/four_eyes... , only... worse.
* I find that while I fly quite freely, I can't actually get above everything. If I fly above one tree line, the next one is even higher, if I fly above that, there's suddenly an even bigger cliff, eventually I get stuck. I can transition from land into space but only if I close my eyes within the dream during the transition. I think this is actually because I don't fly much, and when I do it is as a passenger on a commercial flight; I have no internal frame of reference for really flying. Amusing while awake, sometimes frustrating when asleep.
* Any given trick only works twice well, then a couple more times with effort, then stops working, in quick succession. If I can walk through walls or windows, it works like that. If I pick up a weapon (hey, I'm a guy), it works like that. Flight tends to work like that. I think it may actually be neural, just like saying the same word continuously for 30 seconds makes the word "stop being a word".
I am aware of the possibility that this is a self-fulfilling prophecy. It may be. Doing any sort of science on this topic would be tricky for this reason, I think. For instance, this article says that light switches never work in dreams... well, mine do about 50% of the time. At least the text or time tests have some science behind them; ISTR claims that those parts of the brain really do shut down more so it makes sense that the vast majority of dreams will have trouble there. Still, every once in a while I do get a bit of stability on some text or something. I try not to think about it too hard, lest my brain talk itself into not giving me any clues it is asleep....
Hmm well I won't speak to the "New Age-ness" of any of this stuff as I like to stay as far away from that side of things as possible. But I believe proving to yourself that you can have at least some control over your own dreams can provide some significant benefit. When I was quite young, about 4 or 5 years old I had this recurring dream that I was being chased though our family home by a Wampa (from Star Wars, why I had seen The Empire Strikes Back at that age is another story). This dream freaked the hell out of me as you can probably imagine. I told my mother about it and she said that I could control my dreams, that I either needed to make friends with the monster or kill him and that I had the power to do either one. I'm not sure how long after that it was, but there came a night and a dream where I did in fact make friends with this huge white monster, took him downstairs by the hand and showed him my toys. Now 25 or so years later, that was the last recurring nightmare I ever had. I barely have any nightmares at all in fact. I attribute this mainly to this experience at an early age.
So for me, I guess I would call that pretty life changing, but perhaps not in the way that others might think of when they attribute the term "life changing" to lucid dreams. I feel that there is some degree of mastery one acquires over one's own mind with these kinds of exercises. A feeling that this is my mind and I control it. That's always been pretty valuable to me.
Actually, I do agree with the nightmare control aspect.
"I either needed to make friends with the monster or kill him and that I had the power to do either one."
There is a third possibility: Let it catch you. That's actually how I got rid one of my recurring nightmares. I'd swear there was a second's pause as my brain asked itself "Uh, now what?"
I now fairly frequently get some imagery in my dreams that have the capability to spark nightmares, but they pretty much never get past the very beginning phase, because my brain knows from experience nothing bad can happen. Big scary monster with teeth chasing you? The best answer is to hop right on in. Another big scary monster with teeth inside the first one? Hop right on in again. Recurring dream involving fear of heights? Just fall. Eventually it'll all work itself out.
Looking back at it, I guess that's kind of what I did too, although not thinking of it that way. To this day I can remember the scene in my dream. I stopped at the top of the stairs that went down into our basement, turned around and let the monster walk up to me. I can't remember if I said anything, but I do remember holding my hand out and taking his hand.
I now fairly frequently get some imagery in my dreams that have the capability to spark nightmares, but they pretty much never get past the very beginning phase
That's pretty much how I feel as well, except I wouldn't say frequently for me. Occasionally I'll get imagery that is kind of nightmarish, but it usually doesn't seem to escalate beyond a certain point.
Facing one's fears, be they real or imagined usually has some value, in my experience.
I experimented with lucid dreaming a few years ago, and did it for many times. I saw myself in mirrors and also in my own image of I myself sleeping somewhere. My own image was always wrong in some way, and the memories of some of those images are still completely creepy many years after the dreaming happened (I still have goose bumps for the memory of the image of one of these specific dreams... quite crazy really since it wasn't a very bad image that you wouldn't see in a bad horror movie for example).
I was practicing this for sometime, and was able to do it for a while. At some point, I completely stopped remembering my dreams, and once that happened, lucid dreaming stopped too.
I also had a number of dreams where I consciously decided within my dream that I wanted to wake up and did.
My strategy for reaching this kind of lucid dreaming was to use a reminder of some kind. I had read that some studies were using a kind of apparatus attached to ones head, like glasses that detected when you had entered REM sleep by the movement of your eyes and turned on and off soft lights pointed at your closed eyes to alert you that you were dreaming.
I had nothing so sophisticated, so I went with a simple radio next to my bed, playing softly as I went to sleep. I would at times hear the radio in my dreams and it would alert me that I was in fact dreaming.
It's a fascinating area of self exploration that I would recommend to anyone who has the time.