I've done a similar thing a few years ago when I was first starting work on my guild hosting company's code.
At the time, the main thing hosted on that machine was my WoW guild's website, which I had been working on for close to a year, and was beginning work on converting the site over to a general purpose guild hosting site.
I was doing some work for a client, setting up a mirror of sorts for some kind of yearbook thing I had built for them. For that, I made a script that would mirror the online yearbook with wget, zip up the whole directory, then clear out the mirrored pages (all I cared to store and serve was the zip file).
All of my websites were stored in /www on the server, and the raw yearbook was located at /www/clientname/www. Inside the clientname directory, I had the mirror script which was something like this:
wget -whatever_options http://whatever.address www
zip yearbook.zip www
rm -fr /www
Unfortunately, because of how frequently I type / before www to get to my web dev directory, I instinctively put "/www" in the script where I just wanted to do "www". I ran the script, checked to make sure and it looked good, and deployed it to a cronjob.
My heart sank when I tried loading my guild page a few minutes later (just to see what was going on on the forum, if anything), and it served up a bunch of 404s.
I went to /www/guildsite and saw it completely empty, and almost immediately figured out what had happened. At that point, I had to get my composure and figure out what I was going to do (I did not have backups or source control). I unmounted the directory, and went off to lunch with a friend, shaking with anxiety.
Upon return, I started writing out a perl script to scour the device byte for byte looking for PHP scripts, looking for instances of <? and then capturing the following 2000 lines or so, assuming that would be a sufficient buffer. When the script was done, I set it to run on the partition, and 45 minutes later I had a few hundred files to work with.
I had to manually go through every file (which were named numerically in the order they were found in the filesystem) and determine if it was the most recent (or recent enough) copy of the code, clear off any trailing bytes, and rename it to the filename it used to have. Luckily I could remember the name of almost every file in the system. It took about 8 hours to go through the few hundred files and recover them.
Needless to say, I learned my lesson after that, but the feeling of victory I got from recovering those files from the device was epic.
6 years later, I realize that that's a rather trivial thing to do, but at the time, I didn't know what I was going to do, and remembering that the file system doesn't clear all the bytes of the a file just it's reference gave me tons of hope.
Ha - I'll bet everybody here has a story that starts like that. (Although your heroic save was heroic!)
My sad delete story was in 1982 - I had a fantastic graphical robot-vs.-robot game stored on tape on an HP portable computer (I was in high school). For some reason, the delete command on the HP had a variant that deleted from a given block forward on the tape, as I recall, and for some other reason, my fingers just decided to type that variant even though I had never done it before.
I still miss that game. It just may have grown in my memory, but I'm pretty sure that was the coolest piece of software ever written anywhere at any time.
Mine was plugging in the power connector backwards on a super important ide drive (it was the only backup for an already failed server) and watching smoke issue from its controller board.
Having to explain what I'd done to the boss was so scary, I slunk home with the drive and traced the power circuit with my oscilloscope until I found a capacitor that had blown short. I soldered on a through-hole replacement and it worked!
I pulled the data and felt like king of the whole frikkin world for the next week or so.
My gosh that's one of the best stories I've heard in a long time. The thing that worries me is that things like this are going to me rarer and rarer - there seems to be less interest in electronics, and less electronics that can be hack-fixed like that.
I am a software engineer by degree, but also took many electrical engineering courses in uni, and I can honestly say that it is starting to make a comeback. Sparkfun, Adafruit, MAKE and many other places are starting to make it more accessible, cheaper and easy to learn. The Arduino has been a boon, providing people with a cheap but powerful microcontroller to get started.
While yes technology is getting smaller I have found that with many parts I can now easily find replacements online, I can get advice from other professionals, I can easily figure out how something works so that I can fix it. I've currently got a power supply sitting on my work bench that has a weird issue and I am slowly going through, making a net list and building a schematic with part numbers in an attempt to isolate the fault.
Maybe I am a rare breed, but seeing as how the interest at Maker Faire keeps going up, and interest in electronics also keep going up I will assume that eventually more and more people will get into experimenting in this field.
>Mine was plugging in the power connector backwards on a super important ide drive (it was the only backup for an already failed server) and watching smoke issue from its controller board.
yep this could cause you to lose a job.
> Having to explain what I'd done to the boss was so scary, I slunk home with the drive and traced the power circuit with my oscilloscope until I found a capacitor that had blown short. I soldered on a through-hole replacement and it worked!
pulling this off, however, could land you a job!
lesson: mistakes are human. talent and initiative are rare.
Yeah, the same thing happened to me a couple of years ago and at the time I thought I was the coolest guy on the planet when I recovered most of my data. I had accidentally deleted all my chat logs. Luckily, all the chat logs had a timestamp in them, so I just searched for all timestamped lines on the disk and dumped them to a file. I then wrote a script to group and sort all the logs based on the date and who I was talking to and recreate all the files. It worked better than it had any right to especially since I was doing everything on a live disk!
In actual content, it's convenient that you had timestamps to work with. That eliminates a lot of the need to trim off trailing bytes. Kudos to you for the epic save, and on a live (mounted?) disk too? Living dangerously :)
At work, we had a USB drive full of Ghost images. Someone else didn't know how to use Ghost that well and they managed to nuke the partition table by trying to write an image on the USB drive onto the drive itself... or something like that anyhow (newer versions don't let you do that any more, I note). Fortunately, they didn't destroy the data itself.
I rebuilt the partition table and saved all of our backups.
That's like the time my supervisor was done setting up a server, and then was going to image it. He had two identical drives in the machine and imaged the empty to the full drive instead of vice-versa... I was the one to cleanup the mess, but learnt a lot.
There’s an important motto to bear in mind here: data you haven’t backed up is data you don’t want. To be fair, this does bite us all at one point or another, but once it’s got you once you make damn sure it doesn’t get you twice.
Also, these days I get shaky just doing FTP deploys. Give me capistrano and git, or even better a direct Heroku push any day of the week.
There wasn't any version control software out there 6 year ago? Oh, I've been programming for 7 years, I remembered that I used VSS when I started working. But maybe I was using pirate version and you couldn't afford it.
There were plenty of RCSes out at the time, and I knew about them, but didn't use any of them. It was still a pet, personal project and I didn't consider it important. I had used VSS at my previous job and hated it, and I hadn't taken the time to learn any of the OSS VCSes at the time.
It was laziness, nothing more, and I got burnt playing with fire.
At the time, the main thing hosted on that machine was my WoW guild's website, which I had been working on for close to a year, and was beginning work on converting the site over to a general purpose guild hosting site.
I was doing some work for a client, setting up a mirror of sorts for some kind of yearbook thing I had built for them. For that, I made a script that would mirror the online yearbook with wget, zip up the whole directory, then clear out the mirrored pages (all I cared to store and serve was the zip file).
All of my websites were stored in /www on the server, and the raw yearbook was located at /www/clientname/www. Inside the clientname directory, I had the mirror script which was something like this:
Unfortunately, because of how frequently I type / before www to get to my web dev directory, I instinctively put "/www" in the script where I just wanted to do "www". I ran the script, checked to make sure and it looked good, and deployed it to a cronjob.My heart sank when I tried loading my guild page a few minutes later (just to see what was going on on the forum, if anything), and it served up a bunch of 404s.
I went to /www/guildsite and saw it completely empty, and almost immediately figured out what had happened. At that point, I had to get my composure and figure out what I was going to do (I did not have backups or source control). I unmounted the directory, and went off to lunch with a friend, shaking with anxiety.
Upon return, I started writing out a perl script to scour the device byte for byte looking for PHP scripts, looking for instances of <? and then capturing the following 2000 lines or so, assuming that would be a sufficient buffer. When the script was done, I set it to run on the partition, and 45 minutes later I had a few hundred files to work with.
I had to manually go through every file (which were named numerically in the order they were found in the filesystem) and determine if it was the most recent (or recent enough) copy of the code, clear off any trailing bytes, and rename it to the filename it used to have. Luckily I could remember the name of almost every file in the system. It took about 8 hours to go through the few hundred files and recover them.
Needless to say, I learned my lesson after that, but the feeling of victory I got from recovering those files from the device was epic.
6 years later, I realize that that's a rather trivial thing to do, but at the time, I didn't know what I was going to do, and remembering that the file system doesn't clear all the bytes of the a file just it's reference gave me tons of hope.