You're driving along a road and you notice a pothole. You pull over to the shoulder, put your hazards on, open up the trunk, take out a reflective vest and tape measure, then you begin to analyze the pothole. You spend an hour analyzing the depth and formation of the pothole and determine that the cause is due mostly to poor mixing of asphalt. You call the town hall and learn that the road construction crew uses a Caterpillar BG500E wheeled asphalt paver. After some extensive research you determine that poor mixing occurs from the inferior design of the BG500E's auger and that upgrading to the BG600D with its improved auger would cause better asphalt mixing and produce paved road conditions less conducive to potholes.
By now it's 9:30pm. It's dark and cold. You realize what your original purpose was: Dinner with friends. That was two hours ago. You missed dinner, but hey, you got some satisfaction.
The above was more of an analogy about Yak Shaving than Rabbit Hole Syndrome, but there are parallels. Like you, I used to be obsessive about details and solving subproblems. I used to come home from work and work on my own side projects for similar reasons as your own.
But then an advisor said something like, "You need to focus. If you want this thing of yours to succeed, you have to focus on making it succeed. Nothing else should matter." So I stopped my side projects and I became so effective at building our product that my employees wondered if I ever slept.
Even within the context your product, surely there are cases where you're presented with a problem that can be solved with several different depths of thoroughness. How do you, generally, choose a solution which optimizes for, success, time, and personal gratification?
The first step is admitting that you can't optimize for every direction. Facebook has a motto, "Done is better than perfect," which I used to think was cheesy, but realized is quite brilliant. "Perfect" leads to satisfaction, but "done" leads to revenue.
Say you want to try alternative payment processing methods on your site. Your site it tightly integrated with PayPal, and you want to add Stripe. You could (A) spend 1 day to bolt on Stripe with a hack in the HTML, (B) spend 4 days and add enough views to the frontend to make Strip work in a non-disgusting manner, or (C) spend two weeks and refactor the entire backend with a generic interface that will handle an infinite number of payment solutions.
All three lead to success, but you can't optimize for both time and perfection. Set a time budget. Leave breadcrumbs for the next programmer. Instill a culture that says, "Our code isn't perfect, we know this, but we do what we can with what we have." Done is better than perfect.
Although I agree with your idea in principle, there are only so many times you can just hack something on top before you start to wonder if the refactoring technique is actually the better way to go. You could just end up with a hacky system held together with sticky tape.
With forethought, you should have built your system originally with method C in mind. If he had done this at the beginning instead of just hacking it together then all this work after wouldn't have been needed!
1) Apply conscious planning and forethought. Some things are clearly going to be used a lot, some things might not. Use bolt-on hacks where appropriate, invest time for things you expect to be using forever.
2) When the longevity of the tool is not easy to pin down, test it out with a hack. Spend 1 day adding Stripe the hacky way, and keep a slushbucket of time budgeted for revisiting experimental hacks that look like they will last a while.
That's my general approach, anyway. Most of the time, a hack is good enough and anything more is a waste of time. The story is admittedly a little different for me though; for me, software is a tool to do my job- software is not my job.
I think I see it differently as we have just spent 3 years developing a CMS at our company and every decision has been a well thought out and concious one. We spent 3 weeks planning when we first conceived the plan.
We have only had to do 2 refactors in that time on 2 controls. That was more about understanding our customers needs more as we progressed rather than not enough planning though.
Previous to that I built a custom ecommerce site for a wholesale distributor. That whole process was more suited to a quick hack over a well thought out decision. That's why I say I agree in principle, so long as the project allows it.
Don't get suckered into premature optimization. If you're still small, bolting stripe on lets you accept money and get back to building the rest of the product. Sometime in the future it will make perfect sense to redo the entire payment system. Today is not that day.
"Done is better than perfect" is brilliant. It is motivational, succinct, and easy to believe. From my particularly context of a side project with no real deadlines, however, I can have perfect if I want to. There's no race against the clock, nobody to report to, nobody relying on my results -- just me and my idea.
At any rate, I'll tweak the "done-perfect" lever more towards done.
Whether you think there is a race against the clock or not, time passes regardless; your mind changes, your body ages, technologies change, people change. By the time you're done with your "perfect" creation, it will be too late to matter.
Until the last few years, I think I was of the mentality that would follow (C) or (B), but have recently switched to (A) -- just ship it and fix whatever's not perfect later; the opportunity cost of not shipping is too high.
Facebook's motto reminds me of something from a Holman slidedeck: "Nothing great was ever not shipped."
Always keep the end game in mind and put in place a process where you are consistently asking yourself how what you are doing is getting you there. If you decide to go down the rabbit hole anyway, you have to admit to yourself that you are procrastinating rather than solving problems. This works for me, and I am 2-3x more productive. As soon as I get distracted, like right now, I know that I am wasting time writing a post on HN rather than attacking my goal.
Gauge the impact the diversion will have to your core competency. If the actual benefits and time estimates are overwhelmingly positive, do it. If it's anything less, or is just to address a nearly non-existant bullshit edge-case, make note of the thought and move on.
Don't sabotage the larger vision for short-term gratification.
The sad thing is, this example, while ludicrous, is perfectly justifiable, at certain scales. If that town is, say a big one on the East coast, your two hours (or even half a day, or whole day!) are little loss compared to the savings that will occur from the analysis. Hell, your hypothetical "curious person" would be well worth the investment of a six figure salary for the city!
That being said, I guess it depends on what results these random maunderings produce, and it's probably a crapshoot as to whether they are more often than not to result in favorable outcomes. Also, I like your example (obviously :)
Unless it's a hobby or an intellectual exercise, a brilliant solution without a customer/market is a waste of time at all scales, destined to moulder.
Chances are, you'd approach Town Hall with the solution to all their pothole problems and they would ignore it. To make it worthwhile, you'd either have to have earlier signed a contract with Town Hall to figure out their potholes (ie. have a customer) or develop a business based on your solution (ie. develop a market).
By now it's 9:30pm. It's dark and cold. You realize what your original purpose was: Dinner with friends. That was two hours ago. You missed dinner, but hey, you got some satisfaction.
The above was more of an analogy about Yak Shaving than Rabbit Hole Syndrome, but there are parallels. Like you, I used to be obsessive about details and solving subproblems. I used to come home from work and work on my own side projects for similar reasons as your own.
But then an advisor said something like, "You need to focus. If you want this thing of yours to succeed, you have to focus on making it succeed. Nothing else should matter." So I stopped my side projects and I became so effective at building our product that my employees wondered if I ever slept.
Stay on target. Make it to dinner.