Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Why HN is slow lately
270 points by pg on March 11, 2009 | hide | past | favorite | 122 comments
We've had a huge spike in traffic lately, from roughly 24k daily uniques to 33k. This is a result of being mentioned on more mainstream sites. I hope this spike will subside, like past ones have. In the meantime I may temporarily hack a few things to make the site faster, like putting fewer results on threads pages.

You can help the spike subside by making HN look extra boring. For the next couple days it would be better to have posts about the innards of Erlang than women who create sites to get hired by Twitter.



Ok, ok, enough Erlang submissions. You guys are like the crowdsourced version of one of those troublesome overliteral genies. I meant more that it would be better not to submit and upvote the fluffier type of link. Without those we'll be fine.


This place was starting to look like λchan, thanks for putting the cork back on it.


Can that request to skim the fluffy links be permanent, please? :)

[edit]


I'm in total agreement. In the effort to become boring, Hacker News just became more interesting than it has been in the last couple weeks (silly links to Erlang Shen aside).

I come here for this sort of stuff, or for thoughts on business, not to hear what computer nerds think about macroeconomics, politics, and flirting. If I wanted that stuff, I know Reddit's address.


While I agree with the sentiment, I don't like the example used.

I found the twittershouldhireme blog a good lesson: KNOW your audience. That girl was surely noticed by Twitter execs and she did something simple that no one else in her position thought to do.

Erlang posts also belong here and politics usually does not, but don't pick on the twittershouldhireme girl. She's fantastic.


While I didn't find the twittershouldhireme lady very interesting, I don't know why you're complaining that you don't like my using it as an example, since I didn't.

Was this reply was meant for another comment?


PG said that the twittergirl had little value and you agreed, nothing personal, just saying I happen to have found value in it.

What I will never understand about this community:

I get negative Karma for defending an article that was clearly appreciated by some and this guy gets 10+ points for saying "what are you complaining about?" (something of zero value.)


Actually, I said I agreed to the post I was replying to, not to PG's post. Sorry you think it had zero value.

For what it's worth, I never said nor meant to imply that I thought the website of the lady who wanted a job with Twitter isn't suitable for HN, though I don't find it to be exemplary. It's a whole lot more on target than (as previously stated) articles on macroeconomics, politics, or flirting.

None of this is to say that I actually disagree with your point on the silly nature of what garners karma and what collects down-mods, but that is a topic for another time.


This community is confusing me more than you...

When it comes to up-voting and down-voting... you will see all of these crazy things!


holy shit, i stepped away for like 30 minutes and the entire front page is Erlang. I just submitted the Innards of Erlang article as a response to PG's post/an inside lol for the community (the article is a decent primer though). Almost the entire front page is a bit much guys...


I chuckled when your story came up, and then I laughed out loud when, like you, I came back 20 minutes later and the entire front page is Erlang stories.

Many Lolz.

In the next few years we'll begin saying, "You remember when pg told us to submit Erlang stories?" "Oh yeah, good times."


Or in a few years it could be - remember when HN was not only about Erlang?


Boring is fine. All Erlang, all the time is a problem


As of 22:12 European time, it's the first page and the second page. Olé!


:) What's European time? I imagine we (you and me) share Central European Time (CET) but ET?


I've been driven to the new page for the first time.


Ooooh, HackerNews meme. You realize how tempting it is to submit this to TechCrunch, etc., don't you?! TC Linkbait Headline: "Elitist Hacker News shuns the rest of us by pretending to be about Erlang! Oh, and somehow this relates to Twitter!1!!!1!!"


I don't think it's memeworthy. Too much of a "you had to be there" thing.

I just can't think of a situation where it would be funny to say "quick everybody lets talk about Erlang."


Quick everybody lets talk about Erlang!


I checked the frontpage and had to look at the calendar: for a brief while I thought "Wow, March really flew by. Is it April already?"

Btw, I think you just got yourself an idea for April fools: accept only links about Java or Visual Basic.


Even better: esoteric languages. Maybe someone will write the first ever HTTP server in brainfuck?


No way, man, write yourself a web framework: http://nanoweb.si.kz/manual/mod_bsp.html


Hey, this was my first belly laugh at work in quite a while. The downside is that I will now have to skim every boring title to make sure I havent missed a gem.


I too suffered from this... both the laugh and then the concern of missing something good.


pg the Problem isn't that fluffier stories are up its that people are hungry for more of everything. I read Hacker News, Reddit, and even Digg in that order mostly every day and if there was another site that provided interesting stories and or decent comments I'd add it. It is the pursuit of information that's driving me mad.


You could probably do what Stephen Colbert does and direct everyone to vote for you in various contests around the net. The Paul Graham wing of the International Space Station perhaps?


Seeing that many Erlang links is truly entertaining - a true phenomenon


I have missed this side of HN. Feels like christmas.

HN should have esoteric/alternative articles like that at all times.


I've been laughing so hard at the extreme geekiness of these articles, for some reason the video on youtube was of the funniest things I've seen in a long time :D


What were the mainstream publications that mentioned HN?


It'd be interesting to see the stats on how much it affects the site's traffic!


i call collusion. somewhere there some back channel irc rolling...


Browsing from blackberry I thought I was on the wrong page. Lulz


i love hacker news


phew :D so glad you commented this. I think some people were taking it too literally..... there is a limit ;)


pg, I have wanted to read about your thoughts on the Ruby programming language for a long time now, I have simply not wanted to impose or be a nuisance by clamoring for an essay . . . that would be absolutely great I think, I have gathered a few quotes from you about Ruby, and they are all positive (it is one of the reasons I developed an interest in the language) and I would love to read more: positive, negative, and neutral.

Thank you pg.


Until I read this, I wondered about the sudden popularity of Erlang


Same here. I couldn't fathom why Erlang would be that hot of a topic all of the sudden.


Erlang, the next Ruby On Rails ?


Damn, and here I was about to throw out all of that code I'd written and to jump on the Erlang bandwagon lest I be left behind.


You can help the spike subside by making HN look extra boring

I wouldn't mind if, in the service of this cause, the editors killed more fluffy stories.


Or just give high-karma users the ability to down-vote submissions that have more than 30 points.

The 30 points limit makes sure that stories don't get killed for the wrong reasons. In fact they don't get killed at all, they just get moved down the page a little faster than they otherwise would.


This sounds like a really really good idea.


Sorry guys, I'm here to stay. HN's so much more intelligent than Digg, Reddit, etc. Even the tech parts of those sites don't manage to filter out the junk as well. I hope it stays this way, though...

Oh, and I actually found out about the site from Coding Horror (http://www.codinghorror.com/blog/archives/001236.html), though I did read the Stack Overflow article as well (via Reddit: http://www.reddit.com/r/programming/comments/8368a/the_value...).


Ever considered just shutting the door to new registrations at peak times to throttle growth? or at least only allowing new members that have been invited from existing members?

I've seen some really whacky comments the last week or so.


An invite-only strategy might actually backfire and make the site even hotter (see gmail, etc), although it will be interesting if the feature only turns on at certain times/due to certain conditions.


I'd say the most irritating could be people who:

  * See link to HN somewhere, follow it
  * Sign up, post some lame comments, never come back.
So my thought was just adding some sort of barrier to sign up, to deter some of the people who only want to sign up so they can post a pun or some other non-HN worthy comment.

(NB. This is only a hypothesis, I don't know how many people sign up post a few lame comments and never return).


A 1 day comment ban after sign-up would be more effective I think.


I thought about this a while back - a big negative to it would be that if someone sees traffic coming from HN and comes to sign up and answer questions and feedback in the comments. I've seen lots of great first comments after signing up by people who had their content reached.

A similar idea would a hard limit on comments for new people for the first day? 3 or 5 perhaps?


We do a 5 day comment ban at http://forum.doom9.org/ works very well. Spam is a rarity, and the talk is quite good. Example: http://forum.doom9.org/showthread.php?t=145561


Perhaps, but that might affect lurkers (perhaps even long term lurkers) who finally decide they have something useful to say and want to say it - now.

I tend to mostly lurk, but I did find myself in that situation a couple weeks ago.


Doom9, another forum I visit, does this for 5 days to force new users to read the FAQs and guides before asking their question.

It works well, IMO.


Would a delay between when you sign up and when you can do anything with the account help?


That might work.Sure it will probably make the site hotter, but as long as potential users are evaluated, it probably won't hurt the quality of the site.

If we shut the door to new registrations, we definitely shouldn't make an invite only available from a member. I'm sure I would have had trouble getting an invite that way.

What might work is deciding on a growth rate that can be handled by the site and the community, and then limiting the number of sign-ups each month, or even day. We could just use a list of email addresses for new users and send out X number of invitations a day.

And we could also do more discourage those wacky comments. We could have new users do a short test, something trivial that requires them to have actually read the user guidelines. Maybe a multiple choice quiz where they have to distinguish between a trollish comment and an acceptable comment. Also we could start them out with a probationary account with stricter rules. Maybe if their account is newer than 5 days, and they have less than -8 karma, they get banned, or forced to read the site guidelines again before they posted more comments.

Though if the site does implement anything like an invite system it's going to be hard. Some people are going to accuse the site of being elitist. So it would have to be communicated that any rules are solely for all comments civil and professional, not because we are trying to exclude anyone. Hey, I like some of the pun threads on reddit, and trolls on yahoo answers, but I think this site would get ruined if even a few trollish comments were allowed.


Or include some fairly simple auto-generated questions.

"At what point does y = 2e^(2x) cross the line x = 2, accurate to 2 decimal points? y = __.__ x = __.__" Extremely simple for a hacker, daunting for the general public. (answer: tinyurl.com/b95cdw rounded up with x = 2.00, of course)


That might backfire. I might think less of a site that thought doing trivial math problems was a sign of my level of intelligence.

And making it harder might also be bad. If they are too hard, you'll get people who are just signing up for the entrance test. Just think about how much press google's puzzles on their billboards got. And somebody will of course put the answers on the internet.


You might also get the occasional serious hacker who is just retarded at math (like me!)


I would go further, and venture that 'serious hackers who will make good HN citizens' and 'people for whom the above problem is trivial' are two different sets (though the intersection of those sets might be significant).


Or even some questions to show they've read the guidelines @ http://ycombinator.com/newsguidelines.html


Show new users a fake front page; ones that upvote the content you don't want on the site get dumped.

(I'm only partly joking)


Seems like submitting boring Erlang stories immediately turned into a meme.

First one on HN?


First one I remember, anyway.

As much as I like being pompous and saying HN should try not to go into decline, this one I'm happy about. It was all a joke on PG, it had a terrific effect on the stories here in a way that was instantly noticeable here, and it was actually pretty funny.

Now we just make sure this fades quickly enough that a month from now nobody thinks they're "trendy" for mentioning Erlang Week.


HN's first meme : http://news.ycombinator.com/item?id=98435 [421 days ago]


I'm off to upmod some boring stories.


upmod the what web frameworks are missing. snoooze.


Still I'm curious, what's the underlying problem?

33k/day doesn't sound like a lot of traffic, honestly. But also I can't imagine lisp being that slow either. Maybe add some caching?


If I remember correctly, Arc is an ongoing experiment in language design, not high-performance implementation.


And it's running on top of PLT Scheme, which adds another layer of interpretation.


And it's running as a single process on a single server without a database storing everything on the file system. That it's doing the traffic it is I find quite impressive.


In 2 GB of memory.


You know I'd love to see a writeup of the issues and challenges you've faced with this persistence strategy as the site grew. You know, more details of what you decided to lazy load, why, and how, and the impact it had.

Technically speaking, I find hacker news persistent strategy one of the most interesting things about the implementation.

I use a similar strategy for my blog and just playing with larger datasets I've certainly run into hard limits on what seems to be acceptable.


Upstream. Both ways. In the snow.


Extrapolating from the memory-growth graph, how much time would a 16GB ram machine buy you?


I'd have thought that most of HN's traffic is on the front page, and it's associated comments. I could well be wrong - But given this I'd have thought 2GB was plenty, and wasn't the bottleneck.


Ok maybe not: http://news.ycombinator.com/item?id=512801


Are there any articles on the "HN storing everything on the file system"?

I am quite interested in the architecture, because I found this approach nice for some sites I was building. They were not very large though.


If you're keen, you can get the source code and take a look: http://www.arclanguage.com/install

File system stores aren't that uncommon (aside from the VSAM days, even in RDBMS times+) - it's a common approach for Wiki implementations too.


Also, unless your DB tables are in-memory tables, you are using a very fancy interface to a file system store (that is usually in a nontrivial format.)


> without a database

Why?

I can understand not using a database if this was improving the performance.


PLT Scheme is not "interpreting" code. More than that, Arc adds a bunch of huge overheads that could make things substantially faster. I had some patches lying around that made things around 4-5 times faster (including the news server).


but it's calling all those ar_funcall2 or whatever ... not that much indirection really.


33k users per day is piddly little bits. I think that a platform change is called for, the current recipe is clearly no longer up to the task. Which is a pity, because the lisp code really is quite elegant. It's just that the web and high performance and lisp are not usually used in a single sentence (other than this one...).


It would be interesting to know what is the likely reason for each traffic spike. Perhaps you could have a page that shows this information in a way similar to the Google finance news items:

http://www.google.com/finance?q=NASDAQ:GOOG


I'd wager it is due to Stack Overflow's recent article.


Done.

http://news.ycombinator.com/item?id=512178


Do you think this is related to the multitude of company launches in YCW09 in preparation for demo day? That's a lot of TechCrunch posts.


Man do I love Erlang. I rushed over to HN as soon as I heard a new Erlang community had popped up!


> You can help the spike subside by making HN look extra boring

What, submitting a post where I laboriously implemented Surreal Numbers step by step wasn't boring enough?


No, it was a great read :-) I like what you're doing with http://github.com/raganwald/homoiconic


Can we submit posts about women who use the innards of Erlang to create sites to get hired by Twitter?


This question might sound ridiculous to majority of people commenting here, but why are you making the site boring instead of trying to improve its performance (putting additional hardware, ditching Arc, whatever)?


From what I can tell, correct me if I am wrong, part of the purpose of this site is to operate as a testing grounds for Arc. The focus is on making the language better. So rather then improving performance by providing better hardware, it becomes a matter of improving the site through code. Either by changing the code running the site, or by improving the language itself. But given that this is more of a side project, the resources needed to improve the site are limited. Given the unusual spike in traffic, rather then using limited resources to fix it, or sacrifice the motivation to improve the code/language, the boring article approach is being used.


Yes, that. And the site is not trying to be the most popular social news site on the planet. It aims to serve a specific niche that doesn't yet require more processing power.


I come to the site daily, but haven't really noticed anything being really all that slow. Could just be timing.

One thing I like doing is following the Hacker News twitter acct: http://twitter.com/newsycombinator/

It tweets popular stories and keeps me from visiting the site all that often (unless its a comment link).

Its not a fix, but if anyone out there didn't know about it you should follow. It'll keep you from visiting the site every 5 minutes to see if anything new has been posted.


I see people took the Erlang suggestion seriously :P

But I wonder, isn't it a bit silly.. I mean, new people will read this post and find out your evil plans to kic.. err, gracefully lead them out :)


if they were going to stick around anyway, then its worth it for the comedy factor alone.


I kind of love this, despite having little or no interest in Erlang. It's like that sequence from _The Wiz_ where Richard Pryor keeps declaring different colors to be in style and everyone shows up wearing them. pg should come along every so often - once a month maybe - and just announce that all submissions should be about an obscure language.


I'm curious, where does a site like HN have its bottleneck: SQL accesses, server bandwidth, CPU or memory consumption by PLT?


There is no database (see the other comments in this thread)


More people visiting a site does not make it bad..but more submissions and vote-ups could be the reason for the dilution. Just make that process as hard as possible. How hard can it be to make a user-unfriendly html-hiendish registration page, eh ? :)


The webserver is written in Arc, right?

I wonder how hard it would be to inspect the referrer and if an abnormally large influx of users start coming from a specific referrer, start redirecting to programming.reddit.com or stackoverflow or something.


Please, don't start Ruby vs. Ergland or Erlang doesn't scale... :)


Great. Now you've actually gotten me interested in knowing what Erlang is all about... and I've already got enough side projects to do in my spare time.


I can't believe pg implied that Erlang is boring, haha.


So, HN is now like the rockstar who puts on a baseball cap and shades and thinks he's out at the WalMart incognito... Ahhh, I get it now. Shhhh ;-)


If you're having such a hard time keeping up with the traffic, why not offload the burden of the site to some community members?

Or what about throwing HN up on EC2?


I almost thought the site was attacked by a group of guys that just trying to vote up all Erlang related sites. Now I know the reason.


I had no idea what Erlang was until a few minutes ago. At this rate I might soon be an expert. naa, feed me more Erlang.


I like the current version of HN and it's incredibly fast, so why change it?

Congrats for the 33K unique a day!


I feel the urge to twitter/blog that second paragraph, because it's funny. Must...refrain...


I have never laughed this hard when checking out the HN front page everyday morning.


I was wondering what all those stories about Erlang were doing on the frontpage :-)


Are the new viewers gone yet?


I just love the energy of the community - Erlang everywhere!


This is brilliant social engineering, and rather amusing!


thats fine but at least let us do this:

http://news.ycombinator.com/news/!erlang


actually if you click on that link now the result is kinda funny


Erlang clogging the pipes...


Did you add the word "many" on the saved list, instead of the number of saved stories, for performance tuning?

women who create sites to get hired by Twitter Well, I thought of flagging this story, but I saw everyone is interested in discussing it!

Anyway, HN is really slow since the last week, and it's being down for 4 to 5 times daily - for me. But when I ping 67.15.104.17 , it works fine.

Maybe killing the spam automatically should help? I am not sure.


Hooray!

I vote for a permanent ban on fluffy links.

The Fluffy category ought to include:

- MBA related stuff

- Suit related stuff

- Content free linkbait articles that mention the words "hacker" or "hackers" in the title, like the absolute jewel we had yesterday, where one of the supposed must have skills for a hacker is a high wpm typing speed (I really do think that - in the near future - humans will be able to train monkeys to type)

- Anything that would appeal to the majority of the "social networking" crowd

- Anything that offers anything remotely related to the "program software (or any other worthwhile computer activity) with this product, no brain, education, or hard work required!" crack dream

- Did I mention MBA/Suit related stuff? :)

- Wannabe MBA fodder

- Last but not least, I mean (sigh), how can I say this without coming off negatively? well, can't be said, but I will do it. I really do think that the best Heuristic is the Room Temperature Test: anything that would appeal to people whose IQ hovers even close to room temperature ought to be kept off the site period - this includes all of the above, particularly the MBA/Suit Neanderthal stuff.


HN is slow lately because PG is a dreamer and too much tied to personal and insignificant taste details (which never combine with a greater user base):

HN should have been written in CL.

;)


I'm all set with Erlang now.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: