IDEA BOLT: What if one creates a central website that offers free storage and search capabilities and ask people to upload their dissertations and theses?

What do you think about that idea?

The hard part is convincing people to upload their work. Even in physics the use of the arXiv is not uniform. In some areas, almost every published and unpublished paper is posted, while in other areas of physics hardly any papers are. For example, compare the category Quantum Physics with Space Physics.

Usable? No.

I'm not sure if theres some piece of information thats trying to be portrayed here (other than quantity of dissertations per major) but as a browser, it's pretty useless. It employs mystery meat nav heavily, mostly because you have to hover over a dissertation to view any info about it. Imagine finding a paper you liked, and then going back 15 minutes later to try and find it.

If there is some data thats trying to be shown its not clear to me what it is. Some of the inner circles are in columns which seem to indicate correlation, but I cant figure out if thats accurate or not. For instance, if I click on philosphy, I see a dot in the direction of Food Research, and when I hover over that dot I see a thesis about "Practical reasoning and the varieties of agency".

What is trying to be portrayed here?

How dissertations in an area overlap with other areas.

Calling it "Stanford Phd dissertation browser" sets up the wrong expectations, but it's an interesting infographic.

Perfect example of bad data viz.

It's automatically clustering topics radially based on proximity to each other and to other disciplines.

Click on CS. Then click on CS again (now in the center). CS + Music is out by the Music field. CS + EE is all in a column pointed to the EE field.

It's a tool for discovery.

If I went looking for list of articles, linking CS + Music, this doesn't help me. I'd want to click CS, click music, and then receive a list of articles matching that correlation. Instead I have to hover over each dot, and scrub through years to even see titles for things. Sure, I get the "excitement" of not knowing what I'm looking for, but that is not a usable tool for discovering articles.

Think of this as an experiment in exploring a document collection at a higher level than search. Specifically, what you're seeing is Stanford's dissertations through the lens of a text model that tries to distill high-level patterns in the data. It doesn't always succeed, but it often hits the mark. There are plenty of ways that the visualization and the underlying text model could be improved.

For the curious, I'll tell you a bit more on how the numbers are computed: we build a unigram language model of the contents of every Stanford department based on their dissertations. Then, we posit that every dissertation comes from a mixture of those department models (using a supervised topic model, Labeled LDA). This lets us infer, for every dissertation, a weighted mixture of departments that best characterizes that abstract. So, say, dissertation X is 60% computer science, 20% physics, and so on. These scores are aggregated to compute the average similarities between departments, and are sliced to give the view over time.

So what you're looking at is, essentially, a visualization of word overlap between departments measured by letting the dissertations in one department borrow from words from another department. Which departments borrow the most words from which others?

When you zoom in two-levels (click on a department twice), individual dissertations are plotted on a line between each dissertation's home department and it's next highest scoring department. So the relative position of two dissertations near each other is not meaningful unless they are on the same radial line. Dissertations from other departments that have a high score for the central, focused department, are also shown.

For instance, take a look at Computer Science in 2005. You'll see three dissertations along the radial line to Linguistics - those are the three students that graduated from the Stanford NLP group that year. There are plenty of other places you find similar things that work, and also places where things don't work as nicely as you'd expect.

The visualization Jason built was really interesting from the text modeling perspective, because it let us experiment with many model variations (lda, tf-idf, etc etc) to see how well each matched our intuitions. This model, though still wanting, was by far the best. Good enough, even, for us to put online for the world to play with, and for hacker news to pick apart ;)

Picked it up from the NLP Lab twitter stream. BTW, thanks for the awesome NLP tool chain! :)

The browser showed two overlaps: "Low-Power dynamic amplifiers for pipelined A/D conversion" and "Precision clock synthesis using direct modulation of front end multiplexers/demultiplexers in high speed serial link transceivers"

The first of these mentioned "cell phones" in the abstract. There was no evidence of any cell biology link in the second.

The visualization may be interesting, but I'm not so confident in the quality of the data.

For instance: Comp Sci -> Ethics: "Designing interactions that combine pen, paper, and computer". Comp Sci -> Radiology: "Securing untrustworthy software using information flow control" Comp Sci 98 -> Physiology: "Consistent overhead byte stuffing"

Could be a heck of a lot better. Especially given the long almost-locked-up pauses, and the inability to keep a block of text up when you move the mouse away. All the little things add up, making me doubt the creator used it themselves at all, aside from making sure it functioned.

I'm guessing they're using something like a KL divergence between the distributions over words (smoothed?) given each topic, but that could be way off.