Sometimes the result is wrong, or it's not as big or as general as claimed. Or maybe the provided instructions are insufficient to replicate the work. But sometimes the attempt to replicate a result fails, because the person doing it does not understand the topic well enough.
Maybe they are just doing the wrong things, because their general understanding of the situation is incorrect. Maybe they fail to follow the instructions correctly, because they have subtle misunderstandings. Or maybe they are trying to replicate the result with data they consider similar, but which is actually different in an important way.
The last one is often a particularly difficult situation to resolve. If you understand the topic well enough, you may be able to figure out how the data is different and what should be changed to replicate the result. But that requires access to the data. Very often, one side has the data and another side the understanding, but neither side has both.
Then there is the question of time. Very often, the person trying to replicate the result has a deadline. If they haven't succeeded by then, they will abandon the attempt and move on. But the deadline may be so tight that the authors can't be reasonably expected to figure out the situation by then. Maybe if there is a simple answer, the authors can be expected to provide it. But if the issue looks complex, it may take months before they have sufficient time to investigate it. Or if the initial request is badly worded or shows a lack of understanding, it may not be worth dealing with. (Consider all the bad bug reports and support requests you have seen.)
I definitely think all these are important, even if in different ways. For the subtle (and even not so subtle) misunderstandings it matters who misunderstands. For the most part, I don't think we should concern ourselves with non-experts. We do need science communicators, but this is a different job (I'm quite annoyed at those on HN who critique arxiv papers for being too complex while admitting they aren't researchers themselves). We write papers to communicate to peers, not the public. If we were to write to the latter each publication would have to be prepended by several textbooks worth of material. But if it is another expert misunderstanding, then I think there's something quite valuable there. IFF the other expert is acting in good faith (i.e. they are doing more than a quick read and actually taking their time with the work) then I think it highlights ambiguity. I think the best way to approach this is distinguish by how prolific the misunderstanding is. If it is uncommon, well... we're human and no matter how smart you are you'll produce mountains of evidence to the contrary (we all do stupid shit). But if the misunderstanding is prolific then we can be certain that ambiguity exists, and it is worth resolving. I've seen exactly what you've seen as well as misunderstandings leading to discoveries. Sometimes our idiocracy can be helpful lol.
But in any case, I don't know how we figure out which category of failures it is without it being published. If no one else reads it it substantially reduces the odds of finding the problem.
FWIW, I'm highly in favor of a low bar to publishing. The goal of publishing is to communicate to our peers. I'm not sure why we get so fixated on these things like journal prestige. That's missing the point. My bar is: 1) it is not obviously wrong, 2) it is not plagiarized (obviously or not), 3) it is useful to someone. We do need some filters, but there's already natural filters beyond the journals and conferences. I mean we're all frequently reading "preprints" already, right? I think one of the biggest mistakes we make is conflate publication with correctness. We can't prove correctness anywhere, science is more about the process of elimination. It's silly to think that the review process could provide correctness. It can (imperfectly) invalidate works, but not validate them. It isn't just the public that seems to have this misunderstanding...
Things are easier when you are writing to your peers within an established academic field. But all too often, the target audience includes people in neighboring fields. Then it can easily be that most people trying to replicate the work are non-experts.
For example, most of my work is in algorithmic bioinformatics, which is a small field. Computer scientists developing similar methods may want to replicate my work, but they often lack the practical familiarity with bioinformatics. Bioinformaticians trying to be early adopters may also try to replicate the work, but they are often not familiar with the theoretical aspects. Such a variety of backgrounds can be a fertile ground for misunderstandings.
Sure. You can't write to everyone and there's tradeoffs to broadening your audience. But I'm also not sure what your point is. That people are arrogant? Such variety of backgrounds can also be fertile ground for collaboration. Something that should happen more often
Top journals are not inherently prestigious. They are prestigious because they try to publish only the most interesting and most significant results. If they started publishing successful replication studies, they would lose prestige, and more interesting journals would eventually rise to the top. (Replication studies that fail to replicate a major result in a spectacular way are another matter.)
"Having taste" is mostly about predicting the future. Which problems are worth studying, which problems you are capable of solving, and which solutions turn out to be important, in retrospect. If there was an actionable way of developing taste in something, the activity itself would probably be so predictable that it would not be a particularly good research topic.
Taste is mostly about having a good intuition on the topics where your intuition is worth following. It tends to develop with experience. But if you want to develop the kind of taste that helps picking good research topics, you need the right kind of experience for that field of research. Experience that turns out to be of the right kind, in retrospect. If your experiences and interests align (again in retrospect), you will probably develop a good taste for research problems in your field of interest. But that requires some amount of luck, in addition to everything else.
That seems even less actionable, and somewhat misaligned with the OP article. “Taste” implies an ability to distinguish between a good example and a bad one. If it’s only recognizable in retrospect then it’s just another name for survivorship bias.
If you need actionable guidelines, you may not be the right person to do research. At least not now.
Research is all about studying topics of uncertain value. You have to commit to a project long before you can say if it's actually worth doing.
Taste comes with deliberate effort and experience. It doesn't tell you that a topic is definitely worth studying, but it increases the likelihood that you will guess right.
What is the point of writing the prescription to “have good taste” then?
Either the reader already has it, in which case there’s no point in being told that. Or the reader doesn’t, in which case you have declared that good taste cannot be taught.
Perhaps the author’s next article should be How to win the lottery: be lucky which is just about as actionable.
It's helpful to tell people that they are in uncharted territory and can't rely on running on autopilot even if you don't have a new map to give them. Whether they can make their way or not is unclear, but the first step is just making sure they understand that they're now in a place where they need to make their own way and can't fully rely on existing maps. Otherwise they might not even realize they need to start asking "am I doing the right thing right now" by themselves.
You can cultivate good taste by intentionally taking in a lot of information about what's in the field, and what you like and what you don't like about it. This could be commenting on elements of film, fashion, photography, but it can also be having a sense of what you like to see stylistically in a contract, in a framework, or in corporate culture.
I recall reading an interview about a legendary developer, and the majority of the interview was not focused on his coding decisions or the structures he built, but it was about a notebook that he kept with voluminous notes about what was good and what wasn't. That notebook is a materialized version of 'taste', and it's certainly something almost anyone could put together with enough effort and time.
> But if I had to summarize it in one sentence, it would be that taste comes from practicing the skill of research, keeping your focus always on identifying what works and what doesn't.
Instead of following general guidelines, focus on figuring out what works and what doesn't in each specific situation. Keep doing that for many years, and your taste will develop. Remember that you are training your intuition, not developing a set of exact rules.
Eh. I think my point is that the OP is presented as a “how to” (literally: “how to do important research”) and then it immediately dodges the question by saying “have good taste”. That does not help anyone do important research or improve the quality of the research they do; it’s a cop out.
If I wrote about “how to paint great art” or “how to cook great meals” or “how to build great things” then it would be silly to say “have good taste”—even if that’s part of the answer. It won’t help anyone else to improve in any of those endeavors.
It's the jargon, I think. PL research is in an awkward position, where the jargon is not shared with the much wider community of people using programming languages daily. From the other side, it looks like there is a small body of theoreticians using impenetrable language for discussing topics I'm supposed to be familiar with, because they are a core part of my day job. It's much easier to accept jargon, when it's used in a clearly separate field.
Some of the terminology is just unfortunate. For example, I have an intuitive understanding of what a type means. The meaning used in PL theory is somehow wider, but I don't really understand how.
And then there is my pet peeve: side effect. Those should be effects instead, because they largely define the observable behavior of the program. Computation, on the other hand, is a side effect, to the extent it doesn't affect the observable behavior.
But then PL theory is using "effect" for something completely different. I don't know what exactly, but clearly not something I would consider an effect.
Geekbench 6 Multi-Core is fundamentally a single-task benchmark. It measures performance in workloads, where the user is not running anything significant in the background. If you are a developer who wants to continue using the computer while compiling a large project in the background, Geekbench results are not particularly informative for you.
I've personally found that Apple's Pro/Max chips have already too many CPU cores for Geekbench.
A new term was needed some decades ago. "man" titles have not been politically correct for a while, "member" sounds awkward and bureaucratic. In some other languages, "soldier" can be used for all military personnel, while English ended up with a more narrow meaning.
"Awkward and bureacratic" is literally the point of naming conventions commonly adopted by democracies. Titles like "president" or "prime minister", departments like "Department of Defense", referring to government employees as "civil servants", etc. are all intentional measures meant to strip away the prestige and egotism associated with positions of authority in an effort to avoid it going to people's heads, and to remind them that they are meant to serve the good of the public that pays for their existence rather than ruling over them.
"Service member" is awkward, because it has too many syllables. People won't use it when shorter alternatives are available. And it's bureaucratic because it's unspecific. It doesn't tell anything the service those people are members of, and it doesn't tell what kind of work they do.
The peer review process in CS is pretty bad, especially in conferences with a single round of reviews. When you combine this with high rejection rates, peer review becomes more about finding excuses to reject a submission than about trying to improve it.
Conferences also don't work that well as publication venues, as they often require that one of the authors must attend the conference physically. And it's not as much about money than about visa policies and travel restrictions. Even in the 2000s and 2010s, when international travel was easy, people from non-Western countries could often not get visas to attend conferences. And today the situation is much worse.
I've been to three international conferences in the past year. One was held in Europe. People from Russia and Israel had to present remotely, the former due to an ongoing war and the latter due to an unexpected war. Another was in the US, and there were fewer Europeans than usual, as many were not willing to take the risk. And the third one was in Japan. People from China could not attend due to increased tensions. People from Israel were there, but they were worrying if they would make it home before the next war. (They made it, and the war started a day or two later.)
There are hundreds of reputable research universities around the world. Top-5 departments can't meaningfully change the culture of a field on their own. Top-100 perhaps could, but the coordination problem is much bigger on that level.
That was years ago. Today's armies have plenty of cheap anti-air weapons deployed everywhere, and they manage to intercept the vast majority of long-distance drones cost-effectively.
The Gulf War era world, where a small professional force using expensive high-tech gadgets could win a war without the enemy being able to fight back, is gone. Today you still need the small high-tech force to win, but you also need a larger cost-effective force for defense.
Didn't we just see something like 15% of Russia's nuclear bombers wiped out from a drone attack last year?
A guy got close enough to shoot at and almost kill a Presidential candidate during the last election -- imagine how different things would have been if he had just used something like this[0] instead.
The first time someone attacks an outdoor music festival with drone strikes like that asshole who shot up that concert in Vegas is going to change society for the worse. :-/
That particular attack against Russian bombers was done by undercover saboteurs working deep inside enemy territory. The drones being discussed here are essentially cheap cruise missiles. For example, you have a ~200 kg drone carrying a ~50 kg warhead that is launched using a rocket booster and capable of flying 1000+ km. In Ukraine and Russia, 80-90% of such drones are routinely intercepted by cost-effective air defenses.
>Today's armies ... manage to intercept the vast majority of long-distance drones cost-effectively.
Army singular. There is one army on the planed doing that right now and trump hates its leader thus didnt even ask for help with eventual drone defense.
The solutions already exist and have been proven on the battlefield. Peacetime military forces are just slow to adapt, as there are no real incentives to adapt quickly.
Drones such as the Shahed are little more than cheap mediocre cruise missiles. Because they are cheap, the enemy can launch them in large numbers. You counter them by detecting them early and then using plenty of cheap mediocre anti-aircraft weapons. Mostly guns and interceptor drones (=cheap mediocre anti-aircraft missiles).
Sometimes the result is wrong, or it's not as big or as general as claimed. Or maybe the provided instructions are insufficient to replicate the work. But sometimes the attempt to replicate a result fails, because the person doing it does not understand the topic well enough.
Maybe they are just doing the wrong things, because their general understanding of the situation is incorrect. Maybe they fail to follow the instructions correctly, because they have subtle misunderstandings. Or maybe they are trying to replicate the result with data they consider similar, but which is actually different in an important way.
The last one is often a particularly difficult situation to resolve. If you understand the topic well enough, you may be able to figure out how the data is different and what should be changed to replicate the result. But that requires access to the data. Very often, one side has the data and another side the understanding, but neither side has both.
Then there is the question of time. Very often, the person trying to replicate the result has a deadline. If they haven't succeeded by then, they will abandon the attempt and move on. But the deadline may be so tight that the authors can't be reasonably expected to figure out the situation by then. Maybe if there is a simple answer, the authors can be expected to provide it. But if the issue looks complex, it may take months before they have sufficient time to investigate it. Or if the initial request is badly worded or shows a lack of understanding, it may not be worth dealing with. (Consider all the bad bug reports and support requests you have seen.)
reply