Between this and the "Sunset time and the economic effects of social jetlag: evidence from US time zone borders" paper [0], it seems like the issue is the size of the discontinuous jump in time, not necessarily that we change the clocks. So why not "smear" the DST<=> ST transitions by having four half hour transitions, once each quarter?
> So why not "smear" the DST<=> ST transitions by having four half hour transitions, once each quarter?
Very easy answer: Because it's already painful twice a year, and that would be making it even worse.
That answer is similar to the one for questions like "why do we have wide time zones that are somewhat inaccurate, rather than setting every clock based on the exact position of that clock?".
I’d be okay with every day having a different # of seconds. That way we slowly adjust with no discontinuity, but the nominal start time of school/work stays the same.
While this feels would be a disaster for other reasons like: “How many seconds are in an hour?” -> “Depends, no one knows.” … that’s already the case with our existing leap seconds.
(This sounds like kicking the can down the road to me; making the maximum discrepancy a minute could take 50-100 years and then you need a leap-minute or equivalent).
No. But the Earth has sped up a bit, so they're less necessary than at some times in the past. And we're on schedule to change them to be some kind of bigger, yet-undecided adjustment (perhaps a whole minute) before the year 2035 comes to a close.
If we move to leap-minutes, the Earth will do whatever it does, and it is expected that we'll be able to run on atomic time for a period of decades or perhaps even a century before we need to make another adjustment like we've done with leap seconds in the past.
Looking at the linked scoring prompt (resume_evaluation_criteria.jinja) [0], I immediately see several red flags that suggest the output won't be reliable. (I'm developing an LLM intensive application where the stakes are high enough that I need the LLM output to be reasonably correct.)
1. The prompt is trying to get the system to do all of the evaluation steps at once. Instead, the system should break down the task of resume evaluation into its subcomponents and have separate prompts for each component. Like "evaluating open source contributions" should be its own task. Same with "assessing the complexity of software projects on the resume." Fwiw, each of the tasks contained within the prompt is woefully underspecified.
2. The prompt leaves spreads of ~10 points up to the LLM, when it's doubtful that humans are that well calibrated. Take for example:
> SCORING CRITERIA Open Source (0-35 points)
HIGH SCORES (25-35 points):
- Contributions to popular open source projects (1000+ stars)
- Significant contributions to well-known projects
- Google Summer of Code (GSoC) participation
- Substantial community involvement
Are all of these 35-point examples? Is one a 26-point example? If not, what's the difference? If an expert can't reliably make the judgement, the LLM is going to struggle too. One partial fix is to get rid of the ranges and just say all of these are worth 30 points. An additive point scheme would be better...
3. The authors of this prompt have left an incredible number of judgement calls up to the LLM, when that's the very thing you want to minimize. Using the same example as above...
- Are all contributions to open source projects with 1000+ stars equal?
- What counts as a "significant contribution"? Doesn't that imply that the LLM has to know or read through all of the commits in like the last ~6 months at minimum for the project to understand what the given contribution meant to the project? That itself isn't impossible with tool usage, but again, that'd be a separate task.
- What on earth counts as "Substantial community involvement"? Why didn't the prompt authors define this, or at least give a few examples?
Honestly at this point maybe someone should build a tool that scans prompts for adjectives...
4. This sort of thing is just asking for trouble:
> SCORES MUST NEVER DEPEND ON:
Candidate's name, gender, or personal demographic information
Just remove this stuff before you send the rest of the resume to the LLM. Even if you ask it not to, it's not a person, it's a very fancy statistical distribution generator. All of the input (including the name) will affect the distribution that gets generated. (This one is not unlike Andreessen's "don't be a sycophant" prompt.)
5. Obviously this one depends on the LLM in question, but instead of writing things like:
> DO NOT RETURN A RESUME SUMMARY. RETURN ONLY THE SCORING EVALUATION IN THE SPECIFIED JSON FORMAT. Analyze the following resume and provide a JSON response with this EXACT structure (all fields are required):...
The system should utilize the "structured output" option, which guarantees a fixed output format. Also, fwiw, the JSON should force the LLM to pick between categorical options as much as possible. Forced-choice structured output should, at least in theory, cut down on hallucinatory responses and constrain judgement calls.
6. One major thing that's not in the prompt is anything about traceability. This system should be designed so that humans can review the logs and make sure this is working as intended.
7. Another thing that is missing in the file is what I'll call evidence of a theory of coding / coder quality. Most of the examples are designed to have the LLM assess proxies for code quality, not code quality itself. Surely both should be taken into account?
I'm not an expert at evaluating coders. But two pretty basic LLM-answerable thing I would ask is: How well do a candidate's 5 most recent commit messages match the contents of those commits? Do the claimed technical skills on the resume match their GitHub code? (i.e., if they say they know R, is there any evidence of that on their GitHub?)
8. The prompt also seems unaware of what it's asking the LLM to do:
> LIVE DEMO BONUS: Projects with working live demos should receive 10-20% higher scores
This implies that the LLM can use tools, but even then, I'd be pretty wary of its ability to fully execute this part of the prompt without more detailed instructions, examples, and guidance. There are very likely tons of edge cases here.
Would it be correct to say you have experience building LLM based workflows like this? I'm guessing so, given by your critiques and suggestions of better approaches. Can you recommend any books/sites/other for learning these kind of dos and donts?
While I have some experience, I'm afraid I am learning mostly through a) trial and error, and b) using an llm chat interface to double check my thinking.
This being said, here are my top recommendations:
1. Build your system against real targets. Had HackerRank continually tested their prompt against 2-3 real resumes that were scored by hand, I think some of the issues would have immediately popped out. The people who built the prompt thought they could magically skip the hard part of articulating a preferred decision making process by having the LLM do it. But LLMs are much better at scaling a pre-existing decision-making process rather than inventing one, let alone the same one, from scratch, every time.
2. Think about what it would take to get motivated undergraduate interns to do the task from end to end, step by step. That's essentially what your workflow will need.
3. If the LLM can't do a step or sub-task reliably, then it's time to decompose those subtasks into even smaller chunks.
Awesome, thank you! I am starting to do some work building LLM workflows and would like to stand on some giant's shoulders to skip the initial flailing :)
I'd like to offer a counter-point to many of the comments here. While I understand being stymied and frustrated by a product one is paying for...
At the same time, I personally think the tradeoff between "having guardrails" and "some users are unhappy with the product" is well worth it. Think of what would happen if all of us who aren't so well intentioned could exploit Fable in terrible ways. Surely this tradeoff is better than saying "we can't make it perfect, so whoops, we aren't going to have any guardrails at all"? Especially because Anthropic did pretty extensive red-teaming of Mythos & Fable...
They haven't though. There's a long term plan here, and the goal is power and wealth. Short term moves that appear irrational turn out to be rational (from a greed perspective) when you factor in other considerations, like: Use their own AGI to create every software product on Earth and swallow the worlds economy. And we're kindly feeding their systems our codebases, IP and business decision-making so they can do exactly that.
Not a single thing Anthropic has done has been altruistic, and it never will be. It's all smoke and mirrors for the end goal.
Why invent new motives for Anthropic when their real motives are plain and obvious and have been confirmed time and time again by their behavior over the last few years? Their concern is their own power and wealth. Every other conceivable motive is secondary to that.
The "guardrails" are just Anthropic's attempt at building a moat. Guarantee they'll be seeking regulation around AI as well to ensure a form of regulatory capture. Guardrails, in this context, are useless. Anyone who's sufficiently motivated will either get around them, or will just run their own model on their home hardware. There's already tools that one can use to remove the guardrails present in open weight models.
Guardrails against what? Rehashing public wikipedia information?
Execution matters, and they did a trurly horrible job that crippled their product to the point of being useless and a joke. Huge mistakes were made and im sure they regret it already, heads will roll.
Reactions to this are a bit curious. It's a satirical comment on how (presumably) initially well-intentioned younger founder-types get swept up in / by perverse incentives. The implication is that younger people who are still figuring out who they are and coming into their own may be more susceptible to these kinds of incentive traps.
The first section that showcases the fraud that has been committed is something I have no problem with, just as I have no issue with web3isgoinggreat.com. The "at risk" section is based on a mathematical/algorithmic joke. This is explained by the "methodology" section below it, which makes it clear that the equation used to calculate "risk" here is not entirely unlike the Drake equation for the probability of extra-terrestrial life.[1]
Eh, I think selection effects are more prevalent than an earnest good faith actor who got swept up into perverse incentives.
Forbes 30u30 is a clarion call for the most ambitiously Machiavellian among us.
They’re not subject to any different incentives than the rest of us. But they’d certainly have a higher rate of sociopaths and more garden variety Machiavellis than genpop.
I was curious to pin down the definition of Machiavellian:
> Manipulation & Deceit: Using charm, lies, and calculated moves to influence others.
> Lack of Empathy: A cold, detached, and unemotional demeanor that disregards the feelings of others.
> Strategic Long-Term Planning: Unlike impulsive psychopaths, high-Machs are patient, planning, and can delay gratification to ensure success.
> Cynical Worldview: Believing that people are inherently weak, untrustworthy, and that "the ends justify the means".
> Low Affect: Possessing limited emotional experience, often leading to a detached, "puppet-master" role rather than seeking the spotlight.
The only traits that seem bad are the lying and lack of empathy. The rest seem neutral (low emotional experience is something we hackers tend to identify with), sensible (random people tend to be untrustworthy), or admirable (delayed gratification).
Using charm and calculated moves to influence others isn’t a bad thing. It’s the basis of flattery.
I wish there was a positive version of Machiavellian which cut the lies and lack of empathy. Those are genuinely bad.
No, because smart people realize they are playing an iterated game and that behaving in a way that people identify as Machiavellian is actually suboptimal in the long run.
So they're smart enough to be calculated and stupid enough not to be so calculated that they look untrustworthy.
> No, because smart people realize they are playing an iterated game and that behaving in a way that people identify as Machiavellian is actually suboptimal in the long run.
Even if you are right coincidentally (which I wouldn't be so sure about), that's still poor argument assuming you realize your belief in what optimal strategy is what it is - just an educated guess.
Maybe you missed the bottom section? There's plenty of comments taking umbrage at it.
Alternatively, you think it's okay to make up stuff about young people because they got a seed round. That's stock-human behavior but it's not rational or kind.
I was specifically thinking of Cursor when I said “companies with twenty billion dollar valuations”.
My point, as I think was clear, was that criticising the founders of billion dollar companies via satire is not “punching down” by any means. Nor is it libel. You are throwing words around without meaning.
(and “young people”, there we go with the smolbean stuff again. If they’re too young to face criticism then they’re too young to be CEOs of billion dollar companies. You can’t have it both ways)
"Punching down" was about the watchlist section, not Cursor. You brought up Cursor, I didn't, and only after the fact.
"There we go with the smolbean stuff again": I never said that or anything like it. You're putting an argument in my mouth and then swatting it down. Twice now.
"If they're too young for criticism they're too young to be CEOs of billion dollar companies. You can't have it both ways." Scroll the watchlist. Most of those people aren't running billion dollar companies. That's the whole point. I definitely agree not all CEOs are good people and I generally agree the irrational argument all CEOs no matter of age are more likely to be net-destructive to society. That's the most extreme version of what you're saying, and we likely agree on it.
So we agree the conduct towards Cursor, and whatever other companies you want to name, is fair game. The only question is whether that extends to literally everyone on the list. I don't think it does. That's it.
Obviously this is quite unfortunate. While these cases can highlight latent mental health problems, it's still an issue that such things being exacerbated. I also think it will be interesting if anyone ever quantifies whether some LLMs are more likely to induce AI Psychosis than others. I'd be surprised if the guard rails are functionally identical from one LLM to the next, and there is a clear role for regulation to play here.
Some choice quotes:
> “What we’re seeing in these cases are clearly delusions,” he says. “But we’re not seeing the whole gamut of symptoms associated with psychosis, like hallucinations or thought disorders, where thoughts become jumbled and language becomes a bit of a word salad.”
> There seem to be three common delusions in the cases Brisson has encountered. The most frequent is the belief that they have created the first conscious AI. The second is a conviction that they have stumbled upon a major breakthrough in their field of work or interest and are going to make millions. The third relates to spirituality and the belief that they are speaking directly to God. “We’ve seen full-blown cults getting created,” says Brisson.
Also, for her podcast, the well-renowned couples therapist Esther Perel recently counseled a data scientist who was starting to fall in love with a chatbot he created, even though he is well aware of how the algorithm works [1]. I found it worth listening to. Perel very gently points out that a) he deluding himself and b) the deeper issue is the individual's sense of self-worth / self-esteem.
I worked in tech for ~8 years and am now finishing up a PhD. One thing that stood out to me is how that in most scientific fields review papers tend to receive more citations than empirical work. This post looks at citation patterns and argues that scientific progress depends just as much on abstraction and synthesis as it does on empirical tests.
When I worked at arXiv I looked at usage statistics that we didn't make public because we didn't want people to get the wrong idea.
One thing we knew is exactly that: the most viewed papers were review papers. You read a lot of them on the road to a PhD.
Another strange thing about review papers is that they escape the usual standards for evaluation in science. That is, as an outsider I can appoint myself to write a review paper without doing any research in the field, and it's possible I could do a very good job. One of the fun things I did in grad school was make a bibliography and short review of papers on the phenomenon of "Giant Magnetoresistance" at the request of the experimentalist on my committee.
A few months ago, OpenAI shared some data about how with 700 million users, 1 million people per week show signs of mental distress in their chats [1]. OpenAI is aware of the problem [2], not doing enough, and they shouldn't be hiding data. (There is also a great NYT Magazine piece about a person who fell into AI Psychosis [3].)
The links in other comments to Less Wrong posts attempting to dissuade people from thinking that they have "awoken their instance of ChatGPT into consciousness", or that they've made some breakthrough in "AI Alignment" without doing any real math (etc.) suggest that ChatGPT and other LLMs have a problem of reinforcing patterns of grandiose and narcissistic thinking. The problem is multiplied by the fact that it is all too easy for us (as a species) to collectively engage in motivated social cognition.
Bill Hicks had a line about how if you were high on drugs and thought you could fly, maybe try taking off from the ground rather than jumping out of a window. Unfortunately, people who are engaging in motivated social cognition (also called identity protective cognition) and are convinced that they are having a divine revelation are not the kind of people who want to be correct and who are therefore open to feedback. Because one could "simply" ask a different LLM to neutrally evaluate the conversation / conversational snippets. I've found Gemini to be useful for a second or even third opinion. But this means that one would be happy to be told that one is wrong.
It's probably an artifact of how I use it (I turn off any kind of history or "remembering" of past conversations), but when I started becoming really impressed by tools like claude/chatgpt/etc. was the first time I was chasing down some dumb idea I had for work, convinced I was right, and it finally gently told me I was wrong (in its own way). That is exactly what I want these things to do, but it seems like most users do not want to be told they are wrong, and the companies are not very incentivized to encourage these tools to behave that way.
I have identified very few instances where something like chatGPT just randomly started praising me (outside of the whole "you're absolutely correct to push back on this" kind of thing). I guess leading questions probably have something to do with this.
In one recent thread about StackOverflow dying, some people theorized that the success of LLMs and thus failing of SO could mostly be attributed to the amount of sycophancy of LLMs.
I tend to agree more and more. People need to be told when their ideas are wrong, if they like it or not.
SO was/is a great site for getting information if (and only if) you properly phrase your question. Oftentimes, if you had an X/Y problem, you would quickly get corrected.
God help you if you had an X/Y Problem Problem. Or if English wasn't your first language.
I suspect the popularity is also boosted by the last two; it will happily tell you the best way to do whatever cursed thing you're trying to do, while still not judging over English skills.
It became technically incorrect. You couldn't dislodge old, upvoted yet now incorrect answers. Fast moving things were answered by a bunch of useless people. etc.
Combine this with the completely dysfunctional social dynamics and it's amazing SO has lasted as long as this.
The technically incorrect issue is downstream of their rigid policies.
Yes, answers which were accepted go Python 2 may require code changes to run on Python 3. Yes, APIs
One of the big issues is that accepted answers grow stale over time, similar to bitrot of the web. But also, SO is very strict about redirecting close copies of previously answered questions to one of the oldest copies of the question. This policy means that the question asker is frustrated when their question is closed and linked to an old answer, which may or may not answer their new question.
But the underlying issue is that SO search is the lifeblood of the app, but the UX is garbage. 100% of searches show a captcha when you are logged out. The keyword matching is tolerable, but not great. Sometimes Google dorking with `site:stackoverflow.com` is better than using SO search.
Ultimately, the UX of LLM chatbots are better than SO. It’s possible that SO could use a chatbot interface to replace their search and improve usability by 10x…
SO is officially dead according to the graph of number of questions posted per month.
Google+SO was my LLM between 2007-2015. Then the site got saturated. All questions were answered. Git, C# Python, SQL, C++, Ruby, PHP, most popular topics got "solved". The site reached singularity. That is when they should have frozen it as the encyclopedia of software.
Then duplicates, one-offs, homeworks started to destroy it. I think earth society collectively got dumber and entitled. Decline of research and intelligence put into online questions is a good measure of this.
> People need to be told when their ideas are wrong, if they like it or not.
This is one of those societal type of problems rather than a technological one. I waffle on the degree of responsibility technology should have (especially privately owned ones) in trying to correct societal wrongs. There is definitely a line somewhere, I just don’t pretend to know where it is. You can definitely go too far one way or another - look at social media for an example
It all has to do with specific filler words you use when prompting, especially chatGPT. If you use words that suggest a heavy (and I mean you really have to make the LLM know you're questioning), then it will question to an extent as you imply. If you look at the chats that they do have from this incident, he phrased his prompts as more convincing rather than questioning (i.e "Shes doing this because of this!") So chatGPT roleplays and goes along with the delusion.
Most people will just talk to LLMs like they are a person, even though LLMs won't understand the difference in complex social language and reasoning. It's almost like robots aren't people!
Companies want the money and continual engagement. People getting addicted to AI, as trusted advisor or friend, is money in their pockets. Just like having people addicted to gambling or alcohol, it's all big business.
It's becoming even more apparent, that there is a line between using AI as a tool to accomplish a task versus excessively relying on it for psychological reasons.
> A few months ago, OpenAI shared some data about how with 700 million users, 1 million people per week show signs of mental distress in their chats
Considering that the global prevalence of mental health issues in the population is one in seven[1], that would make OpenAI users about 100 times more 'sane' than the general population.
Either ChatGPT miraculously selects for an unusually healthy user base - or "showing signs of mental distress in chat logs" is not the same thing as being mentally ill, let alone harmed by the tool.
Having a mental health issue is not at all the same thing as "showing signs of mental distress" in any particular "chat". Many forms of mental illness wouldn't show up in dialogue normally; when it would, it doesn't necessarily show up all the time. And then there's the matter of detecting it in the transcript.
I don't know the full details, but 700M users and 1 million per a week, means up to 52M per year though I imagine a lot of them show up multiple weeks.
You also don't take into account that the userbase itself is shifting.
That being said: Those of us who grew up when the internet was still young remember alt.suicide.holiday, and when you could buy books explaining relatively painless methods on amazon. People are depressed. It's a result of the way we choose to live as a civilization. Some don't make the cut. We should start accepting that. In fact, forcing people to live on in a world that is unsuited for happiness might constitute cruel and unusual punishment.
Maybe, just maybe, we should fix the fucked up world we created instead? Shunning the modern culture of individualism would be a great first step, followed by promoting communal culture. Live exactly how we evolved to live for hundreds of thousands of years.
> Because one could "simply" ask a different LLM to neutrally evaluate the conversation / conversational snippets.
The problem is using LLMs beyond a limited scope, which is free ideas but not reliable reasoning or, goodness forbid, decision-making.
Maybe the model for LLMs is a very good, sociopathic sophist or liar. They know a lot of 'facts', true or false, and are can con you out of your car keys (or house or job). Sometimes you catch them at a lie and their dishonesty becomes transparent. They have good ideas, though their usefulness only enhances their con jobs. (They also tell everything you say with others.)
Would you rely on them for something of any importance? Simply ask a human.
Fair question. While I'm not an expert on AI Alignment, I'd be surprised if any AI alignment approach did not involve real math at some point, given that all machine learning algorithms are inherently mathematical-computational in nature.
Like I would imagine one has to know things like how various reward functions work, what happens in the modern variants of attention mechanisms, how different back-propagation strategies affect the overall result etc. in order to come up with (and effectively leverage) reinforcement learning with human feedback.
I did a little searching, here's a 2025 review I found by entering "AI Alignment" into Google Scholar, and it has at least one serious looking mathematical equation: https://dl.acm.org/doi/full/10.1145/3770749 (section 2.2). This being said, maybe you have examples of historical breakthroughs in AI Alignment that didn't involve doing / understanding the mathematical concepts I mentioned in the previous paragraph?
In the context of the above article, I think it's possible that some people are talking to ChatGPT on a buzzword level end up thinking that alignment can be solved via "fractal recursion of human in the loop validation sessions" for example. It seems like a modern incarnation of people thinking they can trisect the angle: https://www.ufv.ca/media/faculty/gregschlitt/information/Wha...
> maybe you have examples of historical breakthroughs in AI Alignment that didn't involve doing / understanding the mathematical concepts I mentioned in the previous paragraph?
Multi agentic systems appear to have strong potential. Will that work out? I don’t know. But I know the potential there.
Per the article, this seems even better than the headline would suggest:
> Histotripsy generally seems to stimulate an immune response, helping the body attack cancer cells that weren’t targeted directly by ultrasound. The mechanical destruction of tumors likely leaves behind recognizable traces of cancer proteins that help the immune system learn to identify and destroy similar cells elsewhere in the body, explains Wood. Researchers are now exploring ways to pair histotripsy with immunotherapy to amplify that effect.
Not only do people leave the US but stay in Academia, plenty of people leave the research pipeline after receiving years and years of highly specialized, expert training. As an American who used to work in Tech and is currently getting a PhD, the geographic constraints on the (top tier) academic job market are more severe than people outside of Academia typically realize. It's a shame, because if it were the norm that science could happen by university-trained experts but in non-university institutions, we could a) fix the leaky pipeline, and b) see greater scientific progress.
What I mean is that if you don't like the company you work for in, say, SF, you can switch companies without having to switch houses. In Academia... it's akin to going to conservatory for classical music: you have to travel to where the orchestral openings are. This is a bit of a legacy problem from Wilhelm von Humboldt's idea to combine teaching and research, which led to the modern university system.
I'm far from the first person to say this, btw. Convergent Research's "Focused Research Organization" concept as well as The Arc and Astera Institutes are a few recent examples of people trying to provide escape routes from having to deal the large degree of "institutional tech/systems debt" in university contexts. For a great essay on why this is necessary, see "A Vision of Meta-science" (highly recommended if you are interested) [1].
The good news is that people are starting to come around to the idea that the scientific ecosystem would benefit from more diversity in the shape, size, and form of science-generating institutions.=The NSF just announced a new program to fund such "independent research organizations." I think this could give people who want to go into the sciences as a second career and who have a bit of an entrepreneurial tendency a new kind of Job opportunity [2]. We talk about Founders all of the time in Tech, we should probably have some equivalent in the best possible sense of the term, in the Sciences.
[0]https://pubmed.ncbi.nlm.nih.gov/31030116/