For my projects, I can produce with LLMs a codebase that I'd be proud of, but it requires extensive use of the plan mode and nagging anytime the implementation looks much more complex than the feature at hand requires.
Anytime I tried spec driven development, it produced total garbage, the problem is that the model usually swings too far, if they write unnecessary complexity, a nag about it and you risk it code golfing, this problem across 100 lines of specs and you're guaranteed it will swing too far on some segments you just wanted X slightly more than Y (usually for me it was reliability dropped for better readability since # users = # developers = 1)
Another thing I dropped is steering implementation design too early, start with the goal and nag in the direction you want, it works well for me but might not work well for big complex codebases.
For now I love LLM coding but I'm sure my opinion will change if I have to review code from a developer that does the 1 prompt = 1 PR without looking at the code jutsu.
Let's suppose each models was subsidized at 70%, so that we only pay 30% of the cost. They would loose much more money per token on the more powerful models. It's in their interest to encourage the use of the less expensive models. Let's say they increase Luna subsidies at 90%. They would still "save" relative to the use of the more expensive models.
The company losing money does not mean the model inference in API is 70% subsidized- that’s a crazy leap in logic. Obviously the massive number of free chatgpt users are getting subsidized 100%.
If the frontier labs are indeed facing threats from open weight models, we would first see this in pricing pressure from the base model (like Luna) first.
High-performing open weight models being released recently, and your customers looking into working with multiple providers as a result, are a great reason to drop prices on your non-frontier offerings.
Although I'm sure there are some efficiency gains, the technology is too new and labs are scrambling to release too quickly to think that the low-hanging optimization fruit has been picked already.
Yeah, like to see more disclosures about this. Not to mention the whole background debate on the real costs for the AI companies, at what point prices go up to reflect that, or the possible consequences of being overly reliant on this technology.
I got that recently, I really didn't like that question.
For the n-th percentile version, the obvious solution is sorting and it takes 10 seconds to get to that point, 5 minutes of implementation with tests. Good. It's all downhill from here.
Then you get hit with the "it's a data stream" and you realize you have to implement a balanced tree on the spot which I wouldn't describe as fun.
You may or may not be able to implement that. I did not. Blabbered something about Rust having sorted B-Trees and I don't think Python has them -- they do not on the standard library.
Then the interviewer leaned heavily on the "reduce memory usage" and I couldn't come up with a solution (no shit it's Ω(n) and he didn't even tell me to go fetch for a randomized algorithm). I later understood he expected the reservoir sampling solution which is basically keeping a representative group of size K that is a good proxy of the whole stream, it goes like this: keep the K first elements, any elements after that replaces any element of our sample at random.
What I did after 10 minutes of weird silence is to assume the data stream follows a normal distribution and computing the P-percentiles by computing the running mean and standard deviation.
I felt frustrated at the end of the interview because it really felt like a big gotcha of either you know the reservoir sampling "leetcode trivia" or you don't.
Literally the second I read "it's a data stream" I knew the answer was going to be reservoir sampling.
RS is really interesting to me. many people you talk to can realize you can compute the mean of a data stream (https://www.geeksforgeeks.org/web-tech/expression-for-mean-a...) without knowing the exact formulation. And it's not far from that to think of a sampling strategy to decide if a new sample should go into a fixed-size reservoir. (for all of these, I know specific hints that will usually help people get to the next step).
The only reason I know RS is because it was in the google3 monorepo and I was looking for interesting codes to use and found it. There was an associated Sharding class, LexicographicRangeSharding (https://www.mongodb.com/docs/manual/core/ranged-sharding/) which you could use to find near-optimal split points in sorted string tables so your mappers didn't end up with hotspots. If you had shown me Algorithm R in a stats class, I don't think I would have appreciated it at all, but seeing the code implementation and a useful example made it click.
I'm pretty confident I would've been able to come up to the solution in better circumstances, maybe even without hints.
But it was clear in this case that the interviewer just took a question from the company's bank of questions and was alt-tabbed for half the interview, I have felt the energy early and I was also half-checked out.
I'm aware I'm saying this post factum, but I had a very fun first interview with that company and matched well with the first interviewer so my expectations were high, and then I got hit by the big tech style interview when it was an early stage startup.
Good point. I've ended interviews (as a candidate) when I believed the interviewer was being lazy (for example, administering a leetcode question and not even changing any of the details like input or output data). I ended up writing my own questions that aren't in leetcode because I interview candidates now. And I give 100% attention to the candidate.
>"it's a data stream" I knew the answer was going to be reservoir sampling.
But it's only an approximate percentile. Unless the interviewer mentions that an approximate solution is OK, you would be stuck. (And it's not fair to ask the candidate to ask whether an approximate solution is ok given that almost every problem has an easy "approximate" solution which is not explicitly not what they're looking for).
Yeah, you basically have to grind leetcode or get lucky - having a interviewer giving you enough hints or come up with a (for you) novel idea on the fly.
That sort of stuff is bullshit I assume meant to boost the interviewer’s ego. Anyone can come up with shit like that given time to prepare or the internet.
Unless you work in some highly specialised field maybe.
From talking to someone that worked at YouTube for 15 years, they still had a lot of core Python code in 2016 that was legacy from the OG company/team and that code needed to be transitioned to follow the Google way of doing things in C++/Go.
I don't think it was distinct enough from the Google culture like Android was at the start of the acquisition but it seems they had leeway to do their own thing.
The impact is local though, it would be only a problem if the median small company is more messed up than the large co.
It not likely to happen because being small there are more threats or market forces to deal with so they cannot do as they please. Monopolies or just economies of scale affords large co and the small number of executives that control them outsized influence - both good and bad.
Anytime I tried spec driven development, it produced total garbage, the problem is that the model usually swings too far, if they write unnecessary complexity, a nag about it and you risk it code golfing, this problem across 100 lines of specs and you're guaranteed it will swing too far on some segments you just wanted X slightly more than Y (usually for me it was reliability dropped for better readability since # users = # developers = 1)
Another thing I dropped is steering implementation design too early, start with the goal and nag in the direction you want, it works well for me but might not work well for big complex codebases.
For now I love LLM coding but I'm sure my opinion will change if I have to review code from a developer that does the 1 prompt = 1 PR without looking at the code jutsu.
reply