As a manager, the first thing you do is get to know your people. Some of them will be very strong and trustworthy. You give them the hardest work, and you ask them the least. That’s how you scale your team’s scope without getting overburdened. And that’s why teams have key people.

But when you work with LLMs, you still need to understand most of the ideas yourself if it’s a serious product. Because in the end, it means nothing to “trust” an agent. You’re the one responsible for what you and the LLM ship.

Engineers who have avoided learning the soft skills are going to have a harder time adjusting.

People are different, they will be sometime affected by their emotional situation, surrounding, no matter how much they're being paid, You need to understand their mental situation, did he got scolded by the upper management today? He might not be on his best of his capacity right now. Or they might've pulled an all nighter and really not in their best position.

If someone going to treat people like an LLM, definitely is not going to have good time

> It's a lot easier to re-establish goodwill when you can clear the memory and context of a program vs. a human though.

That's the point. You can't erase human memories, experiences, (unless you bonk their head hard enough according to movies in my subcontinent, another bonk might reverse too) which will definitely affect their decisions and results.

I would imagine that managing a team of AI-agents is totally different from managing a team of people.

It's about knowing the capabilities of people, what can they do, what do they excel at and where they need support.

The exact same applies to LLMs, each of them have their own very distinctive styles as well as strengths and weaknesses.

Just like human engineers, LLMs can't just be plugged in to the same role as the previous one with the same instructions and expect the same results.

Thus: soft skills.

“Soft skills” in management just means figuring out how to get what you want from the people you have available to you. In that respect those skills translate to using an LLM.

For instance, how do you motivate people to work long hours, put in extra effort, feel proud of their work? How would you do that with AI?

They may be softer, but they're really not an identical set of soft-skills.

To illustrate the difference, imagine: "Hey, you've got all those soft-skills from tweaking the AI stuff, right? I need you to motivate Bob to get his head back in the game, but without causing him to resign."

Why? One, the companies like Google or Anthropic or OpenAI are working hard for it not to be a skill. That's the whole point. Second, these system are opaque, so there is no understanding to happen, only superstition, which might be wrong or change tomorrow.

I beg to differ. It is a simple skill that a great many have, but that doesn't make it “not a skill” - there are certainly many that don't have it, or don't want to practise it. Though I wouldn't name it specifically for Google, it is the more general “finding information online” skill which feels more specific because for many people these days it doesn't extend much beyond using Google or whatever their browser's default search service is.

People without the skill are quite evident: many of the closed duplicates on SO and similar sites are due to people lacking the skill to find information in old answers and effectively just asking others to look things up for them, the same for this week's 20th+ “my first layer has these bumps and gaps, what is going on?” question on any 3D printing forum (facebook groups etc.) that could be answered by scrolling down a few posts, and I'm sure the equivalent happens in groups serving any other plaything/hobby/skill/whatever.

Neither are skills that a large portion of users of those services pursue to any meaningful extent, I'll grant you that. They also certainly are not synonymous with the term "soft skills" as I know it. So I think I am on your side of the fence on that part any way.

But I think you can't. It seems to me, instead, one is better at googling/prompting the better they are in a particular domain, but it only applies in that domain. Like knowing a jargon is not a skill, knowing the domain is.

https://arxiv.org/abs/2201.11903

There are certainly local “finding information online” experts in many families and social groups.

> and how do we know they are an expert

They are the ones who get mentioned a lot in conversations in the manner “I'll have to ask [name]” with the implication that [name] will look up the information or know it from previous occasions people (possibly this specific person and [name] is getting sick of them asking and not remembering simple answers!) have asked.

https://www.youtube.com/watch?v=fo1BR9itwOY

https://blackhat.com/presentations/bh-usa-05/bh-us-05-long.p...

https://en.wikipedia.org/wiki/Offensive_Security

If you can always say “just use it right” every time a critique comes up, then we aren’t having an honest conversation about the limitations of these tools.

If you enter the same prompt 3 times the results are of pretty significantly different quality. ChatGPT literally has you A/B test for them sometimes. They’re right to call it superstition - it feels like we’re making incantations and hoping for the best a lot of the time.

Prompting LLM’s still feels like a constant game of guess and check. At best you can argue it’s an educated guess. I don’t know about you but I didn’t learn math by guessing and checking, I frequently had to work backwards and review where I went wrong and/or I had the answer given to me with the work shown so I can learn. I can’t do that with a prompt. When I get bad results (which we all frequently do) I just guess what it didn’t like, try again, and pray for a better result.

I do a lot of guessing and validating in learning maths. It is pretty efficient way to build that conceptual understanding. I even try to predict the next big theorem as I am listening to a lecture or reading a text book. It's engaging.

And honestly, I am enjoying learning this new way to make code I am pleased with. Using the LLM effectively and for quality deliverables is different from typing in many surface ways, but modularity of thought, iterative design and implementation, simplicity and generality, documentation, all still pay off.

But that's the fundamental property of it - it is stochastic by nature. The skill is to learn how to sandwich deterministic logic between layers of randomness. Determinism doesn't live in the model. It lives in the harness you build around it. You can't make the model deterministic (it simply cannot be), so you make the system deterministic instead. Validation before, validation after, the randomness stays contained in the middle.

Talking to an LLM is not a skill, having a meaningful conversation leading to practical outcomes is.

Humans are called experts in many domains whereby they cannot achieve duplicate results. In fact, in what domain can an expert do that?

Nobody crosses the same river twice.

Thats where the "skill" part comes in. Like your 3 attempts at email that `almost always improve`. This is where you input the "previous versions", not directly back into the model. It's the "soft skill" of being flexible and adjusting based on how an entity responds to the input. Not learning how to adjust the input (by using previous attempts to inform the next) to more optimally direct the output given the state of the llm (chat? agent? model, effort lvl etc) it will seem chaotic.

The skills wont make an llm* deterministic, same as applying these soft skills to people. Give the same input to a person in `3 times separate, unrelated attempts` you are likely to get 3 different seemly chaotic outcomes. The "skill" is in being able to take what you received as output the first time, and make adjustments based on previous attempts while accounting for the state of the entity for the next attempt.

* In no way am I saying they are conscious beings or whatever nonsense by using people in the analogy. There are, however, parallels in how a set of soft skills (and this is why they are "soft" skills) can be used to get more optimal results from an entity that should never be expected to act as a pure function.

LLMs don't have emotions?

I suppose you could describe having the modesty to admit to yourself when you don't understand and research something deeper could be described as a soft skill, but I'd say it's a stretch. You are dealing with yourself in that scenario, not others.

For software engineering, you're perfectly right that reading the code is one way to build that trust. But the industry has evolved many other methods, and I think those (and new methods based on those) will become more and more important in the future.

While I agree at face value, I also believe a lot of managers and PMs do not have enough work to justify 40 or more hours of work a week, so attending standup, meetings, etc. is performative attempt at self-preservation.

Of course, there are many managers and PMs that are leading death marches, so I know it's highly workplace dependent.

It is somewhat new for most ICs to need this skillset, as opposed to tech leads/staff folks. What books would you suggest for this new reality?

You can trying to get around this with RAG and markdown files and skills but you're basically building from scratch the "tools" on how to remember the codebase that you take for granted with people

With the exact same tools you use with LLMs: proper documentation and detailed instructions.

Then you can grab any random consultant or LLM with a fresh context and get them to work.

I don't think anyone would mind having a competent manager or PM in a standup, someone that is actually contributing towards finding solutions and ways to move forward.

They are so convincing that an emergent property of the Turing Test is also being shown: that real humans are called bots by people that genuinely believe the other is a bot.

The standard 3-person Turing test with 2 people talking and a third observing and trying to decide which is a computer, if any, has been summarily defeated.

(For the record, my comment above was in jest, to point out the real experience of humanity in most coding environments)

These days, I find it difficult to believe there's a human who couldn't be fooled with the right prompt.

When I run the Turing test, it's me testing the computer. I don't care if anyone else isn't able to discriminate. Of course there is no "standard Turing test" as that would imply that it's an some kind of bot testing another AI and obviously that fraud with issues.

The Turing test is a human testing the computer and that human is me.

The reason I think that is because it puts communication into a very simplified and regimented framework, so simple and unable to adequately answer the needs of the communicating parties that nobody actually uses it for the intended purpose. The actual communication happens between people who actually need to work on something, in the format that allows more freedom, with more aids, more prep time, perhaps over multiple sessions.

Sometimes, probably, as a manager, you have to work with a very low quality workforce, lacking motivation and simply avoiding doing any useful work as much as possible (eg. some overseas outsourced project that gets paid by an hour). In this case, standups become a soft punishment tool: something to verify that workers showed up to work, that they didn't lie about the progress made so far etc. You would still have to do the former part of the management, but now you also have this kindergarten-teacher style chore on top of it.

By most I mean maybe 75%

In my experience, most middle managers, probably 90%, have no clue how to manage software developers. It’s not that hard to manage teams IMHO, but you need to focus on the right things.

This looks like we choose to hear what we want to hear.

The understanding argument is about making the point that "The point was always to augment" the human understanding when we worked with the machines.

Engineers are finding a new set of challenges but nothing of the sort that managing real people requires.

PMs, managers and leaders behaving like status update junkies is a side effect and not the reason why those roles exist.

There is also a variation of Amdahl's law - if you automate more things, the predictability of remaining work will decrease, because it will now take more time.

Also, formal languages still trump natural language. Despite LLMs; I think it's a stepping stone to something better but "vibe coding" will turn out to be unsustainable.