Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I've been working in data roles for 10 years and hold a masters in ML. I've hired and managed each of the roles you mentioned. I think of the responsibilities of each of those roles as:

-ML Engineers as building software infrastructure to scale machine learning inference and training.

-Data engineers focusing on data infrastructure and pipelining into either model inference, training, or other business intelligence platforms

-Analysts consume the product of the data engineer in the BI platform or excel, where the results would be consumed as a report in some form.

-And ML Researchers would be those inventing novel machine learning algorithms to deploy in the ML Infrastructure managed by the ML Engineers

-And data scientists to deploy well-known ML algorithms or statistical inference on varying datasets on the ML Infrascturue or as a slide deck.



How hard could it be to find one person who can do all that?


Depends on the amount of data, reports, pipelines... If the company is small you might not have any of these problems. Every Mom&Pop store has some sort of data to run the business but they don't need a "data" person.

Once you have 10s of datastores + pipelines, 100s of reports and a "data lake" in the TBs you'll likely be needing specialized people.


So far I've spent my career in small teams / startups and it's starting to become apparent that a lot of what's assumed in these titles only applies in larger corporations where resources are abundant and it makes business sense to have a specialist focused on a single aspect.

Unfortunately I'm at a point where I have 'jack of all trades master of none' syndrome and it's causing me to fall in between the cracks professionally. I'd like to move to a larger company where I can develop deep expertise in a narrow topic.


ymmv, but as a data scientist at young startups, I often am the one giving new tasks to the software engineers, and facilitate teaching and training if they need help.

Most of those roles a software engineer can do.


Thanks for the breakdown!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: