Yes unless the industry sees value in a step change in the scale on offer to regular devs, progress on massive nets will be slow.
Hooking things together is pretty much my job. I have used ROS extensively in the past but now I just hook things together using python.
But I consider what Tesla is doing to be pretty promising, and they are layering neural nets together where the output of three special purpose networks feed in to one big one etc. They call that a hydra net. No framework like ROS is required because each net was trained in situ with the other nets on the output of those nets, so I believe all compute logic is handled within the neural network processor (at some point they integrate standard logic too but a lot happens before that). Definitely watch some Karpathy talks on that.
And currently I am simply not skilled enough to compose multiple networks like that. So I could use multiple standalone networks, process them separately, and link them together using IPC of some kind, but it would be very slow compared to what's possible. That's why I say we're "not there yet". Something like Tesla's system available as an open source project would be a boon, but the method is still very labor intensive compared to a self-learning system. It does have the advantage of being modular and testable though.
I probably will hand compose a few networks (using IPC) eventually. I mean right now I am working on two networks - an RL trained trail following network trained in simulation on segmentation-like data (perhaps using Dreamer V2), and a semantic segmentation net that is trained on my hand labeled dataset with "trail/not-trail" segmentation. So far my segmentation net works okay. And a first step will actually be to hand-write an algorithm to go from segmentation data to steering. My simulation stuff is almost working. I built up a training environment using Godot video game engine and hacked the shared memory neural net training add on to accept image data, but when I run the sim in training on DreamerV2, something in the shared memory interface crashes and I have not resolved it. [1]
But all of this is a hobby and I have a huge work project [2] I am managing myself that is important to me, so the self driving off road stuff has been on pause. But I don't stress about it too much because the longer I wait, the better my options get on the neural network side. Currently my off road rover is getting some mechanical repairs, but I do want to bring it back up soon.
First off, amazing farm-bot project! I am looking forward to reading the details on your site.
Thx for the pointers on Tesla. Had not seen the Hydranet stuff. There was a Karpathy talk about 2 weeks back at a CVPR workshop .. he revealed the scale of Tesla's current generation deep learning cluster [1]. It is insane! Despite being in industrial research, I don't foresee ever being able to touch a cluster like that.
A lot of our current research involves end-to-end training (some complex stuff with transformers and other networks stitched together). There was a CVPR tutorial on autonomous driving [2], where they pretty much said autonomy 2.0 is all about end-to-end. I've spoken to a few people who actually do commercial autonomy, and they seemed more skeptical on whether end2end is the answer in the near-term.
One idea we toy with is to use existing frozen architectures (OpenAI releases some and so do other big players) and do a small bit of fine-tuning.
Yes unless the industry sees value in a step change in the scale on offer to regular devs, progress on massive nets will be slow.
Hooking things together is pretty much my job. I have used ROS extensively in the past but now I just hook things together using python.
But I consider what Tesla is doing to be pretty promising, and they are layering neural nets together where the output of three special purpose networks feed in to one big one etc. They call that a hydra net. No framework like ROS is required because each net was trained in situ with the other nets on the output of those nets, so I believe all compute logic is handled within the neural network processor (at some point they integrate standard logic too but a lot happens before that). Definitely watch some Karpathy talks on that.
And currently I am simply not skilled enough to compose multiple networks like that. So I could use multiple standalone networks, process them separately, and link them together using IPC of some kind, but it would be very slow compared to what's possible. That's why I say we're "not there yet". Something like Tesla's system available as an open source project would be a boon, but the method is still very labor intensive compared to a self-learning system. It does have the advantage of being modular and testable though.
I probably will hand compose a few networks (using IPC) eventually. I mean right now I am working on two networks - an RL trained trail following network trained in simulation on segmentation-like data (perhaps using Dreamer V2), and a semantic segmentation net that is trained on my hand labeled dataset with "trail/not-trail" segmentation. So far my segmentation net works okay. And a first step will actually be to hand-write an algorithm to go from segmentation data to steering. My simulation stuff is almost working. I built up a training environment using Godot video game engine and hacked the shared memory neural net training add on to accept image data, but when I run the sim in training on DreamerV2, something in the shared memory interface crashes and I have not resolved it. [1]
But all of this is a hobby and I have a huge work project [2] I am managing myself that is important to me, so the self driving off road stuff has been on pause. But I don't stress about it too much because the longer I wait, the better my options get on the neural network side. Currently my off road rover is getting some mechanical repairs, but I do want to bring it back up soon.
[1] https://github.com/lupoglaz/GodotAIGym/issues/15
[2] https://community.twistedfields.com/t/a-closer-look-at-acorn...