That's a great question! You probably need 2x the thickness of the magnets just so there's some amount of deformation possible. Could you clarify which modeling you are talking about for the deformation estimation?
Custom geometries will require a new network, yes!
Got it, makes sense! For the deformation, I mean would there be enough information to determine how a mesh of the eFlesh would be deformed, to cause the observed magnetic readings
The fun thing about using microparticles is that there's no dead zone! In fact, the edge response is even stronger (as you can see on the video on our website) because despite the distance from the chips, the skin is much more deformable at the edges.
I did watch the video but that's not the same as a precise repeatable experiment. As you say the edge response does seem stronger which means the sensor response is not linear across its surface. I guess I'm thinking of precision manipulations of objects in predictable ways, which is probably not your original intent, but it seems likely you could improve the sensing at the edge. An experimental measurement of the response might show some nonlinearities across the surface which you might consider correcting by using a microparticle cap that varies in thickness or correcting it in software, to produce a more accurate sensor surface. While it seems quite useful as it is, adding precision may expand the possible uses, such as finer manipulation of more fragile objects. It would also be interesting to see how the response varies for different kinds of contacts with objects, such as gripping a cube by the corners and by the sides, by the sides of a sphere, soft objects in various orientations, maybe others. Another possibility is being able to infer the mass of an object when the sensors are used to lift an object. The deformation at that time may directly correspond to weight. Together it may be possible to do some rough object identication, such as "pointy contact surfaces with mass of 20g". Combined with steroscopic cameras to ID the object, this could give a machine learning algorithm more to work with when figuring out how to manipulate objects. You might be able to use the ability to measure slip and the known distance between the grippers to tell how soft an object is, and together with camera input, decide how fragile an object is and whether the gripper is crushing it. The force changes during a crushing motion might indicate if the object is just soft or if it is semirigid and might break. Besides gripping, you could explore pushing and rubbing objects as well. Rubbing could tell you something about surface texture, which is also related to what the object is made of. Maybe there are uses in rolling objects between the grippers also? To reorient the objects along an axis of rotation while simultaneously characterizing the nature of the object?
While the sensor gives us direction vectors, they serve as good proxies for contact location, as we showed with ReSkin, https://reskin.dev.
That being said, the exact quantities the policy depends on are hard to interpret, given the use of deep learning. This could potentially be modality agnostic, but there has been no sensor so far that has shown (1) the ability to detect intuitively relevant quantities like contact location and 3-axis forces, and (2) sufficient signal consistency for deep learning models to generalize across instances. This was a key motivating factor for AnySkin, and we found a relatively straightforward fabrication procedure that enables this for magnetic sensing.
Curious, could you not calibrate using a force sensor, then include the output as a learning parameter. This seams a naive approach, which likely means it has been tried early on with other low hanging fruit, but I'm curious what the analysis of that approach is. Is there a fundamental reason this wouldn't work?
The reason we don't want to do this is that it is difficult to cover all possible characteristics. Say we do single point contact localization, and 3-axis forces prediction. What happens when we have multi-point contact? The calibration has only been used to calibrate/align in a lower dimensional space. This is primarily why not needing calibration and baking this into the hardware is a lot more appealing. The user/designer no longer needs to think about the task and the dimensions of alignment required for that task.
With capacitative sensors, it is unclear from existing literature if it is possible to detect shear. Additionally, they generally operate at significantly lower frequencies.
While this is possible, it would create stress concentrations within the elastomer and could significantly affect its durability. We saw this effect even when using larger magnetic particles as with ReSkin, https://reskin.dev. If instead we make the elastomer more rigid, it would worsen grasp stability.
I think you could 3d print TPU half shells w/ some reasonable infill (i'm sure there's one with good force transmission characteristics for this?), and then seal the magnet array inside of the two halves.
This is a very insightful summary, thank you! A few things to add about AnySkin that might be relevant:
- AnySkin expressly handles wear and gunk by being replaceable. So if it wears out, and you have a heuristic or learned model for the old skin, it will work pretty well on the new skin! We verify this through an analysis of the raw signal consistency across skins, as well as through visuotactile policies learned using behavior cloning. We found swapping skins to work for some pretty precise tasks like inserting USBs and swiping credit cards.
- Could definitely be used for part motion detection
- Soft, inflatable grippers are effective, but often passive. AnySkin is not just soft, but also offers contact information from the interaction to actively ensure that blueberry doesn't get squished!
- This sensor would be key for robots that seek to use learned ML policies in cluttered environments. Robots are very likely to encounter scenarios where they see an object they must interact with, but the object is occluded either by their own end-effector(s) or by other objects. Touch, and an understanding of touch in relation to vision becomes critical to manipulate objects in these settings.
- Industrial robots do have very sensitive motor and arm feedback. However, these systems are bulky and unsafe to integrate into household robotic technologies. Sensors like AnySkin could be used as a powerful, lightweight solution in these scenarios, potentially by integrating with some exciting recent household robotics models like Robot Utility Models.
- ReSkin, the predecessor to AnySkin, has previously been used quite effectively for fabric manipulation! (see work from David Held's group at CMU). AnySkin is more reliable as well as more consistent and could potentially improve the performance seen in prior work.
> - Industrial robots do have very sensitive motor and arm feedback. However, these systems are bulky and unsafe to integrate into household robotic technologies. Sensors like AnySkin could be used as a powerful, lightweight solution in these scenarios, potentially by integrating with some exciting recent household robotics models like Robot Utility Models.
I bet having good touch sense would let you get away with much cheaper mechanical systems for the robots.
Industrial robots are mainly bulky because they need to be very robust and precise at almost incredible speeds. (I work with those). Its not uncommon to have a 500+ kg robot on 500kg rail (totaling 7 axes) to actuate a 1mm wide, 5cm long nozzle, and moving it at speeds of 1+ meter/second, while navigating it in a gap where it has 0.5mm space on each side of the nozzle. Consistently, all day every day.
Lots of industrial robots arent even meant to touch their work piece, yet the robustness is the only way to make the whole assembly rigid enough.
I can imagine a touch-sense equiped arm could be made way smaller (less rigidity being compensated by quick enough feedback loop), but the speeds would probably have to decrease quite a bit. Not a problem for home robots tho.
We are just collecting emails on the Google form as contact information to get more details when shipping samples. I am sorry that the form is asking for a google account - we will fix that as soon as possible.
Yes, and importantly we find that visuotactile policies work even when replacing skins. This hasn't been shown before, to the best of our knowledge, and opens the door to a number of exciting large-scale applications of this sensor.
Custom geometries will require a new network, yes!