What is a sensory weaver? [Curiosiate] tells us: “A device which takes sensory data feeds in and converts it in various ways on the body as information streams as though a native sensory input.” As an example, they’ve built one.
This one, called “MK2 Lockpick” is a wrist-mounted array of linear actuators, with a lengthy design/build log to peek into. We don’t get PCB files (blame EasyEDA’s sharing), but we do at least get a schematic and more than enough pictures for anyone interested to reproduce the concept – the levels of bespoke-ness here warrant a new PCB for any newcomers to sensory weaver building, anyway. We also get a story of a proof-of-concept thermal input sensory weaver. The team even includes a lessons learned da, and plenty of inspiration throughout the posts on the blog.
This kind of tech is getting more and more popular, and we are sure there will be more to come — especially as we keep getting cool new gadgets like linear actuators in form of replacement parts. For instance, the actuators in this sensory weaver are harvested from Samsung S23 smartphones, and you could probably find suitable ones as iPhone replacement parts, too. Looking to start out in this area but want a quick build? Look no further than the venerable compass belt.
Everything is a sensory weaver.
Well, not quite – just devices that take information from beyond biological awareness and convert it into a data stream within awareness, for purposes of doing sensory substitution, addition, or expansion – because it’s too much a mouthful when the device can do all them, even simultaneously.
lets take a stone as an example, if you grab a stone in your hand and watch it. then the stone is providing a sensory feed to your hand and eyes, converting mass (or weight??) into something you can perceive and your eyes are doing something similar but with light so you can say hey look there is a stone on my hand! Sadly, our mind is prisoner of our senses and time. This bring to my mind a hot debate we had at my alma matter years ago, someone stated (and I still agree with him) that there is no such a think as “digital sound”.
Well, if using the summarized version of the longer quote from the site you’d be right, but there’s more nuance to what is trying to being described by the term.
From the linked description:
” Devices which take sensory data feeds in, crafted information abstracts outside the realm of normal biology, and transpile it in various ways on the body as information streams as though a native sensory input. ”
It is specifically about devices that take abstracts from beyond the current biological access, and specifically so as to represent an awareness of it in the mind over means of sensory substitution, addition and expansion. So in your rock example, the visuals of the shape, the depth mapping of it, could be converted to haptics, or to audio, such as in the case of the vOICe (https://www.seeingwithsound.com/), for those who don’t have access to the same sensory modilities.
So with that description hopefully a rock doesn’t fit anymore! :)
It is a term to try to refer to a whole class of devices, and the conversion they are are doing to grab information from beyond our current sensory/conceptual access because of our “prisoner” status mentally to the senses.
Hence the name lockpick, a tool to start to open new doors of perception. Because there is a whole lot more than vision, or even thermal awareness, spatial awareness for full 360°, and more to feed into the mind. (Neosensory, a company that does audio to haptics has something like 70 products in the works using similar concepts).
Although you are right about mass, but also in a way not, because we normally do have awareness of mass from our biological side, even if not specifically of that rock. it isn’t conveying anything aside from information about the device itself in that case, which isn’t quite following the sensory substitution, addition and expansion aspect, because then holding a rock would be sensory expansion, to gain awareness of the abstract of mass of the rock specifically. A more relevant example might be tying yourself to a scale’s sensor for haptic feedback, so you can know precisely how much weight is going on without looking at the scale?
I see, so a sensory weaver has to provide a measurement of the input being considered , got it, so far so good, but assuming we get a measured output, now how we can tap in that datastream without using our five senses? that is why I don’t get it when someone talk about sensory expansion. sensory substitution is easy and fun, (I remember old electronics magazines where they had diy projects like converting sound or distance to light or motion to walk in the dark,like bats)
Another example, recently HAD covered this interesting project of visualizing wifi rf signals, an is so cool ! someone may argue that we are doing sensory expansion here, but in reality we are only watching a greenish light from a display. so the black box get rf energy( measured in Joules) and is outputing energy (greenish light that we can grab by our naked eye. Of course I’m not expert on anything, but I think in the end all kind of energy has to be filter by our five senses in order to hit our brain, unles we are Bruce Willis who can see dead people :)
Seems we’ve hit the max depth of comments here, cannot reply to your latest!
We do use the 5 senses, it weaves in a new sense in the way we construct “senses” (or more accurately perspectives) into abstract information, which can be utilized to navigate either reality, or conceptual space. Take depth perception, for example – this “sense” is built from bifocal vision, hearing of sounds, and proprioceptive movement through space.
A blind individual, or deaf both have access to that information abstract awareness, just different perspectives upon them, and routes of updating awareness into the state of them (blind may miss objects moving if in a noisy environment, deaf similar if outside of visual range).
Same for wetness, which comes from texture, pressure and temperature – we have no “sensor” for wetness, and yet it presents itself as a distinct sensation mentally, even if over an existing medium, it is used to construct a perspective, a sense of awareness of something.
In that way, sensory weaving includes finding abstract information out in reality beyond our sensory range normally (or constructed abstracts), and generating reliable patterns of information by which to convert it into the biological perception, but is more the means of how each type of abstract may be best translated for each person. So after a while such as with sensory substitution – you don’t “feel” the sound, you are gaining an automatic perspective into the abstract information, via a “sense” similar to that of depth perception or wetness, because of the reliability and contextualizing of the patterns the mind does.
Similarly, there have been studies showing activation in the visual cortex for facial recognition, when using sensory substitution prothesis:
https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2022.921321/full
This partly may be because of place/grid cells, if looking for fun resources on that: https://youtu.be/iV-EMA5g288, mapping relationship between abstracts, but also as a means of accessing new abstracts environmentally, automatically.
I’d say unless it has integrated long enough with the biological side so as to become a “perspective” intuitively in the mind, things like the wifi visualization isn’t so much sensory weaving as it would be “abstraction regulation” to convert information from beyond biological to one we can, but not as a continuous medium to perception itself like a sense of depth or weight from an external scale would be. One could probably create a sensory experience off the information sources from such things, however!
You are right, in a way again – the data is indeed routed over the 5 senses, but because of how the mind already constructs it’s “view” of reality, we do indeed seemingly gain access to new “perspectives” and senses in the way of sensory substitution and depth perception and similar “psudo-senses” (perhaps a better term). It injects and modifies the perceptual manifold, which may be getting into the philosophical side, and is still WIP clarity of term wise from me, for usage in explaining this sort of concept overall.
There is lots of rambling and musing but also sources to lots of interesting neuroscience and otherwise resources on the site linked, especially the “jailbreaking” post. (Which if just wanting resources, can find them all at the bottom of the page). I’ve put the link to that, as the website for the comment if clicking my name on this one.
Keep in mind I’m also not an expert on anything, and also not really an author/technical writer – so use skepticism if you do read any vs the sources directly!
This evidence has given rise to a new theory of brain organization proposing that brain specializations are driven by specific sensory-independent computations rather than by sensory-specific processes as classically conceived (Heimler et al., 2015; Amedi et al., 2017)….
So basically the biology gave us a bunch of senses to catch what is going on but once the information hit the brain is used to construct a model of the enviroment and the situation around us, and even depriving those basic senses or changing the way the brain gets the feedback it still finds ways to actually “know what is going on”, amazing hu? Amazing is so deep levels, maybe that why I feel so sad when I see those litle dogs running behind the owner’s car, I mean for a person would be like it’s ok they going to the mall and comeback later, but for a tiny brain like that of the pet, that experience (I guess) is traumatizing and he is maybe in shock unable to find a solution for the dilemma. Maybe we could even say that the origin of faith is rooted to the impossibility of our brain to tap into the conditions of our enviroment, if things goes too fast, to dangerous, or we simply can not know our circunstances and surronudings (think of the first humans or similars) our brain maybe just stop trying to find a meaninful insight and stuck in what we know as faith.
Everything something new!
Rule 34. Has to be.
For that you should be more thinking of https://github.com/buttplugio, which I did consider for these augmenting purposes, briefly… (And hope isn’t against comment rules to mention)
How about a nice game of chess?
Already a thing for chess https://learn.adafruit.com/cheekmate-wireless-haptic-communication/overview
Some related content in this podcast:
https://freakonomics.com/podcast/your-brain-doesnt-work-the-way-you-think/
“Hearing” with body parts other than ears, for example.
Some other work by David, in form of a recent paper of his covering more technical aspects with lots of nice references, for anyone interested in the science side on a more serious vs hobby level.
https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2022.1055546/full
Went crosseyed after a couple of sentences into the linked site’s description of “sensory weaver”. Why use five words when a thousand convoluted ones will do?
The ur-example that really should be linked is the North Paw, which has popped up on HaD a few times. https://hackaday.com/2015/08/13/hackaday-prize-entry-a-sixth-sense/ is one implementation.
I’m open to suggestions to help clarify the concept – I’d rather start with too much information to try to describe something, get a handle on it, than too little. What was too convoluted, in the first couple sentences?
Words are paint on a canvas, and I’m not an artist. I’m an engineer, not an author, that’s for sure.
Northpaw is a nice example! That’s one of a bunch of different examples of augmentive equipment people have made over the years in this sort of field. The future is only going to get weirder and wilder, hopefully.
ELI5, please.
Tl;Dr:
Perception is a mental map built from sensory input and memory, not just raw data. Through patterns like size, brightness, and pitch, the mind interprets and constructs reality. Technology can extend perception by introducing new sensory patterns (e.g., using wristbands for haptic feedback or devices that convert sound to vibrations for the deaf), offering new “lenses” for understanding moment to moment for reality but also the way we think about things potentially. Devices already do this, from fields of sensory substitution, addition and expansion research – what these do, to construct patterns of meaning to the mind is what I’m focused on with the term sensory weaving. Sensory weavers then, as a shortend way for me to refer to devices that help the mind access new perspectives, similar to how other tools like telescopes or microscopes expand our understanding of the world. (Seeing as the wristband can be used for all purposes).
Have longer explanation too, but decided to keep this short in case that already helps. (Not entirely sure the part wanted to be explained)
The 5 year olds you know seem to read Wittgenstein before breakfast. Those that I know phased out at the first word.
Anyway, thanks for trying but I can see this is going nowhere.
The writing on the website is the least concise writing I think I’ve ever seen. Lots of high-abstraction prose, very light on details.
I would start by suggesting a couple of examples of specific or practical applications.
Unfortunately, it is sort of about abstract information and manipulation, as well as how the mind constructs meaning around such things- which means you may like even less the writing of abstraction regulation even less.
Hopefully can clarify in a more stripped down way for here though.
The first paragraph gives a description that basically says sensory weavers are devices that do what sensory substitution, addition and expansion devices do – so those examples, and then goes on into what I’d rather like from an all-encompassing device that handles all those concepts, where sensory weav(ing) is what all those tools do in order to construct a new perspective mentally. The crafting of signals matched to biological limitations and methods of sense-making, which may be unique to each person as well.
The example of a “sensory weaver” itself would be devices that do all 3, which is what the wristband made does – but it need not be a wristband, or even haptics – it could take form of audio, visual or even a mix, in order to thread together patterns of generated signal discernable by the mind to construct a new “sense” around, in the way depth perception is constructed from multiple signals.
Specific, practical applications is taking new perspectives – what those are or to what end vary massively – many people probably won’t need a general purpose device vs one that does a single application, at least not unless wanting an exploration platform.
The practicality to me is for exploration, to find new perspectives, combinations thereof that might offer utility, insight for specific applications and purposes (which there are a fair number of – some of the posts on specific sensory threads cover some potential applications of those single sensor abstracts). Curiosity, of what isn’t seen, and this allows one to then have a look behind the curtains so to speak.
Benefits and applications of sensory substitution, addition and expansion in general already has pretty rich literature – although the aspect of finding various combinations of signals to filter or “tune in” to different perspectives, or mixing together multiple and how that generates new perspectives isn’t as deeply covered, in some areas. Usually it is about the same “perception” matching one in reality, vs exploring new ones when it is done.
Why do I think at an old 007-Movie now, where the villain played a kind of “computer game” against bond and “enhanced” the experience with fancy electro shocks? :-)
Ahhh, I remember, “Never say never” – https://youtu.be/MI9_cLu5-JY?si=aDey0G7tuMSa24p6&t=72
That’s a lot of syllables to say “haptic”
Reminiscent of situational awareness aids that I-worked-with-a-guy-who-worked-on a few decades ago, but I wasn’t sure I could mention.
But I see this in the public press now, so I’m sure it’s not classified: https://news.erau.edu/headlines/vibrating-vest-helps-combat-flight-deck-disorientation