I agree that providing something more tangible than just a number would be beneficial for some operations. But I think it would get annoying quickly. Having difficulty moving a "heavy" file is the opposite of a good user interface. Every manipulation should require as little mental and physical effort as possible. Apart from that, I can't apply force with my mouse — it just clicks.
I believe a purely visual approach could work well. For example: every file icon has the same front area (basically the rectangles we have now), but visually extends to the back with some sort of stylized 3D effect, according to file size. So a small text file looks like a thin sheet of paper, a 10MB file might look like it's made of thick cardboard, a 2GB video looks like a box with considerable depth. The scaling should probably be logarithmic, not linear, to work well with human perception.
You are right. This is just an exploration. The main motivation was to play with the pressure input from Apple Taptic engine. I was trying to applying "physical" properties to virtual objects.
Designing for visual is what is missing, I would want to work on the visual plus audio to compliment the "heaviness" I think someone else also mentioned in the comments on experiments with audio to express physicality to
I don't think the visual approach does much for interactions, so that's a separate concern. This is more about feel. E.g. inertia applied to CoG, so when you grab it outside CoG, large files will tilt more as you drag. Light files would probably start spinning even.
Now, applying inertia for the movement itself would be annoying. Please don't do that. :)
You talk about interactions. In my mind the useful part is the added information density which comes easy (we are good at judging object sizes visually, not by numbers). I also want this information before interacting with the files. The value is knowing in advance, at a glance, what you're dealing with.
When you mention CoG, spinning etc. it becomes game-like — not something I would want in a productivity focused interface. But if you're developing a game or a toy experiment, go for it.
Thats the micro interaction that needs to be explored with iteration. The taptic engine pressure response (IMO) is not the best , but applying some heuristic ux designs I think we can approximate CoG and inertia.
I’ve often wanted a setup where it became physically harder to send an email to me the more unread ones I have to deal with. Like having to cram an extra letter into a pigeon hole that’s already full.
Or maybe make the Slack send button extra hard to press when sending messages after hours. Like you need to apply all the pressure and sustain that pressure over some time and only then the message goes. The emergency and anxiety is built into the interface X-D
Heavy is relative. If you're working with videos, everything is heavy in terms of file size relative to most files. Yet, a small text file could be just as important as any video.
I think this is a fun thought experiment that is fundamentally a bad product idea.
I think this type of concept is worth exploring. Side channel feedback to the operator of a machine is getting less noticeable. Hard drives don't whirr and click like they used to. Cars don't have transmissions that shift.
When you pick up a physical object with your hands, you don't assume the heavier the object, the more important it is. Same with file size.
But if you pick up your carton of eggs every morning you'll know if you have enough left to make an omelette.
If you make a backups it would be nice feedback to feel it weigh about what you expected. When making room on a disk you could juggle a few folders to feel if they'll fit or not.
There was some advanced facility (nuclear reactor? particle accelerator?) that laid microphones near the machinery and put various speakers in the ceiling of the control room; helped precisely detect and pinpoint problems immediately.
That said I'll prefer just seeing the size of the file or folder in bytes as a number.
I'm personally more interested in feeling other system metrics, like network traffic or memory bandwidth.
I've always thought it would be neat if the accelerator pedal on cars had some sort of force feedback that was proportional to the amount of power the engine is putting out. That way the driver would be able to feel how hard they're demanding the car to work, and hopefully they would adjust their driving habits to go slower on steep hills, not hard accelerate out of traffic lights, etc.
Yeah! I liked seeing the "miles per gallon" meter that some cars have for that purpose. IIRC driving habits account for at least 10% of fuel efficiency losses, and by my napkin math, you could drop carbon emissions in the US by 5% if you 1. put such a meter in every car and 2. drivers heeded and learned from it.
While modifying pedals is risky, maybe you could take an OBD-II data stream and turn instantaneous power output into sound, or vibration... or lower your music volume the harder you push it...
I understand. My motivation was to play with the pressure input with "heaviness". Fundamentally , pressure is a continuous input that becomes harder to perform progressively. We can apply this characteristic ins some other relationship too.
Hearing the HDD back in the day was important to understand whether the computer was working; It seemed like a loss when we moved to SSD, but SSDs are so fast that sound isn’t a necessary sensor anymore.
Its true , the interaction only works when its surrounded with other interaction. Moving all the heavy files in an video editing bay with this much pressure will just accelerate carpal tunnel.
I was interested in exploring to give more physicality to the virtual objects , they are "opened" , they "moved " I was playing with the idea if they can be "lifted" or be able to "stick" with pressure etc.
See "Sonic Finder" for the Macintosh.[1] Heavy folders made bigger thumps when opened, or dragged and dropped. It was not widely used and disappeared. I did try it once. Cool, but not useful.
Very interesting, I'm wondering how that would feel in XR!
I already have quite a few demos with manipulating files, e.g. Immersive file browser (via remote WebDAV directories) https://video.benetou.fr/w/rHZTnX5MnHdWvWTPa2Rsw4 so curious how I could try that there.
In WebXR there is no pressure value but maybe more fingers could be needed. Also maybe for heavier files there could be some "lag" where the cube representing the file does follow the pinched fingers but with some delay proportional to the file size. Any suggestion welcomed!
Neat, I'd happily tinker with one but that's quite a demanding setup. From what I can tell from ManusVR and similar haptic devices there is no consumer driven cost dropping curve, unlike screen resolution thanks to mobile phones, that allow these to benefit from the same miniaturization and democratization as other things (screen, CPU/GPU, IMUs and other sensors). Maybe some MEMS breakthrough will enable cheap high resolution haptics but for now it remains bulky and expensive.
Awesome. I want to marry this with a concept I had a while ago around a computer keyboard with pressure sensitivity that adjusted the font size in proportion to how hard you struck the keys.
I like the idea of being able to enter into BILLY MAYS MODE just by furiously typing.
You might have to mess around with the software to figure out the details, but from a hardware perspective a hall effect keyboard should be at least able to infer the speed a key is travelling at when bottoms out.
That was entertaining to try as a demo, but I did not enjoy the UX of it because I don't have a proper sense of how much force I'm using on a touchpad. I have poor dynamic range in the amount of force I exert there.
My heavy files do feel heavy because it takes ages to move them through my painfully slow storage media. Can't say I'm a fan. It's a fun concept though.
I think exposing more of the internals helps. Consider what a mechanic can extract from the sound of an engine: in that sound a lot of information about the state of that engine is encoded. And it is a lot more than some programmer could abstract by simply having the motor vibrate if it is okay and have it not vibrate if it isn't. This however means you have to trust the user they will devolpe a sense of meaningful differentiation.
Physical objects interact with their environment in certain ways as a reflection of what they are, it is certainly an interesting question how this could be extended into the digital domain.
But they don't take more time to move (on the same partition). Also it would be nice to see at a glance if a folder contains a lot of data (number of bytes, not files) before starting an operation.
if you are moving a folder of files within a partition (hey, this was your idea) then you don't care how many or how big they are. only if you move them to a different partition would you care, but in that case as you hover over the other partition, you'd want the folder to become helium or hydrogen light, so you would need to exert more force to get it to touch down in the folder. oh the humanity!
You can emulate pressure sensitivity on Android by tracking the change in the radius of the touch point. With a little effort, it can be made to be nearly identical to the iOS system. I'd go into it or link you to some code, but it was like a decade ago that I last did it. I just remember I got it to work and it was fun.
I still miss it. Worse was they didn't just remove the hardware on newer models, but older models that did have the hardware available had the functionality removed overnight by an iOS update. If I recall it was over some licensing/patent dispute. (plus the feature itself was somewhat polarizing, not everyone found it intuitive)
Interesting concept, but it feels difficult to use. I do think it's a cool demo!
One conceptual issue I noticed with using it is that force touch requires pressure in the opposite direction of how I would understand weight and mass. It feels more like... I'm trying to think of a physical example, trying to force down something with buoyancy. I also expected the weight to affect how fast I needed to drag my finger, but once I exerted enough downward pressure, both heavy and light objects moved the same.
Yup thats a legit issue, built in the design. With your hand over the touch surface there is only one direction of the Z axis that you can apply force. I tried playing with other ideas too. Like , "sticky" where you apply some extra force to "pick it up" or adhesive , where you apply force "to stick it" to the canvas.
Nice experiment!
I agree that providing something more tangible than just a number would be beneficial for some operations. But I think it would get annoying quickly. Having difficulty moving a "heavy" file is the opposite of a good user interface. Every manipulation should require as little mental and physical effort as possible. Apart from that, I can't apply force with my mouse — it just clicks.
I believe a purely visual approach could work well. For example: every file icon has the same front area (basically the rectangles we have now), but visually extends to the back with some sort of stylized 3D effect, according to file size. So a small text file looks like a thin sheet of paper, a 10MB file might look like it's made of thick cardboard, a 2GB video looks like a box with considerable depth. The scaling should probably be logarithmic, not linear, to work well with human perception.
You are right. This is just an exploration. The main motivation was to play with the pressure input from Apple Taptic engine. I was trying to applying "physical" properties to virtual objects.
Designing for visual is what is missing, I would want to work on the visual plus audio to compliment the "heaviness" I think someone else also mentioned in the comments on experiments with audio to express physicality to
I don't think the visual approach does much for interactions, so that's a separate concern. This is more about feel. E.g. inertia applied to CoG, so when you grab it outside CoG, large files will tilt more as you drag. Light files would probably start spinning even.
Now, applying inertia for the movement itself would be annoying. Please don't do that. :)
You talk about interactions. In my mind the useful part is the added information density which comes easy (we are good at judging object sizes visually, not by numbers). I also want this information before interacting with the files. The value is knowing in advance, at a glance, what you're dealing with.
When you mention CoG, spinning etc. it becomes game-like — not something I would want in a productivity focused interface. But if you're developing a game or a toy experiment, go for it.
Thats the micro interaction that needs to be explored with iteration. The taptic engine pressure response (IMO) is not the best , but applying some heuristic ux designs I think we can approximate CoG and inertia.
I’ve often wanted a setup where it became physically harder to send an email to me the more unread ones I have to deal with. Like having to cram an extra letter into a pigeon hole that’s already full.
Or maybe make the Slack send button extra hard to press when sending messages after hours. Like you need to apply all the pressure and sustain that pressure over some time and only then the message goes. The emergency and anxiety is built into the interface X-D
Heavy is relative. If you're working with videos, everything is heavy in terms of file size relative to most files. Yet, a small text file could be just as important as any video.
I think this is a fun thought experiment that is fundamentally a bad product idea.
I think this type of concept is worth exploring. Side channel feedback to the operator of a machine is getting less noticeable. Hard drives don't whirr and click like they used to. Cars don't have transmissions that shift.
When you pick up a physical object with your hands, you don't assume the heavier the object, the more important it is. Same with file size.
But if you pick up your carton of eggs every morning you'll know if you have enough left to make an omelette.
If you make a backups it would be nice feedback to feel it weigh about what you expected. When making room on a disk you could juggle a few folders to feel if they'll fit or not.
There was some advanced facility (nuclear reactor? particle accelerator?) that laid microphones near the machinery and put various speakers in the ceiling of the control room; helped precisely detect and pinpoint problems immediately.
That said I'll prefer just seeing the size of the file or folder in bytes as a number.
I'm personally more interested in feeling other system metrics, like network traffic or memory bandwidth.
I've always thought it would be neat if the accelerator pedal on cars had some sort of force feedback that was proportional to the amount of power the engine is putting out. That way the driver would be able to feel how hard they're demanding the car to work, and hopefully they would adjust their driving habits to go slower on steep hills, not hard accelerate out of traffic lights, etc.
Yeah! I liked seeing the "miles per gallon" meter that some cars have for that purpose. IIRC driving habits account for at least 10% of fuel efficiency losses, and by my napkin math, you could drop carbon emissions in the US by 5% if you 1. put such a meter in every car and 2. drivers heeded and learned from it.
While modifying pedals is risky, maybe you could take an OBD-II data stream and turn instantaneous power output into sound, or vibration... or lower your music volume the harder you push it...
I understand. My motivation was to play with the pressure input with "heaviness". Fundamentally , pressure is a continuous input that becomes harder to perform progressively. We can apply this characteristic ins some other relationship too.
Hearing the HDD back in the day was important to understand whether the computer was working; It seemed like a loss when we moved to SSD, but SSDs are so fast that sound isn’t a necessary sensor anymore.
Its true , the interaction only works when its surrounded with other interaction. Moving all the heavy files in an video editing bay with this much pressure will just accelerate carpal tunnel.
I was interested in exploring to give more physicality to the virtual objects , they are "opened" , they "moved " I was playing with the idea if they can be "lifted" or be able to "stick" with pressure etc.
The size of a file has nothing to do with its importance. But it's a valuable piece of information when you're shoveling data around.
I agree that it is relative, but disagree with your conclusion. I think the relativity you have in mind is what we normally think of as a setting.
A box of large rocks feels different than a box of pebbles. Maybe this could be simulated.
See "Sonic Finder" for the Macintosh.[1] Heavy folders made bigger thumps when opened, or dragged and dropped. It was not widely used and disappeared. I did try it once. Cool, but not useful.
[1] http://sonify.psych.gatech.edu/~ben/references/gaver_the_son...
Thanks for sharing it .
Very interesting, I'm wondering how that would feel in XR!
I already have quite a few demos with manipulating files, e.g. Immersive file browser (via remote WebDAV directories) https://video.benetou.fr/w/rHZTnX5MnHdWvWTPa2Rsw4 so curious how I could try that there.
In WebXR there is no pressure value but maybe more fingers could be needed. Also maybe for heavier files there could be some "lag" where the cube representing the file does follow the pinched fingers but with some delay proportional to the file size. Any suggestion welcomed!
Check this project out : https://www.youtube.com/watch?v=4ev0w_tbuw8
Neat, I'd happily tinker with one but that's quite a demanding setup. From what I can tell from ManusVR and similar haptic devices there is no consumer driven cost dropping curve, unlike screen resolution thanks to mobile phones, that allow these to benefit from the same miniaturization and democratization as other things (screen, CPU/GPU, IMUs and other sensors). Maybe some MEMS breakthrough will enable cheap high resolution haptics but for now it remains bulky and expensive.
I wish more VR headsets would borrow from Windows Mixed Reality which has an analog grip value.
Controllers do have grip values, and one can measure the distance between index and thumb (for pinch).
I don't understand, does any controller other than Valve Knuckles measure individual fingers? What other headset supports that?
Awesome. I want to marry this with a concept I had a while ago around a computer keyboard with pressure sensitivity that adjusted the font size in proportion to how hard you struck the keys.
I like the idea of being able to enter into BILLY MAYS MODE just by furiously typing.
You might have to mess around with the software to figure out the details, but from a hardware perspective a hall effect keyboard should be at least able to infer the speed a key is travelling at when bottoms out.
Related: https://twitter.com/neilsardesai/status/1398665232529215492
noice.
That was entertaining to try as a demo, but I did not enjoy the UX of it because I don't have a proper sense of how much force I'm using on a touchpad. I have poor dynamic range in the amount of force I exert there.
My heavy files do feel heavy because it takes ages to move them through my painfully slow storage media. Can't say I'm a fan. It's a fun concept though.
I think exposing more of the internals helps. Consider what a mechanic can extract from the sound of an engine: in that sound a lot of information about the state of that engine is encoded. And it is a lot more than some programmer could abstract by simply having the motor vibrate if it is okay and have it not vibrate if it isn't. This however means you have to trust the user they will devolpe a sense of meaningful differentiation.
Physical objects interact with their environment in certain ways as a reflection of what they are, it is certainly an interesting question how this could be extended into the digital domain.
Well, they already do in that large files take more time to copy which subconsciously discourages moving them.
But they don't take more time to move (on the same partition). Also it would be nice to see at a glance if a folder contains a lot of data (number of bytes, not files) before starting an operation.
if you are moving a folder of files within a partition (hey, this was your idea) then you don't care how many or how big they are. only if you move them to a different partition would you care, but in that case as you hover over the other partition, you'd want the folder to become helium or hydrogen light, so you would need to exert more force to get it to touch down in the folder. oh the humanity!
You can emulate pressure sensitivity on Android by tracking the change in the radius of the touch point. With a little effort, it can be made to be nearly identical to the iOS system. I'd go into it or link you to some code, but it was like a decade ago that I last did it. I just remember I got it to work and it was fun.
Apple ditched the pressure sensitive screens like 6 years ago.
I still miss it. Worse was they didn't just remove the hardware on newer models, but older models that did have the hardware available had the functionality removed overnight by an iOS update. If I recall it was over some licensing/patent dispute. (plus the feature itself was somewhat polarizing, not everyone found it intuitive)
extremely useful in gaming even!
Thanks !
Interesting concept, but it feels difficult to use. I do think it's a cool demo!
One conceptual issue I noticed with using it is that force touch requires pressure in the opposite direction of how I would understand weight and mass. It feels more like... I'm trying to think of a physical example, trying to force down something with buoyancy. I also expected the weight to affect how fast I needed to drag my finger, but once I exerted enough downward pressure, both heavy and light objects moved the same.
Yup thats a legit issue, built in the design. With your hand over the touch surface there is only one direction of the Z axis that you can apply force. I tried playing with other ideas too. Like , "sticky" where you apply some extra force to "pick it up" or adhesive , where you apply force "to stick it" to the canvas.
It occurred to me that if you think of the force exerted as a car jack, it makes more physical sense.
Try it yourself at https://pressureinteraction.netlify.app
FYI : This only works on Mac Safari with Taptic touch pad :/
If it wasn't for Windows 11's already horrible performance I'd love this idea.