Launching today
Airpoint
Touchless computing with hand tracking and AI agents
67 followers
Touchless computing with hand tracking and AI agents
67 followers
Airpoint lets you control any computer with your hands without any extra hardware. Hand motion and gestures are translated in real time into cursor control, shortcuts, and app/games interactions. Built for anyone whose hands are busy or unavailable: crafters, gamers, builders, devs and more. Take advantage of VR/AR without headsets or extra hardware. The input layer also makes it possible for AI agents (OpenClaw) to autonomously control computers. SDK available to include in apps and games.
Airpoint
Give your OpenClaw agents "hands" by connecting Airpoint to your OpenClaw setup 🦞 and let your agents control your computer (not just the browser!) 😉
Airpoint
@marioandf Love the Minority Report vibes; gesture control without hardware sounds game-changing. Just a simple q; how does Airpoint handle gesture accuracy in real world scenarios like bright lighting or finger fatigue during long sessions, and what's your roadmap for multi-monitor or AR integration?
Airpoint
@swati_paliwal Great questions.
We tackled accuracy by combining algorithms with ML models trained with over 1M samples in different scenarios to predict with >98% precision the expected outcome at each frame.
For finger fatigue, we carefully designed the gestures to be as ergonomically as possible and are expected to work from a resting wrist position too. We currently do not recommend throwing away the mouse or keyboard yet 😅. In long sessions they are still useful. But when working from a distance or working on physical world tasks, it is hard to beat to just point at the monitor.
We already support multiple monitors! 🙌🏻
We are already exploring how could we integrate with AR, since our focus is still not adding extra hardware.
Thanks for the feedback
Product Hunt
Airpoint
@curiouskitty Great question and exactly the main challenge of the project.
First or all we decided to go with index finger tracking for cursor to reduce the need to move the whole hand to move the cursor. Even a resting hand moving index finger is tracked and translated to the cursor.
We decided to add many control layers to for user intent: Fingers pointing at the screen, Voice commands to Resume/Pause/Stop tracking and hand gestures or shortcuts to quickly toggle between them too.
Exactly hitting the right balance between speed, stability and false positives was hard and required a lot of testing and training of AI models. Most webcams are fixed at 30Hz so we do need smoothening, stabilizing and prediction algorithms for a good user experience. Of course that meant sacrificing small portions of precision and responsiveness but the key was to keep the tradeoffs under the human perceptible thresholds to keep the advantages. We kept most of this algorithms accessible in settings though for different user preferences.
However we keep using the latest data and feedback for continuous training of the AI/ML models in order to keep improving with each iteration.
Product Hunt
I personally think this is insane and very promising. I've already started relying more on dictation, if I pair that with hand controls I don't really think we need to be sitting close to screens anymore.
Awesome job, Andrew! How did you get it so responsive?
Airpoint
@gabe Thanks for the comments Gabe! It was a real challenge considering most computers have single decent camera at 30Hz. Took a LOT of iterations through different detection methods and testing for real world responsiveness, but I think we got there and can keep improving with more data and feedback 😊.
Product Hunt
@marioandf I'm very excited to play with this over the weekend :) Congrats again and really nicely done getting it so fluid.
Airpoint
@gabe Thanks. Let me know if there are any suggestions 👍🏻
NextBase
This is incredible! So so good. Love the idea of mapping the hand movements to not just the mouse but to also have shortcuts such as voice.
Airpoint
@imbhargav5 Thanks! That is the idea! Made them customizable to suit everyone´s needs!
Product Hunt
cc @kwindla who founded the company that Minority Report was based on! This is wild.
Airpoint
@kwindla @rajiv_ayyangar 🤯 I had no idea! That is wild. Just researched @kwindla and he is a genius! For sure an inspiration on the project.
Covalent
this is super sick! I think your UI for controls are super cool.
Airpoint
@hem_juvvaladinne1 Thanks! Took a lot of time to polish designing the gestures 😅