Fist detection with jump movement SDK V2

Question

I want to check if the Hand in the frame of motion is currently a fist.

Typically, the proposed method is to search for hand.grabStrength with a value of 1. The problem is that the value goes to 1 even with a "Claw-Like" Hand or everything else with very slightly twisted fingers.

Another approach is to check each finger if it is extended . But it has a similar problem, the fingers are only considered extended if they are completely straight. Therefore, even if I check that all fingers are not extended, the same problem arises as above (clawed hands are recognized as being caught).

Combining the two methods also does not solve the problem, which is not surprising given that they both suffer from the same problems.

Now we have all the bones of each finger, with positions and everything else. But I don’t know where to start with mathematics to determine if a finger is twisted.

I basically have this setting:

 var controller = Leap.loop(function(frame){ if(frame.hands.length>0){ //we only look at the first available hand var hand = frame.hands[0]; //we get the index finger only, but later on we should look at all 5 fingers. var index = hands.fingers[1]; //after that we get the positions of the joints between the bones in a hand //the position of the metacarpal bone (ie the base of your hand) var carp = index.carpPosition; //the position of the joint on the knuckle of your hand var mcp = index.mcpPosition; //the position of the following joint, between the proximal and the intermediate bones var pip = index.pipPosition; //the position of the distal bone (the very tip of your finger) var dip = index.dipPosition; //and now we need the angle between each of those positions, which is where i'm stuck } }); 

So, how do I get the angle between two of these positions (carp to mcp, mcp to pip, pip to dip)? Any ideas?

Answer 1

Well, I think I found a kind of working approach to detect the actual fist, not the claw.

First of all, instead of the positions of the joints, we need distance vectors for each bone.

We then calculate the Dot product between Metacarpal and the proximal bone, as well as the Spot product between the Proximal and the intermediate bone. We can ignore the distal bone, it does not change the result too much.

We summarize all calculated point products (total 10) and calculate the average value (we divide by 10). This will give us a value from 0 to 1. The fist is below 0.5, and everything higher than basically is not a fist.

In addition, you can also check the number of extended fingers on your hand and check if it is 0. This ensures that “Thumbs-up” and similar 1-digit poses are not recognized as “Fist”.

Here is my implementation:

 const minValue = 0.5; var controller = Leap.loop(function(frame){ if(frame.hands.length>0) { var hand = frame.hands[0]; var isFist = checkFist(hand); } }); function getExtendedFingers(hand){ var f = 0; for(var i=0;i<hand.fingers.length;i++){ if(hand.fingers[i].extended){ f++; } } return f; } function checkFist(hand){ var sum = 0; for(var i=0;i<hand.fingers.length;i++){ var finger = hand.fingers[i]; var meta = finger.bones[0].direction(); var proxi = finger.bones[1].direction(); var inter = finger.bones[2].direction(); var dMetaProxi = Leap.vec3.dot(meta,proxi); var dProxiInter = Leap.vec3.dot(proxi,inter); sum += dMetaProxi; sum += dProxiInter } sum = sum/10; if(sum<=minValue && getExtendedFingers(hand)==0){ return true; }else{ return false; } } 

Although this works as it should, I doubt it is the right and best approach for finding Fist. So please, if you know the best way, post it.

---

The solution works just fine, any chance you could explain why you are dividing by 10 and why minValue is 0.5? Thanks!

Well, that doesn't work, to be honest. I will soon start working on a small project whose goal is to improve fist detection with Leap Motion.

As for your questions, we divide the amount by 10, because we have 2 bone joints for each finger with 5 fingers. We want to get the average value from the sum of all these calculations, because not all fingers will be angled the same. Therefore, we want some value to include all these values ​​in a single whole: the average value. Given that we have only 10 calculations (2 for each finger, 5 fingers), we divide the sum of these calculations, and there we go. We get a value from 0 to 1.

Regarding minValue: Trial & Error. In my project, I used a value of 0.6. This is another problem with this approach: ideally, a flat arm should be a value of almost 0, and a fist should be 1.

Answer 2

I know this is an old topic, but if you guys are still around, the answer might be simpler just sphereRadius() ;