How to learn mouse movement?

Question

I have been trying to develop a tool for synthesizing human-like mouse movement in my application over the past few weeks. In the beginning, I used simple methods, such as polynomial and spline interpolation, however, even with a little noise, the result still did not appear human-like enough.

In an attempt to fix this problem, I studied ways to apply machine learning algorithms on real biometric mechanisms of mouse movement to synthesize mouse movements, studying recorded real human ones. Users will compile a profile of recorded movements that will train = program for synthesis purposes.

I searched for several weeks and read several articles on the application of reverse biometry in the generation of mouse dynamics, such as reverse biometrics for mouse dynamics ; however, they seek to focus, however, on creating realistic time from randomly generated dynamics , while I was hoping to create a path from specifically from A to B. In addition, I still need to actually come up with a path, not just a few speakers measured from one.

Does anyone have a few pointers to help noob?

Testing is currently being done by recording movements, and I and several other developers are watching the playback. Ideally, the movement will be able to deceive both the automatic biometric classifier and the real, lively, breathing Homo sapien.

Answer 1

First, I assume that you are recording the movements of human mice from A to B. Because otherwise, it seems impossible to me to synthesize a model for such a movement.

Secondly, what about measuring deviations from the “direct” path, possibly with respect to time. I really suspect that the movements look different for different angles, path lengths, etc., but maybe you can try a normalized model first so that you just stretch (in space and time) and rotate as you need .

Thirdly, training. It would be easiest to just have a set of real moves (in the form in which I discussed above), and a sample from this collection. Rate how it looks. If you really need a probabilistic model, you should evaluate which models are suitable. Is it enough to blur the direct path with a Gaussian noise, the parameters of which you will learn from your training set? Or some kind of (sin-) wave deviation? Or separate models for “getting closer to the button” and “final fixes”. Fitts law may be useful for evaluation.

Answer 2

Fitt’s law gives a very good estimate of the time required to position the mouse pointer. There is a simple explanation in the derivation section, I think you could use this as one of the main building blocks of your application. Start with large movements, place some attacks in the direction and length of the movement, then make a small corrective movement and so on ...

Answer 3

This question reminded me of a website that I knew many years ago, so I visited it and found this in-depth discussion on the topic .

The timing is so similar that it makes me think that this issue is somehow related. In fact, someone in the subject is linked to the same article as you. If it’s not connected, well, there is a link to a lot of people who are discussing exactly what you are thinking about.

Answer 4

I do not think the problem is that it is well defined. There is an important concept, not mentioned so far, that is context. The mouse movement on my screen when Chrome has focus is significantly different from the movement when Vim has focus.

Answer 5

The way you move the mouse depends on the type of device, type of action, user interface elements, getting to know the user interface, the speed with which the user is trying to complete his task, the user's skills, the user's initial failure (for example, misses), the user's emotional state (like many others factors). Do you plan to create several transition strategies to fit different contexts? Also, how well do you know the algorithm you are trying to trick? I assume that it is not widespread, or simply programmed directly against this algorithm.

If a person looks at a path, they can identify the state associated with the path strategy, and may be more prone to deception if they identify it as a state of the man (for example, the user is in a hurry, skips, quickly closes the pop-up that appears, tries again more slowly). The user interface comes into play not only with size and position. I often quickly point to the toolbar, then move the options until I get to my goal. Another example is that I usually dwell on menu items while I am viewing my goal or hovering over the text that I am reading. Are you trying to imitate human behavior or just mouse movement (because I think they are connected to the hip)?

Answer 6

Do you want to simulate a human-like mouse movement because you are doing real-time online training for your game? If your training sequences are static, just record the mouse movements and play the sound effect of the mouse click when you click the mouse button. No mouse movement will feel "real enough" more for you than your own.

Personally, I believe that software experts move their mice too quickly into training videos. I prefer the approach made by the screencasting software that I saw that always moves the mouse linearly from point A → B. The trick was that each mouse step made in the video always took the same time regardless of distance, say, 3/4 seconds, followed by the sound effect of a mouse click.

I believe that they moved the mouse in this way, because then the viewer could foresee the mouse landing zone in the direction and speed of mouse movement at the beginning. In a workout situation, I believe that regular movements like this are softer on the eye and may be easier to maintain / remember.

Answer 7

Have you considered adding mouse tracking to the application so that you essentially record how the user moves the mouse and then analyzes the records?

I have not studied this recently, but I believe that the MouseListener in the Swing application gets the information you need.