JavaFX Moving 3D objects with the mouse on a virtual plane

Question

Since I created my first 3D game in JavaFX, where you could assemble ships from parts with the mouse. This presents a problem because JAVAFX does not seem to have built-in methods that work to convert 2D PerspectiveCamera coordinates to 3D scene space.

Here is an idea of ​​what I'm trying to achieve. The block moved by the mouse should move along an imaginary plane, which always rotates 90 relative to the camera: I tried to solve the problem of trigonometry without much success. I did not attach the code snippet as I am looking for a more general mathematical solution, but will provide it if necessary.

All help would be appreciated!

Desired Result:

Answer 1

As @ jdub1581 points out, Camera is the key to linking mouse movements to your 3D objects in the scene.

For starters, we know about the public PickResult API, which allows us to select a 3D object with the mouse based on some ray-tracing methods made from the camera’s position.

But once we have an object, moving is another problem.

Looking for a solution to this problem (moving 3D objects with a 2D mouse in 3D space) a while ago I found Camera3D in the Toys project in the OpenJFX repository.

It has a promissing method called unProjectDirection :

 /* * returns 3D direction from the Camera position to the mouse * in the Scene space */ public Vec3d unProjectDirection(double sceneX, double sceneY, double sWidth, double sHeight) { } 

Since you asked for a mathematical explanation, this method uses the trigonometry you were looking for. This will give you a 3D vector based on the coordinates (x, y) of the mouse using the private Vec3d class (which we can replace with the public Point3D ):

 double tanOfHalfFOV = Math.tan(Math.toRadians(camera.getFieldOfView()) * 0.5f); Vec3d vMouse = new Vec3d(tanOfHalfFOV*(2*sceneX/sWidth-1), tanOfHalfFOV*(2*sceneY/sWidth-sHeight/sWidth), 1); 

Some additional transformations are used to obtain a normalized vector in the coordinates of the scene.

The next step converts this normalized vector into real coordinates, simply using the distance from the camera to the object, specified by the selection result, and converts the position of the object.

Basically, this piece of code describes the whole process of dragging an object:

  scene.setOnMousePressed((MouseEvent me) -> { vecIni = unProjectDirection(me.getSceneX(), me.getSceneY(), scene.getWidth(),scene.getHeight()); distance=me.getPickResult().getIntersectedDistance(); }); scene.setOnMouseDragged((MouseEvent me) -> { vecPos = unProjectDirection(mousePosX, mousePosY, scene.getWidth(),scene.getHeight()); Point3D p=vecPos.subtract(vecIni).multiply(distance); node.getTransforms().add(new Translate(p.getX(),p.getY(),p.getZ())); vecIni=vecPos; distance=me.getPickResult().getIntersectedDistance(); }); 

And this is a complete working base example:

  public class Drag3DObject extends Application { private final Group root = new Group(); private PerspectiveCamera camera; private final double sceneWidth = 800; private final double sceneHeight = 600; private double mousePosX; private double mousePosY; private double mouseOldX; private double mouseOldY; private final Rotate rotateX = new Rotate(-20, Rotate.X_AXIS); private final Rotate rotateY = new Rotate(-20, Rotate.Y_AXIS); private volatile boolean isPicking=false; private Point3D vecIni, vecPos; private double distance; private Sphere s; @Override public void start(Stage stage) { Box floor = new Box(1500, 10, 1500); floor.setMaterial(new PhongMaterial(Color.GRAY)); floor.setTranslateY(150); root.getChildren().add(floor); Sphere sphere = new Sphere(150); sphere.setMaterial(new PhongMaterial(Color.RED)); sphere.setTranslateY(-5); root.getChildren().add(sphere); Scene scene = new Scene(root, sceneWidth, sceneHeight, true, SceneAntialiasing.BALANCED); scene.setFill(Color.web("3d3d3d")); camera = new PerspectiveCamera(true); camera.setVerticalFieldOfView(false); camera.setNearClip(0.1); camera.setFarClip(100000.0); camera.getTransforms().addAll (rotateX, rotateY, new Translate(0, 0, -3000)); PointLight light = new PointLight(Color.GAINSBORO); root.getChildren().add(light); root.getChildren().add(new AmbientLight(Color.WHITE)); scene.setCamera(camera); scene.setOnMousePressed((MouseEvent me) -> { mousePosX = me.getSceneX(); mousePosY = me.getSceneY(); PickResult pr = me.getPickResult(); if(pr!=null && pr.getIntersectedNode() != null && pr.getIntersectedNode() instanceof Sphere){ distance=pr.getIntersectedDistance(); s = (Sphere) pr.getIntersectedNode(); isPicking=true; vecIni = unProjectDirection(mousePosX, mousePosY, scene.getWidth(),scene.getHeight()); } }); scene.setOnMouseDragged((MouseEvent me) -> { mousePosX = me.getSceneX(); mousePosY = me.getSceneY(); if(isPicking){ vecPos = unProjectDirection(mousePosX, mousePosY, scene.getWidth(),scene.getHeight()); Point3D p=vecPos.subtract(vecIni).multiply(distance); s.getTransforms().add(new Translate(p.getX(),p.getY(),p.getZ())); vecIni=vecPos; PickResult pr = me.getPickResult(); if(pr!=null && pr.getIntersectedNode() != null && pr.getIntersectedNode()==s){ distance=pr.getIntersectedDistance(); } else { isPicking=false; } } else { rotateX.setAngle(rotateX.getAngle()-(mousePosY - mouseOldY)); rotateY.setAngle(rotateY.getAngle()+(mousePosX - mouseOldX)); mouseOldX = mousePosX; mouseOldY = mousePosY; } }); scene.setOnMouseReleased((MouseEvent me)->{ if(isPicking){ isPicking=false; } }); stage.setTitle("3D Dragging"); stage.setScene(scene); stage.show(); } /* From fx83dfeatures.Camera3D http://hg.openjdk.java.net/openjfx/8u-dev/rt/file/5d371a34ddf1/apps/toys/FX8-3DFeatures/src/fx83dfeatures/Camera3D.java */ public Point3D unProjectDirection(double sceneX, double sceneY, double sWidth, double sHeight) { double tanHFov = Math.tan(Math.toRadians(camera.getFieldOfView()) * 0.5f); Point3D vMouse = new Point3D(tanHFov*(2*sceneX/sWidth-1), tanHFov*(2*sceneY/sWidth-sHeight/sWidth), 1); Point3D result = localToSceneDirection(vMouse); return result.normalize(); } public Point3D localToScene(Point3D pt) { Point3D res = camera.localToParentTransformProperty().get().transform(pt); if (camera.getParent() != null) { res = camera.getParent().localToSceneTransformProperty().get().transform(res); } return res; } public Point3D localToSceneDirection(Point3D dir) { Point3D res = localToScene(dir); return res.subtract(localToScene(new Point3D(0, 0, 0))); } public static void main(String[] args) { launch(args); } } 

This will allow you to select and move the sphere on the stage: