I am trying to smoothly scroll some images through a window using DirectX11 through SharpDX in a WPF application.
A bit of background:
Images are time-varying returned signals that were loaded from a file and loaded into D3D as an array of textures, and the image itself is displayed as a series of textured quads (a series of adjacent rectangles in a very long horizontal line). The viewport is configured so that the left and right edges represent time offsets, and the signal is displayed for the period of time that falls between these times. The implementation of DirectX is very simple; square vertices are generated once, and a very small buffer for each frame contains a simple 2D world transform that updates the scale and translation in accordance with the current visible range / magnification, etc. As far as I can tell, the D3D implementation is not part of the problem I am facing - it is very easy to set up and seems to run very fast (as it should be), and although I have some interesting things (streaming textures from disk as needed), I disabled all of them, and I run very simple (small) textures while I try to solve the problem that I have.
Problem:
Scrolling the image is not smooth when the visible time range is animated. The image "trembles" most of the time, and frankly, it looks terrible (when it does not tremble, it looks great).
Setup:
DirectX is passed to D3DImage, with a bit of work being done behind the scenes to get DX11 working - this code is taken from https://sharpdxwpf.codeplex.com/
There are several D3DImages (up to 4) located in the grid (I tested two, both of which contain signals for the same time period and are animated together).
These D3DImages are drawn by DrawingVisual (which is hosted by a custom FrameworkElement), where they are used as the ImageBrush source. The frameworkelement object starts the render as needed, and the drawing render handles the D3D render call and draws a rectangle to fill the control with the D3DImage brush.
The time range value is animated using WPA DoubleAnimation. The visual images currently displayed are tied to this value through INotifyPropertyChanged and trigger rendering (via InvalidateVisual) with every change.
Drawing a visual visual code caused by a change in the "Position" value (beginning of the visible time range):
// update scene per-frame buffer first, with world-view transform using (DrawingContext dc = RenderOpen() { _scene.Renderer.Render(_draw_args); _image.Invalidate(); dc.DrawRectangle(_brush, null, new Rect(viewer.RenderSize)); }
What I tried:
I tried several things (and searched many times) to try to determine if the problem was caused due to the erratic rendering request time, or if the problem is related to the rendering process.
- Interaction with CompositionTarget.Rendering First, driving the update of the position value through CompositionTarget.Rendering and leaving the rest of the process as it is (that is, the elements react to changing this value):
(needless to say, this is a very "test" code):
Stopwatch rsw; long last_time = 0; void play() { rsw = new Stopwatch(); last_time = 0; rsw.Start(); CompositionTarget.Rendering += rendering; } void rendering(object sender, EventArgs e) { double update_distance = rsw.ElapsedMilliseconds - last_time; Position += 10 * update_distance; last_time = rsw.ElapsedMilliseconds; }
The results are worse than a simple double animation.
- Using DispatcherTimer. As above, using a timer in combination with a stopwatch to judge elapsed time is slightly better. Worse than the results, with an interesting note that CPU usage dropped from about 7% (half the core) to 0.7%.
The original attempt, along with options 1 and 2, everyone is trying to update a single value, which then invokes a stakeholder render. Out of interest, I recorded the time difference between rendering requests as they reached the Drawing visualization - although DisptacherTimer actually gave the most consistent results there (both WPF animation and CompositionTarget dropped an odd frame), DisptacherTimer also gave the most nervous animation.
Image refresh, but not invalidation of the visual image. ImageBrush source update updates the displayed image without reprocessing DrawingVisual. This method gave very sharp results.
CompositionTarget.Rendering in the frame element itself. This gave the best batch results, but still not perfect. Jitter will happen and then dissipate, only to return again. In this approach, a structure element that holds DV connections until CompositionTarget.Rendering, and visual queries of the current position, which is animated independently. I could live with that approach.
Trying to wait until the D3D scene is filmed until the image is canceled (without noticeable improvement):
_scene.Renderer.Render (_draw_args);
_scene.Renderer.Device.ImmediateContext.End (sq);
while (! (_ scene.Renderer.Device.ImmediateContext.IsDataAvailable (q))) Thread.yield ();
_image.Invalidate ();
remarks:
- I really don't think this is a performance issue as such. My Dev machine has a good graphics card, 8 i7 cores, etc., And this is a simple rendering operation.
- It really looks like some kind of synchronization problem between D3D and WPF rendering, but I have no idea how to start learning this.
- If I have two images animating in parallel, jitter is much more pronounced on the first of two (usually).
- If I actively resize the window during the animation, the animation is completely smooth (despite the fact that a lot of extra work is being done, as the D3D size is constantly changing).
EDIT
I tried to isolate the problem as soon as possible, and the problem seems to be fundamental to how D3DImage is updated and rendered by WPF.
I changed the WPFHost example in the SharpDX Samples solution so that the simple triangle that displays is animated across the screen. This example is hosted on a DX10ImageSource, which is displayed by DPFCanvas in CompositionTarget.Rendering.
This example could not be simpler and something like “close to metal”, since you can get D3DImage rendering in WPF. One triangle, translated on the screen to a value calculated by the time difference between the renderers. Stuttering stays, coming and going, as if some kind of synchronization problem. This puzzles me, but essentially makes SharpDX unusable in WPF for any smooth animations, which is extremely disappointing.
If anyone is interested in reproducing this issue, SharpDX samples are available here: https://github.com/sharpdx/SharpDX-Samples
I made the following simple changes to the WPFHost example:
void IScene.Render() { ... EffectMatrixVariable wv = this.SimpleEffect.GetVariableBySemantic("WorldView").AsMatrix(); wv.SetMatrix(this.WorldViewMatrix); ... } void IScene.Update(TimeSpan sceneTime) { float x = (float)sceneTime.Milliseconds * 0.001f - 0.5f; WorldViewMatrix = Matrix.Translation(x, 0, 0); }
and in the shader Simple.fx:
float4x4 WorldViewTransform : WorldView; PS_IN VS( VS_IN input ) { PS_IN output = (PS_IN)0; output.pos = mul(input.pos, WorldViewTransform); output.col = input.col * Overlay; return output; }