WebGL Benchmark - What Tests Should I Create?

Question

(I'm not sure if this should be on https://softwareengineering.stackexchange.com/ or not, please comment if you think so)

I am going to create a reference sample for WebGL implementations for my undergraduate dissertation. I'm not sure which tests I should create, and if I should only measure frames in seconds, if I could get some useful data for comparison ...

Currently, I was just thinking about tests like this:

• 1 one-color object
• 1 multi-color object
• 1 textured object
• 1 textured blend object
• 1 textured object with zipper
• 1 textured object with multiple lighting points
• 1 scene processed by texture and using it on another object
• 1 model animation
• all tests with a large number of objects: 50, 500, 5000
• change shaders / programs for rendering (one, two, several times)

This will lead to 40 different tests, but I'm not sure if they are important tests for performance.

A friend suggested testing a complex shader, but since the shader runs on graphics hardware, there should be no difference in the working java application, right?

In any case, I feel that JavaScript performance is the main bottleneck in WebGL.

Update

I finally did my test. After discussing how the reference should work, I created the following: http://martin.cyor.eu/benchmark

Answer 1

What you choose to compare should depend on what you intend to do with the score (or who the target audience is for the assessment and what they intend to do with them); apparently, they should influence some decision - you should start with this decision and step back from it in order to find out what the standard should measure.

I would say that your proposed list falls into the "synthetic" end of the comparison spectrum; they look like a thing that would be more useful for a WebGL / browser developer with a profiler than for someone who is trying to choose between WebGL implementations / platforms to decide which application runs best (see the OpenGL benchmarking history, where all then the nonsense that was previously required for the number of synthetic polygons per second, until SPECviewperf appeared and realistic realistic test cases were introduced).

Answer 2

I agree with @timday that you should reject your investigation to something “real”, and, as you suggested in the commentary, you might want the story to relate to a solution between a desktop or browser.

This is exactly what I'm working on right now. My client has a visualization application that currently runs on the Windows desktop. A typical scene for them has 500,000 triangles, lots of textures and transparency. Currently, their users are not inclined to install the viewer - they usually work in corporate environments where system administrators control what is installed on their computers. And several users would prefer to run visualization on their iPads, where the viewer will not work anyway. Therefore, my client wants to know whether WebGL will solve its platform problems - it does not matter that no browser officially supports WebGL, and that neither IE nor the iPad announced any support.

Remember that any benchmarks you make are relatively meaningless because you are measuring a moving target. Browser manufacturers are working hard to implement WebGL, and they update their beta version frequently. Not only do they work on implementing WebGL, but they have to worry about browser security issues and the overall pipeline flow. This video talks about some issues (and gives you an idea of ​​what to watch). Additionally, performance may vary depending on your OS and graphics hardware.

As you pointed out, as soon as WebGL runs on graphics hardware, it should run as fast as a desktop application. Your tests should try to confirm this, and then you should try to measure where performance is lost as a result of being in the browser. I feel that Javascript itself is not a bottleneck, simply because there is not much Javascript to execute (and it's pretty fast these days). However, as described at the end of the aforementioned video, there may be inefficiencies that occur in Javascript-C ++ bindings, request validation, flow control, and what not. On the other hand, browser makers (at least Google) are working to smooth out these excesses.

One of the things I noticed is not frame rate / performance issues (in my current test, I can display 500,000 textured triangles at 30 frames per second), but these frameworks don't seem very consistent and these frames seem time discarded from time to time. I suspect, but I don’t know, whether this is due to the relatively simple setInterval() method or running the animation in Javascript. (Mozilla mozRequestAnimationFrame may be a way to handle this better).

Although I do not know how useful the above would be for your thesis, it seems to me that you have a rich topic, and you should do more than just write simple tests. Perhaps you should start with some tests, compare browser and desktop performance, and then try to learn best practices for not only the solution between the browser and the desktop, but also for writing WebGL applications.

There are many WebGL infrastructures. I tried a couple, and was very impressed - there is something to learn from them. Depending on your interests and requirements of the thesis, you may also be interested in a comparative analysis.

Be that as it may, I suspect that there is a large community of potential WebGL users who will starve for the information you are about to research.

Answer 3

I do not regularly visit this site, so it seems funny to me that I just read your question, because I am also writing my own topic on this subject. A month ago, I was thinking about the same problem and decided not to compare different WebGL frameworks with each other mainly because I think that the main difference may lie in the mathematical library that they use. As you said, I also came to the conclusion that there should be no difference in the parts running on gpu. Comparing WebGL to desktop graphics was something I hadn’t thought of, but it might be useful. Keep in mind that the latest versions of WebGL from Chrome and Firefox use DirectX through ANGLE for Windows by default.

btw here is some kind of webgl test: http://webgl-bench.appspot.com/

Good luck