TarsosDSP and multithreaded SurfaceView issue

Question

I use TarsosDSP to calculate the pitch frequencies in real time. It uses an AudioDispatcher that implements Runnable and publishes the results using the handlePitch method to use it in the main thread.

I use SurfaceView to draw this value when updating it. SurfaceView also requires another thread to draw on the canvas. So, I have 2 managed entities. I could not control the surface updating through one stream, getting the pitch values ​​from another stream (audiodispatcher).

I just want to use the cent value that I get in the handlePitch () method to update my drawing on the surface. But my application freezes. Any idea?

In MainAcitivity.java (onCreate (...))

myView = (MySurfaceView) findViewById(R.id.myview); int sr = 44100;//The sample rate int bs = 2048; AudioDispatcher d = AudioDispatcherFactory.fromDefaultMicrophone(sr,bs,0); PitchDetectionHandler printPitch = new PitchDetectionHandler() { @Override public void handlePitch(final PitchDetectionResult pitchDetectionResult, AudioEvent audioEvent) { final float p = pitchDetectionResult.getPitch(); runOnUiThread(new Runnable() { @Override public void run() { if (p != -1){ float cent = (float) (1200*Math.log(p/8.176)/Math.log(2)) % 12; System.out.println(cent); myView.setCent(cent); } } }); } }; PitchProcessor.PitchEstimationAlgorithm algo = PitchProcessor.PitchEstimationAlgorithm.YIN; //use YIN AudioProcessor pitchEstimator = new PitchProcessor(algo, sr,bs,printPitch); d.addAudioProcessor(pitchEstimator); d.run();//starts the dispatching process AudioProcessor p = new PitchProcessor(algo, sr, bs, printPitch); d.addAudioProcessor(p); new Thread(d,"Audio Dispatcher").start(); 

In SurfaceView.java (below code is run from the constructor)

  myThread = new MyThread(this); surfaceHolder = getHolder(); bmpIcon = BitmapFactory.decodeResource(getResources(), R.mipmap.ic_launcher); iconWidth = bmpIcon.getWidth(); iconHeight = bmpIcon.getHeight(); density = getResources().getDisplayMetrics().scaledDensity; setLabelTextSize(Math.round(DEFAULT_LABEL_TEXT_SIZE_DP * density)); surfaceHolder.addCallback(new SurfaceHolder.Callback(){ @Override public void surfaceCreated(SurfaceHolder holder) { myThread.setRunning(true); myThread.start(); } @Override public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) { // TODO Auto-generated method stub } @Override public void surfaceDestroyed(SurfaceHolder holder) { boolean retry = true; myThread.setRunning(false); while (retry) { try { myThread.join(); retry = false; } catch (InterruptedException e) { } } }}); protected void drawSomething(Canvas canvas) { updateCanvas(canvas, this.cent); //draws some lines depending on the cent value } public void setCent(double cent) { if (this.cent > maxCent) this.cent = maxCent; this.cent = cent; } 

UPDATE:

Mythread.java

 public class MyThread extends Thread { MySurfaceView myView; private boolean running = false; public MyThread(MySurfaceView view) { myView = view; } public void setRunning(boolean run) { running = run; } @Override public void run() { while(running){ Canvas canvas = myView.getHolder().lockCanvas(); if(canvas != null){ synchronized (myView.getHolder()) { myView.drawSomething(canvas); } myView.getHolder().unlockCanvasAndPost(canvas); } try { sleep(1000); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } 

Answer 1

If I understand your problem correctly, you have an independent source of events working on your own thread ( PitchDetectionHandler ) and SurfaceView , which you want to repaint into your own thread when an event occurs from the source. If so, then I think the whole idea with sleep(1000) is incorrect. You should keep track of actual events and respond to them, not sleep, waiting for them. And it seems that on Android, the easiest solution is to use the HandlerThread / Looper / Handler infrastructure:

Beware of errors in the following code; not only did not try, but did not even compile it.

 import android.graphics.Canvas; import android.os.Handler; import android.os.HandlerThread; import android.os.Message; import android.view.SurfaceHolder; public class SurfacePitchDrawingHelper implements Handler.Callback, SurfaceHolder.Callback2 { private static final int MSG_DRAW = 100; private static final int MSG_FORCE_REDRAW = 101; private final Object _lock = new Object(); private SurfaceHolder _surfaceHolder; private HandlerThread _drawingThread; private Handler _handler; private float _lastDrawnCent; private volatile float _lastCent; private final boolean _processOnlyLast = true; @Override public void surfaceCreated(SurfaceHolder holder) { synchronized (_lock) { _surfaceHolder = holder; _drawingThread = new HandlerThread("SurfaceDrawingThread") { @Override protected void onLooperPrepared() { super.onLooperPrepared(); } }; _drawingThread.start(); _handler = new Handler(_drawingThread.getLooper(), this); // <-- this is where bug was _lastDrawnCent = Float.NaN; //postForceRedraw(); // if needed } } @Override public void surfaceDestroyed(SurfaceHolder holder) { synchronized (_lock) { // clean queue and kill looper _handler.removeCallbacksAndMessages(null); _drawingThread.getLooper().quit(); while (true) { try { _drawingThread.join(); break; } catch (InterruptedException e) { } } _handler = null; _drawingThread = null; _surfaceHolder = null; } } @Override public void surfaceRedrawNeeded(SurfaceHolder holder) { postForceRedraw(); } @Override public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) { synchronized (_lock) { _surfaceHolder = holder; } postForceRedraw(); } private void postForceRedraw() { _handler.sendEmptyMessage(MSG_FORCE_REDRAW); } public void postRedraw(float cent) { if (_processOnlyLast) { _lastCent = cent; _handler.sendEmptyMessage(MSG_DRAW); } else { Message message = _handler.obtainMessage(MSG_DRAW); message.obj = Float.valueOf(cent); _handler.sendMessage(message); } } private void doRedraw(Canvas canvas, float cent) { // put actual painting logic here } @Override public boolean handleMessage(Message msg) { float lastCent = _processOnlyLast ? _lastCent : ((Float) msg.obj).floatValue(); boolean shouldRedraw = (MSG_FORCE_REDRAW == msg.what) || ((MSG_DRAW == msg.what) && (_lastDrawnCent != lastCent)); if (shouldRedraw) { Canvas canvas = null; synchronized (_lock) { if (_surfaceHolder != null) canvas =_surfaceHolder.lockCanvas(); } if (canvas != null) { doRedraw(canvas, lastCent); _surfaceHolder.unlockCanvasAndPost(canvas); _lastDrawnCent = lastCent; } return true; } return false; } } 

And then in your activity class you will do something like

 private SurfaceView surfaceView; private SurfacePitchDrawingHelper surfacePitchDrawingHelper = new SurfacePitchDrawingHelper(); ... @Override protected void onCreate(Bundle savedInstanceState) { ... surfaceView = (SurfaceView) findViewById(R.id.surfaceView); surfaceView.getHolder().addCallback(surfacePitchDrawingHelper); int sr = 44100;//The sample rate int bs = 2048; AudioDispatcher d = AudioDispatcherFactory.fromDefaultMicrophone(sr, bs, 0); PitchDetectionHandler printPitch = new PitchDetectionHandler() { @Override public void handlePitch(final PitchDetectionResult pitchDetectionResult, AudioEvent audioEvent) { final float p = pitchDetectionResult.getPitch(); float cent = (float) (1200 * Math.log(p / 8.176) / Math.log(2)) % 12; System.out.println(cent); surfacePitchDrawingHelper.postRedraw(cent); } }; PitchProcessor.PitchEstimationAlgorithm algo = PitchProcessor.PitchEstimationAlgorithm.YIN; //use YIN AudioProcessor pitchEstimator = new PitchProcessor(algo, sr, bs, printPitch); d.addAudioProcessor(pitchEstimator); // d.run();//starts the dispatching process <-- this was another bug in the original code (see update)! AudioProcessor p = new PitchProcessor(algo, sr, bs, printPitch); d.addAudioProcessor(p); new Thread(d, "Audio Dispatcher").start(); ... } 

Note that SurfacePitchDrawingHelper encapsulates most of the drawing-related logic, and there is no need for your MySurfaceView subclass (which I think is a bad idea).

The basic idea is that SurfacePitchDrawingHelper creates a dedicated HandlerThread when creating a new Surface . HandlerThread + Looper + Handler provide a useful infrastructure for efficiently (effectively) starting an endless loop in a separate thread that waits for incoming messages and processes them one by one. Thus, its efficient open API, besides SurfaceHolder.Callback2 consists of one postRedraw method, which can be used to ask the drawing stream to perform another redraw, and this is exactly what the custom PitchDetectionHandler . A “request” is made by placing a message in a queue that will be processed by the drawing stream (more specifically, our custom Handler in this stream). I did not bother to reduce the real public API to "efficient" because it makes the code a little more complicated and I'm too lazy. But, of course, both "tools" can be transferred to the inner classes.

You have to make one important decision: whether the thread thread should be each incoming message (all cent values) so that it arrives or only the last one at the time of drawing creation. This can be especially important if the PitchDetectionHandler events much faster than the drawing stream can update the Surface . I believe that in most cases it’s normal to process only the last value from the PitchDetectionHandler , but I left both versions in the code for illustration. This difference is currently implemented in code using the _processOnlyLast field. Most likely, you should make this decision and just get rid of this almost constant field and code in irrelevant branches.

And of course, don't forget to put your actual drawing logic inside doRedraw

---

Refresh (why the back button does not work)

TL version; DR

Insult line

  d.run();//starts the dispatching process 

Just comment on this!

Longer version

In our example, we can see that d is an AudioDispatcher that implements Runnable and, therefore, the run method is a method that will be called in a new thread. You may notice that this is important because several I / O operations are performed inside this method and the thread it is running on is blocked. Therefore, in your case, it blocked the main thread of the user interface. Few lines down you

 new Thread(d, "Audio Dispatcher").start(); 

and this, apparently, is the right way to use AudioDispatcher

This is easy to see from the stack traces I asked in the comments for.