All geek questions in one place

Custom Metrics in Apache Spark Ui

I use Apache Spark and the metrics user interface (found on 4040) is very useful.

I wonder if it is possible to add user metrics in this user interface, custom task parameters, but possibly RDD user metrics. (e.g. runtime for RDD conversion only)

It would be nice to have custom metrics grouped by batch jobs and thread tasks.

I saw the TaskMetrics object , but it was marked as dev api and it looks just useful for input or output sources and does not support user-defined values.

Is there a way to spark? Or an alternative?

+4
source share
1 answer

You can use the generic variable support [1] built into Spark. I often used them to implement something similar.

[1] http://spark.apache.org/docs/latest/programming-guide.html#shared-variables

0
source

Source: https://habr.com/ru/post/1607482/

More articles:


All Articles