Programmatically determine the number of cores and the amount of memory available Spark

Question

The Spark Web user interface shows some interesting information about the resources available to the cluster as a whole.

I am interested in the values ​​for:

• Workers
• Cores
• Memory

How can I request these pieces of information about a common cluster programmatically?

Answer 1

Spark really does not disclose this information, it is hidden in the Wizard and transmitted to the WebUI.

However, you can use a little hack, WebUI supports JSON by adding / json / to the page.

So, going to http://<master-host>:<master-port>/json/ will only return the information you are looking for:

 { url: "spark://<host>:<port>", workers: [ ], cores: 0, coresused: 0, memory: 0, memoryused: 0, activeapps: [ ], completedapps: [ ], activedrivers: [ ], status: "ALIVE" }