Hide the property of the spark from being displayed in the user interface of the spark network without using a protective filter

Question

The web interface of the application at http: //: 4040 lists the Spark properties on the Environment tab. All values ​​explicitly specified using spark-defaults.conf, SparkConf, or the command line. However, for security reasons, I do not want my Cassandra password to be displayed in the web interface. Is there some kind of switch to ensure that some spark properties are not displayed?

Please note that I see some solutions offering the implementation of a security filter and using the spark.ui.filters parameter to indicate the class. I hope to avoid this difficulty.

Answer 1

I think there is no general solution to how to hide my custom property from the spark WebUI for previous releases.

I assume that you are using spark 2.0 or lower (I did not see the function described below in version 2.0) because 2.0.1 supports password preprocessing on "*****".

Check SPARK-16796 Release Visible Passwords on Spark Environment Page

When we look at the source code of the Apache spark and conduct some investigation, we can see some treatments on how to “hide” a property in the ui spark network.

SparkUI defaults to the Environment page attached to initialization attachTab(new EnvironmentTab(this))[line 71]

EnvironmentPage EnvironmentPage -gui :

def render(request: HttpServletRequest): Seq[Node] = {
    val runtimeInformationTable = UIUtils.listingTable(
      propertyHeader, jvmRow, listener.jvmInformation, fixedWidth = true)
    val sparkPropertiesTable = UIUtils.listingTable(
      propertyHeader, propertyRow, listener.sparkProperties.map(removePass), fixedWidth = true)
    val systemPropertiesTable = UIUtils.listingTable(
      propertyHeader, propertyRow, listener.systemProperties, fixedWidth = true)
    val classpathEntriesTable = UIUtils.listingTable(
      classPathHeaders, classPathRow, listener.classpathEntries, fixedWidth = true)
    val content =
      <span>
        <h4>Runtime Information</h4> {runtimeInformationTable}
        <h4>Spark Properties</h4> {sparkPropertiesTable}
        <h4>System Properties</h4> {systemPropertiesTable}
        <h4>Classpath Entries</h4> {classpathEntriesTable}
      </span>

    UIUtils.headerSparkPage("Environment", content, parent)
  }

- , sparkProperties - , removePass.

private def removePass(kv: (String, String)): (String, String) = {
    if (kv._1.toLowerCase.contains("password")) (kv._1, "******") else kv
}

, "" (BTW: "" , u )

, . , . SparkSubmitArguments.scala mergeDefaultSparkProperties() spark.cassandra.auth.password sparkProperties ( removePass).

EnvironmentTab web gui ****.