I run the Ammonite shell in the sbt console and try to run spark jobs, I see that the logging level is Debug, and it prints all the debugging level loggers that generate a huge amount of logs and shut down forever.
14:36:14.331 [run-main-0] DEBUG org.apache.http.wire - >> "Content-Type: application/octet-stream[\r][\n]"
14:36:14.331 [run-main-0] DEBUG org.apache.http.wire - >> "Connection: Keep-Alive[\r][\n]"
14:36:14.331 [run-main-0] DEBUG org.apache.http.wire - >> "[\r][\n]"
14:36:14.331 [run-main-0] DEBUG org.apache.http.headers - >> GET /nlp/resources/DecisionSentence/parenUnigrams.txt HTTP/1.1
14:36:14.331 [run-main-0] DEBUG org.apache.http.headers - >> Host: ravellaw.s3-us-west-2.amazonaws.com
14:36:14.331 [run-main-0] DEBUG org.apache.http.headers - >> X-Amz-Date: 20160209T223614Z
Below are the settings that I tried to turn off debug logging, but I still see debug level loggers
Added to build.sbt:
showSuccess := false
logLevel in console := Level.Warn
logLevel in run := Level.Warn
I tried calling the sbt console with a warning:
sbt --warn console
In the sbt console, make the following settings:
System.setProperty("log4j.logger.org.apache.http.wire", "WARN")
System.setProperty("log4j.logger.org.apache.http.headers", "WARN")
System.setProperty("log4j.logger.org.apache.http.content", "WARN")
edited / spark / conf / log4j.properties and changed form form
log4j.rootCategory=INFO, console
to
log4j.rootCategory=WARN, console
Unfortunately, after all these changes, I still see the debug log.