Spark-Metrics 实战

2018/3/30 posted in  Spark

http://metrics.dropwizard.io/3.1.0/getting-started/
http://metrics.dropwizard.io/3.1.0/manual/core/
http://wuchong.me/blog/2015/08/01/getting-started-with-metrics/

使用

The –files flag will cause /path/to/metrics.properties to be sent to every executor,
and spark.metrics.conf=metrics.properties will tell all executors to load that file
when initializing their respective MetricsSystems.

直接添加到命令行后

--files=/yourPath/metrics.properties 
--conf spark.metrics.conf=metrics.properties

小示例

新建metrics.properties文件,内容如下


driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource

*.sink.console.class=org.apache.spark.metrics.sink.ConsoleSink
*.sink.console.period=10
*.sink.console.unit=seconds

全是driver端的信息

jmx-sink

master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink

spark on yarn 查看

http://u007:8089/proxy/application_1509616075703_478985/metrics/json

总结

source里面定义从哪些实例来监控

sink指定你要输出到哪里

刚才这个例子里面,就是sink输出到console