Get Storm Topology Software Statistics

Question

I am creating a monitoring service around my Storm topology and want to be able to get the number of failed tuples in different time windows, just as the Storm interface shows the number of failed tuples in 10 m, 3 hours and 1d.

My monitoring service is currently built on python, so it would be useful if the answer included either a python library or something linguistic agnostic, like going around the CLI or hitting the REST endpoint. I took a look at the CLI Storm, as well as the docs, but still figured out empty-handed about where the Storm UI actually gets information.

EDIT: - Starting a storm version 0.8.2 (unfortunately out of my control), so storm-ui-rest-api (released in version 0.9.2), unfortunately, is not an option until it An attempt was made to update.

Answer 1

Use storm UI API

sqlInjection@foo:~$ curl http://$STORM_UI_HOST_AND_PORT/api/v1/topology/summary

{"topologies": [{"ID": "topology-1-1436004781", "encodedId": "topology-1-1436004781", "encodedId", "name": "topology-1", "status": " ACTIVE "uptime": "40d 21h 51m 59s", "tasksTotal": 16, "WorkTotal": 1, "executorsTotal": 10}]}

sqlInjection@foo:~$ curl http://$STORM_UI_HOST_AND_PORT/api/v1/topology/topology-1-1436004781

{ "msgTimeout": 30, "": [{ "": 3, "": 22336820, "errorLapsedSecs": 755996, "completeLatency": "232,052", "": 22336820, "acked": 22340300, "errorPort": 6703, "spoutId": "KafkaSpout-" "": 3, "errorHost": "", "LastError": "java.lang.RuntimeException: java.lang.NullPointerException\\ backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)\\ backtype.storm.utils.DisruptorQueue.consumeBatch(" "errorWorkerLogLink": "HTTP://: / = -6703.log", " ?": 0, "encodedSpoutId": "KafkaSpout- " }], "executorsTotal": 8, " ": "67D 21h 15m 2" "encodedId" : "-1-1436004781" , "visualizationTable": [{ ": ": [{ ": " : " ", ": Sani-": "default1544803905", ": " : }, { ": " : "__ ack_init" ": Sani-": "s__ack_init973324006", ": " }, { ": " : "__ ack_ack", ": Sani-": "s__ack_ack1278315507", ": " }, { ": " : "__ ack_fail", ": Sani...

, , /.

Answer 2

python , ""

pid = urllib2.urlopen('http://'+host+':'+port+'/api/v1/topology/summary').read()
    data_pid = json.loads(pid)
    for data in data_pid['topologies']:
        if data['name'] == '':
            print 'no topology'
            break
        elif data['name'] == topology_name:
            url_pid = data['id'].encode("UTF-8")
            break
    content = urllib2.urlopen('http://'+host+':'+port+'/api/v1/topology/'+url_pid).read()
    data_content = json.loads(content)
    if data_content['topologyStats'][0]['failed'] == None:
            data_content['topologyStats'][0]['failed'] = 0
    if data_content['topologyStats'][0]['acked'] == None:
            data_content['topologyStats'][0]['acked'] = 0
    if data_content['topologyStats'][0]['acked'] < data_content['topologyStats'][0]['failed']*10:
            global count
            count  = count + 1
            if count == 2:
                    os.system("monit restart "+ monitor_name)
                    logger.info('restart at '+ time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())))
                    count = 0

, http://chenshuxiang.applinzi.com/index.php/2017/09/13/storm-ui/