newrelic - New Relic for Amazon Kinesis worker monitoring -

We are using Amazon Keynes (a queue service) and are a queue reader written in Java. They basically read from the queue and put the data into our datastore. I was thinking that someone has had success in using New Rick to monitor the background queue staff?

Some analyzes are interested in me:

1. How many queue workers are running now? (They raise the top and bottom on the basis of weight)
2. How many messages / seconds is each line worker handling? How does it look over time?
3. How many messages / second full employee fleet handling?
4. Workers request both MySQL and Cassandra which part of their time is spent?
5. We are logging with log4j if workers have created errors / scripts, then what are they? What is the error rate over time?

   Thanks,

   Consulting

   There is no problem with the job of batch in opposing web transactions in the new residue, so there will be no issue.

   Assume that you are starting with Java for which the source code is available, the best way is to use the forwarded agent API: It leaves you in a good place to report any metrics that you like, even if we do not record them automatically. I will answer your questions 1 to 1:

   1) We have some ways to slice this pie, but the easiest one I can imagine is that a NewRelic.recordMetric ("Custom / Queue_worker / Live ", 1) Call me every worker on one minute (since our metric is in the crop circle) to run that call to create a timer, then you can ignore metric values ​​in the custom dashboard () Which will be average - so long as you Knows the area do not have a master on "can" and report it as often as you want, you will not get the desired effect, 1 + 1 + by reporting ... 1 = 1). You will underline the callcount field to see how many workers visited that minute.

   2) In this case, you want to use the same pattern as above. Fortunately, custom dashboards help to carry heavy loads here - NewRelic.recordMetric ("custom / qi_worker / y / number_off_sessions", x) for the number of expressions, y = some unique identifier (GUID? Random value?) ) Per worker ... and more than a minute - and then you can graph custom / QAVI_worker / * / number_off_sources to be placed on just one graph.

   3) Calculate the same custom metric, custom / queue / work / message_centric and graph call on each metric, once again you only value for each worker Can not report because the subsequent metric data will be average on average, but we will keep a good call count for you.

   4) You will get MySQL time free (as long as you are using the mysql or JDBC connector listed here) - it will appear as 'databases' in your graphs and transaction marks. For Cassandra, we do not have any specific instrumentation, but you can use the agent API once again (NewRailic Recercus Timetric ()) to at least record it and graph it separately.

   5) You get an error rate for free, as long as your errors are without exceptions - or you can at any time (or the status of an error that you want to tag) In Handling, NewRelic.noticeError () In addition, if errors are not available as exceptions (clean moves: handle the exception in your code, then restore it so that our agent can see with reference), you have a stack trace and transaction We will get a metadata in NewRelic.addCustomParameter (), we do not process the logfile, although you can write a small program to do that process Import metrics using the same methods, and since we are not licensed for each running host, not agent copy, you can get the license already for the worker for any additional cost.

   There are very easy ways to do this by using Insights - for example, you can access the list of agents running on time without any extra work, and you can get the number of winning numbers The report can not be average, on which you can math and graph them. But this is a different product and I am not trying to upload you :)

   Note: I work for new residue.