At REI we like to measure everything. We record thousands of metrics across all of our services and applications for a variety of things ranging from datasource usage to request timing. For metrics measured from within services we use StatsD with our own custom library which writes our counters, timers, and gauges to it. Some metrics we want to record however cannot be recorded from inside an application or are not really associated with one. For these types of periodically recorded metrics we needed a different solution for sampling data points. Being a java shop we looked to find an existing open source metrics sampling tool but were not able to find one that really met our needs, so we created one: GSampler.

GSampler

GSampler is a Groovy based application that can periodically sample data points from a variety of sources and record them to Graphite or other places. It uses a dynamically re-loadable groovy configuration file to define the samplers and writers of metrics and can out of the box sample data from: groovy scripts, jdbc queries, exec (command line), and ElasticSearch. At REI we use it to record things like:

• Message queue depths
• Database statistics
• Ping times to certain vendors
• Specific error condition counts from logs
• Site Downtime

Because we can run arbitrary groovy scripts, that allows us to do things like make multiple http calls and aggregate results into one metric. The ElasticSearch integration allows us to keep trend data on certain types of errors a lot longer than we can keep the raw logs around in ElasticSearch.

Example Config File:

groovy {
    sampler('text', scriptText('[groovyStat: new StatValue(1, new Date())]'), 'groovy.stats', 1)
    sampler('file', scriptFile('/scripts/groovy/readSomeStats.groovy', ['arg1', 'arg2']), 'groovy.stats', 1)
}

jdbc {
    // registers a JDBC driver, will download driver jar using Aether
    driver('org.h2.Driver', 'com.h2database:h2:1.1.105')
   def cf = connectionFactory('jdbc:h2:mem:', 'sa', '')
    def queries = ["select 'stat.name', count(*) from some_table"]
    sampler('jdbc-stats', jdbcReader(cf, queries), 'sql.stats.prefix', 5, MINUTES)
}

exec {
    // use default metric parsing, <metric name>[whitespace]<value>
    sampler('default', command('default.sh'), 'default', 5, MINUTES)

    // set a custom parser, expects either regex with 2 matching groups or 1 AND metric name
    def p = parser(/Average = (\\d+)ms/, 'ping.average')
    sampler('exec', command('ping 127.0.0.1 -n 1', 1000, p), 'net', 1, MINUTES)
}

elasticsearch {
    def idx = index('http://elasticseach:9200')
    def esQueries = ['posts': 'type:apacheAccess AND verb:POST']

    //for 'term', pass null if you are not doing a terms query
    sampler('es', queries(idx, esQueries, 'term', 5, MINUTES), 'prefix', 5, MINUTES)
}

GSampler also supports direct integration with Git for the configuration file. This makes using it with Docker very easy because you can supply the git repo url as an environment variable and GSampler will clone it via JGit. If given a data directory GSampler will also store the last run of a given sampler, ensuring it doesn’t run more frequently than the specified interval through restarts.

For more information check out the README here: https://github.com/rei/gsampler To Run it or try it out check out our docker image: https://hub.docker.com/r/reicoop/gsampler/