{"action":"closed","pull_request":{"created_at":"2011-11-24T22:04:42Z","review_comments":0,"state":"closed","patch_url":"https://github.com/cloudant/bucky/pull/2.patch","changed_files":13,"updated_at":"2011-11-27T04:23:53Z","_links":{"review_comments":{"href":"https://api.github.com/repos/cloudant/bucky/pulls/2/comments"},"comments":{"href":"https://api.github.com/repos/cloudant/bucky/issues/2/comments"},"html":{"href":"https://github.com/cloudant/bucky/pull/2"},"self":{"href":"https://api.github.com/repos/cloudant/bucky/pulls/2"}},"user":{"gravatar_id":"db0d5b15da76246bed0ba3c253b5ea82","id":227121,"url":"https://api.github.com/users/mk-fg","login":"mk-fg","avatar_url":"https://secure.gravatar.com/avatar/db0d5b15da76246bed0ba3c253b5ea82?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"issue_url":"https://github.com/cloudant/bucky/issues/2","comments":2,"deletions":63,"body":"Getting a lot of metrics into graphite through bucky is easy, but in most cases all retentions but the most granular one *will be calculated incorrectly*.\r\n\r\nProblem is that there are lot of counters in collectd, there is a counter type in statsd and metricsd and these can't be just \"averaged\": if you have \"60s:1d,5m:7d\" retentions, then counter increments like \"1,2,1,2,1\" will be averaged to \"1\" for 5m interval and this will be *totally wrong*, since 5-minute counter should show \"7\" - the sum of all counters for \"60s\" intervals, not the average.\r\n\r\nGraphite 0.9.9 allows to specify aggregation methods via storage-aggregation.conf and manually via whisper-set-aggregation-method.py script.\r\n\r\nBucky *knows* exactly which aggregation type to use for each graphite name, but it has no way to pass this data to graphite directly afaik, hence this change.\r\n--dump-aggregation-methods option make bucky dump lines like \"host.postfix.connection.in.open sum\", which can be then easily processed with stuff like awk to either run whisper-set-aggregation-method.py or build storage-aggregation.conf, or - which is probably the case - both.\r\nDoing the same thing by hand for some tens / hundreds of metrics is just insane.\r\n\r\nI considered writing this data to some dbm, so there will be strictly one record per name, but it introduces complexity which I just couldn't justify - it's much simpier to read plain file with common unix tools than write your own db-parser.\r\nAlthough now I see that I was probably wrong there - since this data is a subject to change at runtime, restarting bucky with this flag just to collect new methods every once in a while is wrong, so I'll probably change target from stdout to dbm (with \"name: first-seen-timestamp, aggregation method\" format), so changes there can be tracked and applied to whisper files and/or graphite configuration.\r\n\r\nChanges in the PR are quite intrusive and can potentially hurt performance, since checks are made on processing of every value, but I thought I'd still create this PR as a kind of RFC, feel free to just reject it if you have a better implementation in mind or consider it unnecessary or out of scope for the app, so I can either change direction here or won't bother you with this again.\r\n","merged":false,"mergeable":true,"commits":4,"id":527505,"base":{"user":{"gravatar_id":"640ceaa25aad5532eb2a9cd6183bd615","id":44642,"url":"https://api.github.com/users/cloudant","login":"cloudant","avatar_url":"https://secure.gravatar.com/avatar/640ceaa25aad5532eb2a9cd6183bd615?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-orgs.png"},"repo":{"created_at":"2011-11-09T20:08:44Z","pushed_at":"2011-11-21T18:19:11Z","private":false,"clone_url":"https://github.com/cloudant/bucky.git","updated_at":"2011-11-21T18:33:21Z","name":"bucky","watchers":3,"svn_url":"https://svn.github.com/cloudant/bucky","master_branch":null,"language":"Python","forks":4,"owner":{"gravatar_id":"640ceaa25aad5532eb2a9cd6183bd615","id":44642,"url":"https://api.github.com/users/cloudant","login":"cloudant","avatar_url":"https://secure.gravatar.com/avatar/640ceaa25aad5532eb2a9cd6183bd615?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-orgs.png"},"id":2744143,"homepage":"http://github.com/cloudant/bucky","description":"UDP frontend to Graphite","git_url":"git://github.com/cloudant/bucky.git","html_url":"https://github.com/cloudant/bucky","open_issues":1,"size":108,"ssh_url":"git@github.com:cloudant/bucky.git","fork":false,"url":"https://api.github.com/repos/cloudant/bucky"},"sha":"55bc54304a2384224bb44aa8b44deedc12f45e5d","ref":"master","label":"cloudant:master"},"number":2,"merged_by":null,"additions":159,"title":"Aggregation methods dump","head":{"user":{"gravatar_id":"db0d5b15da76246bed0ba3c253b5ea82","id":227121,"url":"https://api.github.com/users/mk-fg","login":"mk-fg","avatar_url":"https://secure.gravatar.com/avatar/db0d5b15da76246bed0ba3c253b5ea82?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"repo":{"created_at":"2011-11-17T03:25:09Z","pushed_at":"2011-11-27T03:28:05Z","private":false,"clone_url":"https://github.com/mk-fg/bucky.git","updated_at":"2011-11-27T03:28:05Z","name":"bucky","watchers":1,"svn_url":"https://svn.github.com/mk-fg/bucky","master_branch":null,"language":"Python","forks":0,"owner":{"gravatar_id":"db0d5b15da76246bed0ba3c253b5ea82","id":227121,"url":"https://api.github.com/users/mk-fg","login":"mk-fg","avatar_url":"https://secure.gravatar.com/avatar/db0d5b15da76246bed0ba3c253b5ea82?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"id":2792966,"homepage":"http://github.com/cloudant/bucky","description":"Collectd adapter to Graphite","git_url":"git://github.com/mk-fg/bucky.git","html_url":"https://github.com/mk-fg/bucky","open_issues":0,"size":152,"ssh_url":"git@github.com:mk-fg/bucky.git","fork":true,"url":"https://api.github.com/repos/mk-fg/bucky"},"sha":"664243b9e72669d046302c2c3051d7c2ef97e685","ref":"aggregation_methods_generation","label":"mk-fg:aggregation_methods_generation"},"closed_at":"2011-11-27T04:23:53Z","html_url":"https://github.com/cloudant/bucky/pull/2","diff_url":"https://github.com/cloudant/bucky/pull/2.diff","url":"https://api.github.com/repos/cloudant/bucky/pulls/2","merged_at":null},"number":2}