{"ref":"refs/heads/master","push_id":55162547,"commits":[{"sha":"19d0c3617e17c02867792576307e4ba6fb7e619a","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/19d0c3617e17c02867792576307e4ba6fb7e619a","message":"Update asynchbase from 1.0 to 1.1.0.\n\nChange-Id: I17f4d7b6f2df884970f7d07af8a1a36b29073ac5"},{"sha":"a4ce385a60be2cfee650d27542b8dd192d7b03d3","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/a4ce385a60be2cfee650d27542b8dd192d7b03d3","message":"Add some simple internal helpers to send HBase RPCs.\n\nChange-Id: I21b586f1e6400912554adbfd297ab96028435e00"},{"sha":"5907f9afa34a3ca9868d43998c922c7bb3046a69","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/5907f9afa34a3ca9868d43998c922c7bb3046a69","message":"Don't log \"Connection reset by peer\" errors.\n\nThey just spam the logs unnecessarily.\n\nChange-Id: I6fb9081474c97142888e85b89e8fe130e0aebcac"},{"sha":"f4f405c9295644df5524e71596591f54717b5e13","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/f4f405c9295644df5524e71596591f54717b5e13","message":"Improved schema: compact the data in HBase.\n\nThis is a major change.  New data points written by this code aren't\nreadable by older versions of the code.  The new behavior is turned\noff by default, however.  After turning on the new behavior, instead\nof rolling back, you can turn it off again if you observe problems.\nIn order to turn on the new compaction code, the JVM system property\n\"tsd.feature.compactions\" needs to be set.  You can set this by in\nyour `tsdb.local' file like so:\n  JVMARGS='-Dtsd.feature.compactions'\n\nWith this change the TSD remembers every row it's writing to in HBase.\nPeriodically, it will read back old rows from HBase, and rewrite them\nin a more efficient format.  The compacted form consists mostly in\nconcatenating together all the qualifiers and values inside that row.\n\nWhen the TSD reads a row that isn't compacted properly (or at all)\nit will fix it by overwriting the row again.  So the code is handling\nproperly weird cases such as data points coming in with very old\ntimestamps or out of order data etc.  It also handles properly races\nin case multiple concurrent TSDs attempt to compact the same row at\nthe same time.\n\nIn order for this change to be truly effective, it's important to have\nHBase compactions turned on, this way HBase can purge the individual\ndata points and keep the compacted form only.  If you wish to leave\nHBase compactions turned off, consider triggering them periodically\n(either via a cron job, or manually).\n\nNote: `fsck' currently doesn't support the compacted rows, and will\nemit a warning when it sees one.  Another commit will add support for\ncompacted rows in `fsck' in the future.\n\nChange-Id: Ibc8f0154f7dd79d5337fa7effb645a297f9f94b2"},{"sha":"93905efa343bb0d22baf31616246868697db0ef0","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/93905efa343bb0d22baf31616246868697db0ef0","message":"Make the read path work in O(N) again.\n\nAfter the changes to `RowSeq' to support reading compacted rows, the\ncode kept using naive `DataPointsIterator', which does random access\ninto the data structure.  With variable length encoding, we can no\nlonger locate the n-th value in O(1), so the code needed a O(N) walk\nto find each data point, leading to an quadratic loop when iterating\non all the data points in a `RowSeq'.\n\nWith this change, `RowSeq' uses its own custom iterator implementation\ninstead of `DataPointsIterator' in order to work in O(N) [single pass].\n\nChange-Id: I98d00e1b30463542134e087243c6243e4c03ac3a"},{"sha":"9b783d3d7ec17b744a1c9afe32081fbc2a0c3206","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/9b783d3d7ec17b744a1c9afe32081fbc2a0c3206","message":"Add 2 more stats on compaction activities.\n\nChange-Id: Ieb1961516fb96f11257f99d3c1d07efbd325a660"},{"sha":"d60f118932f79603df2f4c6bf10dbfee159c869c","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/d60f118932f79603df2f4c6bf10dbfee159c869c","message":"Properly handle throttling requests from asynchbase.\n\nCompactions rewrite so much data that they're very likely to cause\nregion splits in HBase.  When they do, they keep sending tons of\nrequests to the HBase client, which eventually makes them all fail\nwith a PleaseThrottleException, due to the nature of its completely\nnon-blocking asynchronous implementation.  When we get such this\nexception, we simply re-queue the row for compaction again, to try\nto compact it later.\n\nChange-Id: Ie7f5c4291c82ce9485243b7265763a399150f3d1"},{"sha":"55021e71d473825677977257d2d41d9c405b7522","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/55021e71d473825677977257d2d41d9c405b7522","message":"Dynamically compute how many rows to flush per iteration.\n\nThis ensures that we flush more aggressively when under pressure.\n\nChange-Id: Ie82f93b080e1841b560e6a34a288cda9e755f65e"},{"sha":"ea61a98786fa77bae892877d2598bbc898caac42","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/ea61a98786fa77bae892877d2598bbc898caac42","message":"Make the compaction thread more robust.\n\nA Throwable that isn't an Exception could kill the thread.  This is bad\nas without this thread the TSD will slowly but surely run out of memory.\nWe can handle OOME by dropping the queue on the floor to free up memory.\nOther Throwables will just cause the thread to die after 1s, but a new\none will be created to replace it.\n\nChange-Id: I2bdb65338da43a0f366cdaa6b2b2e7aeabcd503e"},{"sha":"4c7ff27dafa667f532e470b632155ea2f29a2d25","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/4c7ff27dafa667f532e470b632155ea2f29a2d25","message":"Limit the number of concurrent flushes that can happen.\n\nWhen the CompactionQueue contains, say, 1M rows, and flush() is called,\nthe code was attempting to compact all these rows concurrently.  This\nis unrealistic and can either kill HBase and/or run the TSD out of RAM.\nWith this change, a single call to flush() can only initiate up to 10k\nconcurrent compactions, after which the code asynchronously waits until\nthe batch of 10k is done before flushing more.\n\nChange-Id: I05ec3839e08f13455f80b3aba31acf5f42c7ac38"},{"sha":"4db22e078074aa47c9b8e33974c585b597d45cf9","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/4db22e078074aa47c9b8e33974c585b597d45cf9","message":"Don't flush the entire compaction queue on shutdown.\n\nIt's just too much work and most of it is unnecessary.  Here's why.\nIn a typical setup, you'd have at least 2 TSD to be N + 1.  If you\nshutdown the first one, all the traffic will shift to the other one\nby virtue of tcollectors reconnecting elsewhere.  This is going to\n\"fault in\" almost all the rows that were in the compaction queue of\nthe TSD that got shutdown, so the other TSD will eventually compact\nthem.  It's possible that few rows will be left behind, and it's OK\nsince they can always be compacted later on subsequent reads.\n\nFlushing the entire queue simply takes too long.  If there are, say,\n600k rows sitting in the queue, and the TSD is able to flush 2500/s,\nit would take 4 minutes to flush them all.  So instead we only flush\nthe ones we were really supposed to flush anyway, that is the rows\nthat are over 1h old.\n\nChange-Id: I4594b714ebf0d831c9fb8d314a38154439166ec0"},{"sha":"f60a0d119557fb43a302dd930a2564b734eaf2c5","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/f60a0d119557fb43a302dd930a2564b734eaf2c5","message":"Stop (ab)using reflection in tools.\n\nInstead create an `Internal' module that exposes some internal API.\nAlthough this breaks encapsulation, both the name of the module and\nits Javadoc comment make it clear that this module isn't part of the\npublic API.  This makes the code easier to program / maintain than\nwith reflection, which is a PITA in Java (and that's not necessarily\na bad thing).\n\nChange-Id: I6bf27c0b45524f605838db8b4c7b9d721f435a68"},{"sha":"55e55ffa3abb276d4e21587c159b0f6058bca04a","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/55e55ffa3abb276d4e21587c159b0f6058bca04a","message":"Add support for compacted cells in fsck.\n\nChange-Id: Iceeb1ab40026087a3307f24dd38bebf9e21a5198"},{"sha":"724506066877245d2fa20dff7678942916fcaf9b","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/724506066877245d2fa20dff7678942916fcaf9b","message":"Properly handle malformed query strings.\n\nChange-Id: Iab42fa9429fb32156a22fcf7cdab974a5a2c14db"},{"sha":"7e827e27260625fd7676aa1177c12b91a80a2773","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/7e827e27260625fd7676aa1177c12b91a80a2773","message":"Update to JUnit 4.10.\n\nChange-Id: I46502c17b05a6581f32da9ab8a8b256bc522aa50"},{"sha":"aeaa800d36eeefe97d825c5f0e641180dcb6bb29","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/aeaa800d36eeefe97d825c5f0e641180dcb6bb29","message":"Upgrade to Mockito 1.9.0.\n\nChange-Id: I422549714e3b464e0dc70ff559db84811c84b92d"},{"sha":"3d7958462bafa28b44cd0f58377848033495400b","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/3d7958462bafa28b44cd0f58377848033495400b","message":"Upgrade to PowerMock 1.4.10.\n\nChange-Id: Ic49c7e5af251e9a68e4b922b49fed02ffe2198c6"},{"sha":"4be0e02bb2b501f3c68960b871f8923d645db1b8","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/4be0e02bb2b501f3c68960b871f8923d645db1b8","message":"Upgrade to logback 1.0.0.\n\nChange-Id: Ifd3d2e81646039e435531047b7a036dd8c19b816"},{"sha":"0f6662ec328b35ef996224b58507255ea11c4623","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/0f6662ec328b35ef996224b58507255ea11c4623","message":"Upgrade to SLF4J 1.6.4.\n\nChange-Id: Ie6acf8c253ab9cad26f86756f5557a7268f2fa20"},{"sha":"49a267a2a2448392960b32d496869f640bbf1826","author":{"email":"1e23a9b864bbfcd2c432e234ca9d6011db05a743@stumbleupon.com","name":"Benoit Sigoure"},"url":"https://api.github.com/repos/stumbleupon/opentsdb/commits/49a267a2a2448392960b32d496869f640bbf1826","message":"Upgrade to ZooKeeper 3.3.4.\n\nChange-Id: I87141ebbfdc143749644c9bc5d4997e46904b061"}],"head":"a9652711705c805fb9eeef37de7fc404ad19864f","size":26}