{"comment":{"created_at":"2011-10-13T17:36:48Z","body":"Nate -- I like fully independent components too. That said, I imagine NADS could consult CHRS if available and, despite the undesirable coupling in the code they could remain independently controllable features as far as the user is concerned. That might be a worthwhile trade-off to prevent Hector giving an impression of right-hand-doesn't-know-what-the-left-is-doing.\r\n\r\nPatricio -- agreed not to consult chc. I was proposing CassandraHostRetryService -- confused on the connection between the two.\r\n\r\nThe up-on-the-ring / down-in-thrift edge case aspect is interesting. I wonder if there's a deeper bug? What I did was simulate a node outage with:\r\n\r\non each good node:\r\n   ip route add blackhole <\"bad\" nodet>\r\n\r\non \"bad\" node, for each good node:\r\n   ip route add blackhole <good node>\r\n\r\nThe nodes involved are three Linux boxes in a single data center. I had thought that would yield to a fairly plain-vanilla simulation of a severed network. What do you think?","updated_at":"2011-10-13T17:36:48Z","url":"https://api.github.com/repos/rantav/hector/issues/comments/2397583","id":2397583,"user":{"gravatar_id":"9c2ccd0119fd43ec68aaf4fdd0a5d4d8","avatar_url":"https://secure.gravatar.com/avatar/9c2ccd0119fd43ec68aaf4fdd0a5d4d8?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png","url":"https://api.github.com/users/lorrin","id":351893,"login":"lorrin"}},"action":"created","issue":{"number":301,"created_at":"2011-10-12T23:31:24Z","pull_request":{"diff_url":null,"patch_url":null,"html_url":null},"body":"See the following in the log when a node is down. Aside from being verbose, it seems like pointless work is being done.\r\n```\r\nCassandraHostRetryService 2011-10-12 23:31:57,747 -- WARN -- Downed 10.183.1.254(10.183.1.254):9160 host still appears to be down: Unable to open transport to 10.183.1.254(10.183.1.254):9160 , java.net.NoRouteToHostException: Network is unreachable\r\nCassandraHostRetryService 2011-10-12 23:31:57,747 -- INFO -- Downed Host retry status false with host: 10.183.1.254(10.183.1.254):9160\r\nNodeAutoDiscoverService 2011-10-12 23:31:57,906 -- INFO -- using existing hosts [10.183.1.253(10.183.1.253):9160, 10.183.2.0(10.183.2.0):9160]\r\nNodeAutoDiscoverService 2011-10-12 23:31:57,908 -- INFO -- Found a node we don't know about 10.183.1.254(10.183.1.254):9160 for TokenRange TokenRange(start_token:119155528216427281472993688364183399179, end_token:34084936486192665607150036506241346315, endpoints:[10.183.2.0, 10.183.1.254], rpc_endpoints:[10.183.2.0, 10.183.1.254])\r\nNodeAutoDiscoverService 2011-10-12 23:31:57,908 -- INFO -- Found a node we don't know about 10.183.1.254(10.183.1.254):9160 for TokenRange TokenRange(start_token:34084936486192665607150036506241346315, end_token:76620232351309973540071862435212372747, endpoints:[10.183.1.254, 10.183.1.253], rpc_endpoints:[10.183.1.254, 10.183.1.253])\r\nNodeAutoDiscoverService 2011-10-12 23:31:57,908 -- INFO -- Found 1 new host(s) in Ring\r\nNodeAutoDiscoverService 2011-10-12 23:31:57,908 -- INFO -- Addding found host 10.183.1.254(10.183.1.254):9160 to pool\r\nHConnectionManager 2011-10-12 23:31:57,909 -- ERROR -- Transport exception host to HConnectionManager: 10.183.1.254(10.183.1.254):9160\r\nme.prettyprint.hector.api.exceptions.HectorTransportException: Unable to open transport to 10.183.1.254(10.183.1.254):9160 , java.net.NoRouteToHostException: Network is unreachable\r\n        at me.prettyprint.cassandra.connection.HThriftClient.open(HThriftClient.java:129)\r\n        at me.prettyprint.cassandra.connection.ConcurrentHClientPool.<init>(ConcurrentHClientPool.java:43)\r\n        at me.prettyprint.cassandra.connection.RoundRobinBalancingPolicy.createConnection(RoundRobinBalancingPolicy.java:68)\r\n        at me.prettyprint.cassandra.connection.HConnectionManager.addCassandraHost(HConnectionManager.java:103)\r\n        at me.prettyprint.cassandra.connection.NodeAutoDiscoverService.doAddNodes(NodeAutoDiscoverService.java:68)\r\n        at me.prettyprint.cassandra.connection.NodeAutoDiscoverService$QueryRing.run(NodeAutoDiscoverService.java:53)\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\r\n        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)\r\n        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)\r\n        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)\r\n        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\r\n        at java.lang.Thread.run(Thread.java:679)\r\nCaused by: org.apache.thrift.transport.TTransportException: java.net.NoRouteToHostException: Network is unreachable\r\n        at org.apache.thrift.transport.TSocket.open(TSocket.java:183)\r\n        at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)\r\n        at me.prettyprint.cassandra.connection.HThriftClient.open(HThriftClient.java:123)\r\n        ... 13 more\r\nCaused by: java.net.NoRouteToHostException: Network is unreachable\r\n        at java.net.PlainSocketImpl.socketConnect(Native Method)\r\n        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)\r\n        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)\r\n        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)\r\n        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)\r\n        at java.net.Socket.connect(Socket.java:546)\r\n        at org.apache.thrift.transport.TSocket.open(TSocket.java:178)\r\n        ... 15 more\r\n```","comments":5,"title":"NodeAutoDiscoverService should not discover nodes that CassandraHostRetryService already knows about","updated_at":"2011-10-13T17:36:49Z","url":"https://api.github.com/repos/rantav/hector/issues/301","id":1895868,"assignee":null,"milestone":null,"closed_at":null,"user":{"gravatar_id":"9c2ccd0119fd43ec68aaf4fdd0a5d4d8","avatar_url":"https://secure.gravatar.com/avatar/9c2ccd0119fd43ec68aaf4fdd0a5d4d8?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png","url":"https://api.github.com/users/lorrin","id":351893,"login":"lorrin"},"html_url":"https://github.com/rantav/hector/issues/301","labels":[],"state":"open"},"legacy":{"issue_id":1895868,"comment_id":2397583}}