{"issue":{"created_at":"2011-12-13T15:17:42Z","state":"open","updated_at":"2011-12-13T15:17:42Z","user":{"gravatar_id":"7bd67e58e011c25ba9c0d62757386a27","id":305837,"url":"https://api.github.com/users/lydiam","avatar_url":"https://secure.gravatar.com/avatar/7bd67e58e011c25ba9c0d62757386a27?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png","login":"lydiam"},"comments":0,"body":"I'm not sure if this is  a github issue or if this job was deliberately killed:\r\n\r\nOn tarchive, a disk-fixity.pid file exists that predates the latest run of disk-fixity and yet disk-fixity ran regardless.\r\n\r\n[daitss@fclnx31 daemons]$ cd /var/run/fixity\r\n\r\n[daitss@fclnx31 fixity]$ ls -l\r\ntotal 8\r\n-rw-r--r-- 1 silo daitss 6 Dec  5 11:00 disk-fixity.pid\r\n\r\n[daitss@fclnx31 fixity]$ cat disk-fixity.pid \r\n26696\r\n\r\nThe /var/log/daitss/daemons/fixity.log-20111211 ends with the job incomplete:\r\n\r\n[daitss@fclnx31 daemons]$ tail fixity.log-20111211\r\nDec  9 11:45:53 fclnx31 DiskFixity[26696]:  INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Beginning bulk fixity update for silo /daitssfs/038 \r\nDec  9 11:45:53 fclnx31 DiskFixity[26696]:  INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Starting silo  /daitssfs/038 on Friday, December 09, 2011 at 11:45:53 AM \r\nDec  9 17:18:16 fclnx31 DiskFixity[26696]:  INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Completed silo /daitssfs/038 on Friday, December 09, 2011 at 05:18:16 PM \r\nDec  9 17:18:16 fclnx31 DiskFixity[26696]:  INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Total of 1921 packages: 0 skipped, 1921 succeeded, 0 failed \r\nDec  9 17:18:16 fclnx31 DiskFixity[26696]:  INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Beginning bulk fixity update for silo /daitssfs/039 \r\nDec  9 17:18:16 fclnx31 DiskFixity[26696]:  INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Starting silo  /daitssfs/039 on Friday, December 09, 2011 at 05:18:16 PM \r\nDec  9 23:16:13 fclnx31 DiskFixity[26696]:  INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Completed silo /daitssfs/039 on Friday, December 09, 2011 at 11:16:13 PM \r\nDec  9 23:16:13 fclnx31 DiskFixity[26696]:  INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Total of 2511 packages: 0 skipped, 2511 succeeded, 0 failed \r\nDec  9 23:16:13 fclnx31 DiskFixity[26696]:  INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Beginning bulk fixity update for silo /daitssfs/021 \r\nDec  9 23:16:13 fclnx31 DiskFixity[26696]:  INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Starting silo  /daitssfs/021 on Friday, December 09, 2011 at 11:16:13 PM \r\n\r\nThere are no further entries for this job in any of the fixity logs.\r\n\r\nPrior failed jobs issued XYMON messages and emails, but this one did not.\r\n\r\nTue Nov 29 15:47:11 EST 2011 -- Fixity Check --\r\n\r\ngreen Collect-Fixities PID file does not exist and no running Collect-Fixities Process\r\nred Disk-Fixity PID file exists, however no running Disk-Fixity Process\r\ngreen Tape-Fixity PID file does not exist and no running Tape-Fixity Process\r\n\r\nThe message that triggered this error (email sent at 11/29/2011 3:46pm):\r\n\r\n/opt/fda/etc/silo.weekly/run-disk-fixity:\r\n\r\n/opt/web-services/sites/silos/current/lib/store/silo.rb:307:in `md5': Can't determine md5 for 'E20060816_AAACGI' (located at '/daitssfs/007/68f/ab940396a03793eb754264bc88a34'): No such file or directory - /daitssfs/007/68f/ab940396a03793eb754264bc88a34/md5 (Store::SiloError)\r\n\tfrom /opt/web-services/sites/silos/current/lib/store/silo.rb:403:in `read_lock'\r\n\tfrom /opt/web-services/sites/silos/current/lib/store/silo.rb:401:in `open'\r\n\tfrom /opt/web-services/sites/silos/current/lib/store/silo.rb:401:in `read_lock'\r\n\tfrom /opt/web-services/sites/silos/current/lib/store/silo.rb:303:in `md5'\r\n\tfrom /opt/web-services/sites/silos/current/tools/disk-fixity:195\r\n\tfrom /opt/web-services/sites/silos/current/lib/store/silodb.rb:141:in `each_package_record'\r\n\tfrom /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/dm-core-1.2.0/lib/dm-core/collection.rb:507:in `each'\r\n\tfrom /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/dm-core-1.2.0/lib/dm-core/support/lazy_array.rb:411:in `each'\r\n\tfrom /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/dm-core-1.2.0/lib/dm-core/support/lazy_array.rb:411:in `each'\r\n\tfrom /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/dm-core-1.2.0/lib/dm-core/collection.rb:504:in `each'\r\n\tfrom /opt/web-services/sites/silos/current/lib/store/silodb.rb:141:in `each_package_record'\r\n\tfrom /opt/web-services/sites/silos/current/tools/disk-fixity:158\r\n\tfrom /opt/web-services/sites/silos/current/tools/disk-fixity:152:in `each'\r\n\tfrom /opt/web-services/sites/silos/current/tools/disk-fixity:152\r\n\r\n\r\nIs it OK to delete the pid file at this point?\r\n\r\n\r\n\r\n","assignee":null,"id":2537643,"pull_request":{"patch_url":null,"html_url":null,"diff_url":null},"labels":[],"number":603,"title":"disk-fixity:  repeated crashes","closed_at":null,"html_url":"https://github.com/daitss/core/issues/603","url":"https://api.github.com/repos/daitss/core/issues/603","milestone":null},"action":"opened"}