{"issue":{"pull_request":{"diff_url":null,"html_url":null,"patch_url":null},"state":"open","user":{"avatar_url":"https://secure.gravatar.com/avatar/f1942cbab8fdffe14ecd11f6a76b4f5e?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png","gravatar_id":"f1942cbab8fdffe14ecd11f6a76b4f5e","url":"https://api.github.com/users/ekulakow","login":"ekulakow","id":464385},"milestone":null,"labels":[{"name":"Bug","color":"d95757","url":"https://api.github.com/repos/Yelp/mrjob/labels/Bug"}],"updated_at":"2011-11-08T18:38:29Z","assignee":null,"closed_at":null,"comments":1,"url":"https://api.github.com/repos/Yelp/mrjob/issues/257","body":"Hi guys,\r\n\r\nI've been running a decently large job that I'm having trouble completing because one of the reducers times out on the file upload part after it completes.  I tried adding --jobconf mapred.task.timeout=1800000 to the command line, but it appears that the job terminated after 10 mins of uploading time anyways.  I'm not sure why the upload should be having problems as it doesn't seem that the file is outrageously large, but it is significantly larger than the outputs of most of the other reducers (the key space is not well balanced - I'm going to try fixing this but it shouldn't matter for this bug).  Anyways, it seems like mrjob is timing out stuff in the upload state, and that setting mapred.task.timeout to a larger number isn't listened to.\r\n\r\nI'm on development-v0.2.x.  Am using \r\n\r\nbootstrap_actions: &boots\r\n- s3://elasticmapreduce/bootstrap-actions/configurations/latest/memory-intensive\r\n\r\nand \r\n\r\nbase_tmp_dir: /mnt/tmp/\r\n\r\nAs far as other flags go.\r\n\r\nThe bug was still happening without the memory-intensive bootstrap, though I didn't try changing mapred.task.timeout without it.\r\n\r\nSome log file stuff:\r\n\r\n2011-11-08 02:03:40,421 INFO org.apache.hadoop.mapred.Merger (main): Merging 9 sorted segments\r\n2011-11-08 02:03:41,284 INFO org.apache.hadoop.mapred.Merger (main): Down to the last merge-pass, with 9 segments left of total size: 1648153512 bytes\r\n2011-11-08 02:03:42,535 INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem (main): Creating new file 's3://path/to/output/part-00050' in S3\r\n2011-11-08 02:03:42,537 INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem (main): Outputstream for key 'path/to/output/part-00050' writing to tempfile '/mnt/var/lib/hadoop/s3/output-5430952274196003800.tmp'\r\n2011-11-08 02:03:42,578 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=1/0/0 in:0=1/92 [rec/s] out:0=0/92 [rec/s]\r\n2011-11-08 02:03:42,579 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=10/0/0 in:0=10/92 [rec/s] out:0=0/92 [rec/s]\r\n2011-11-08 02:03:42,579 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=100/0/0 in:1=100/92 [rec/s] out:0=0/92 [rec/s]\r\n\r\n...\r\n\r\n2011-11-08 02:11:14,944 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=9700000/210394/0 in:17798=9700000/545 [rec/s] out:386=210394/545 [rec/s]\r\n2011-11-08 02:11:19,745 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=9800000/210394/0 in:17850=9800000/549 [rec/s] out:383=210394/549 [rec/s]\r\n2011-11-08 02:11:24,922 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=9900000/210394/0 in:17870=9900000/554 [rec/s] out:379=210394/554 [rec/s]\r\n2011-11-08 02:11:32,005 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=10000000/210394/0 in:17793=10000000/562 [rec/s] out:374=210394/562 [rec/s]\r\n\r\nEnd Of File\r\n\r\nAnything else I can look at?","html_url":"https://github.com/Yelp/mrjob/issues/257","title":"--jobconf mapred.task.timeout does not appear to change timeout","number":257,"id":2177622,"created_at":"2011-11-08T18:34:17Z"},"action":"created","comment":{"user":{"avatar_url":"https://secure.gravatar.com/avatar/2b5b729537aaef280639ce21fc2194c1?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png","gravatar_id":"2b5b729537aaef280639ce21fc2194c1","url":"https://api.github.com/users/davidmarin","login":"davidmarin","id":50076},"updated_at":"2011-11-08T18:38:29Z","url":"https://api.github.com/repos/Yelp/mrjob/issues/comments/2671518","body":"I'm not sure what's wrong, but I think we've seen a similar issue at Yelp. I'll post here if/when we find out anything more.","id":2671518,"created_at":"2011-11-08T18:38:29Z"}}