{"issue":{"created_at":"2011-12-15T19:00:27Z","state":"open","updated_at":"2011-12-15T19:00:27Z","user":{"gravatar_id":"e1a8e768c3a43b0c0165c4e112be609f","id":662582,"url":"https://api.github.com/users/psutter","login":"psutter","avatar_url":"https://secure.gravatar.com/avatar/e1a8e768c3a43b0c0165c4e112be609f?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"comments":0,"body":"When running a job with 60,000 input files stored in S3, it can take two hours to check for file existence in generatePlan before the job even beings. \r\n\r\nSolutions:\r\n1. Implement this as many threads, perhaps max(50, number of files / 30)\r\n2. Have a flag to bypass this check \r\n3. Automatically skip this check if there are >30 files, since this is almost certainly a list of files generated elsewhere.\r\n\r\n    private PipePlan generatePlan(List<PhaseError> errors) {\r\n        ...\r\n        while (!toGenerate.isEmpty()) {\r\n            Pipe file = toGenerate.iterator().next();\r\n            toGenerate.remove(file);\r\n            boolean exists = file.exists(baseConf);\r\n            if (exists && !file.isObsolete(baseConf)\r\n                    && (!forceRebuild || file.getProducer() == null)) {\r\n                if (!generated.contains(file)) {\r\n                    System.out.println(\"File: \" + file.getPath()\r\n                            + \" exists and is up to date.\");\r\n                    // ok already\r\n                    generated.add(file);\r\n                    plan.fileCreateWith(file, null);\r\n                }\r\n","assignee":{"gravatar_id":"a1aeff4eaef40532d4ff50690b8d4ec8","id":328462,"url":"https://api.github.com/users/rbodkin","login":"rbodkin","avatar_url":"https://secure.gravatar.com/avatar/a1aeff4eaef40532d4ff50690b8d4ec8?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"id":2570881,"number":2,"pull_request":{"patch_url":null,"html_url":null,"diff_url":null},"labels":[],"title":"Preposterous startup time","html_url":"https://github.com/tap-git/tap/issues/2","closed_at":null,"url":"https://api.github.com/repos/tap-git/tap/issues/2","milestone":null},"action":"opened"}