{"issue":{"title":"load_svmlight_file wastes too much memory","user":{"id":89061,"login":"ogrisel","url":"https://api.github.com/users/ogrisel","gravatar_id":"aee56554ec30edfd680e1c937ed4e54d","avatar_url":"https://secure.gravatar.com/avatar/aee56554ec30edfd680e1c937ed4e54d?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"milestone":{"title":"0.10","creator":{"id":277639,"login":"fabianp","url":"https://api.github.com/users/fabianp","gravatar_id":"eaa2cd9096f3200545585814052a475e","avatar_url":"https://secure.gravatar.com/avatar/eaa2cd9096f3200545585814052a475e?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"open_issues":14,"created_at":"2011-07-05T09:15:09Z","state":"open","due_on":"2012-01-11T08:00:00Z","closed_issues":12,"description":"0.10 release","number":4,"url":"https://api.github.com/repos/scikit-learn/scikit-learn/milestones/4"},"created_at":"2011-09-29T07:50:07Z","labels":[],"state":"open","updated_at":"2011-12-21T17:27:53Z","id":1772598,"html_url":"https://github.com/scikit-learn/scikit-learn/issues/369","closed_at":null,"pull_request":{"patch_url":null,"html_url":null,"diff_url":null},"body":"When trying to load a 1.1GB dataset in svmlight format from kaggle.com `skearn.datasets.load_svmlight_file` is trying to allocate several times this amount as intermediate python lists before being killed by the OS.\r\n\r\nIMHO: This code should be refactored as a two passes algorithm that counts the number of samples (for labels and indptr which has length n_samples + 1) and the total number of non-zero values (for data and indices) so as to pre-allocate them upfront and then to a second pass to actually fill the arrays with the values.\r\n\r\nFYI the former C++ impl that has moved to https://github.com/mblondel/svmlight-loader loads the same dataset without wasting memory.","number":369,"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/369","comments":13,"assignee":null},"comment":{"user":{"id":335383,"login":"larsmans","url":"https://api.github.com/users/larsmans","gravatar_id":"56e17cbc8c5ea5b37d67018db6d030b0","avatar_url":"https://secure.gravatar.com/avatar/56e17cbc8c5ea5b37d67018db6d030b0?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"created_at":"2011-12-21T17:27:53Z","updated_at":"2011-12-21T17:27:53Z","id":3236698,"body":"Can we close this issue?","url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/3236698"},"action":"created"}