{"issue":{"milestone":null,"title":"add mechanism for “advisory” tests","created_at":"2011-11-17T01:30:17Z","user":{"login":"drmuey","id":1200890,"url":"https://api.github.com/users/drmuey","avatar_url":"https://secure.gravatar.com/avatar/bca92f0444b2a1edc12229e7257204ee?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png","gravatar_id":"bca92f0444b2a1edc12229e7257204ee"},"labels":[{"name":"Discuss","url":"https://api.github.com/repos/schwern/test-more/labels/Discuss","color":"02c909"}],"updated_at":"2011-12-30T20:25:03Z","state":"closed","id":2264362,"pull_request":{"patch_url":null,"html_url":null,"diff_url":null},"closed_at":"2011-11-22T18:01:30Z","html_url":"https://github.com/schwern/test-more/issues/216","body":"Currently tests either pass() or fail(), it'd be nice to be able to have certain tests' failure be more of an FYI.\r\n\r\n    ok(...); # if this fails then the test file fails\r\n    \r\n    FYI: {\r\n        ok(...); # if this one fails (and is the only failed test in the file) then the test file passes\r\n    }\r\n\r\nnot sure if a block is the right thing or not, just illustrating the idea w/ a pattern already in use in Test::More\r\n\r\nI'm also not sure what a fail()ed FYI test's TAP would look like (and am not a TAP guru in general) or I'd offer a patch along w/ the idea.\r\n\r\nthanks!","number":216,"url":"https://api.github.com/repos/schwern/test-more/issues/216","comments":4,"assignee":null},"comment":{"created_at":"2011-12-30T20:25:03Z","user":{"login":"schwern","id":25888,"url":"https://api.github.com/users/schwern","avatar_url":"https://secure.gravatar.com/avatar/e55e0ec3c00f834d78fbdddeaa36e308?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png","gravatar_id":"e55e0ec3c00f834d78fbdddeaa36e308"},"updated_at":"2011-12-30T20:25:03Z","id":3316791,"body":"Thanks for the feedback.  To go through them...\r\n\r\n* The test can fail, we still want to know about it, it just isn't important enough to completely kill an install and require a new dist to be created to either address it or make it a todo.\r\n\r\nFirst observation: todo and advisory tests both have the critical attribute of not being able to fail.  They would both presumably hide their diagnostics from the user.\r\n\r\nSecond observation: for todo tests the normal condition is failure, only passing should be reported; for advisory tests there is no normal condition, they should always be reported.\r\n\r\nMy first response would be: mark it todo in the first place.  The traditional reply is: you'll get lots of tests reporting \"UNEXPECTEDLY SUCCEEDED\", because sometimes it succeeds and sometimes it fails, and users will freak out.  This isn't as bad as it used to be now that the test harness doesn't shout about passing todo tests any more.  It uses a much calmer \"TODO passed\".\r\n\r\nThat's the first implied problem with todo tests: they were loud when they passed.  This scared and confused users.  That's been fixed, so you can be more liberal with marking things todo.\r\n\r\nThe second implied problem with todo tests is getting the information back.  You're using \"advisory\" tests as probes to see where things pass and where they fail.  todo tests could be used in the same capacity.  However, CPAN testers doesn't report todo tests back to the author.  Instead of adding a new class of test result, which would require patching everything to understand and report, todo tests only require patching CPAN testers to report.\r\n\r\nThe third implied problem is that rolling a new release is expensive for the author.  Related, it's expensive for the user to deal with frequent, small releases.  This is outside the scope of Test::More, but there's plenty of tools on the author side to make releasing one-click.  Dist::Zilla being the most popular.  I just have my own automated release script.  On the user side, we're working on adding meta information which can identify the type of release which allows a user to know if the release just fixes tests, or if it contains bug fixes, or new features, or incompatibilities or all of them.\r\n\r\nIn conclusion: todo tests work for the \"information gathering\" task which advisory tests would fill; improving them to work better is less work than implementing a new class of test result; they don't provide any additional information.\r\n\r\nA way to let CPAN Testers report todo tests would be to improve TAP::Harness' ability to produce a machine readable summary of the tests on request (via an environment variable) to a file.  They may already have this and it may be a matter of somebody doing the finishing work to make it usable.\r\n\r\n* Lint checks, POD checks, etc...\r\n\r\nAs you noted, these are covered by real tests wrapped with conventions like `RELEASE_TESTING`.  They never need be run on a target machine.  Simply not running them after release is more advantageous as it removes a bunch of unnecessary test dependencies and extra CPU burnt running redundant tests.","url":"https://api.github.com/repos/schwern/test-more/issues/comments/3316791"},"action":"created"}