def lex(body = self.body, token_class = Token)
      return [] if body =~ /^#.*encoding.*/
      Ripper.lex(body).map {|token| token_class.new(token) }
    end
def lex(body = self.body, token_class = Token)
      return [] if body =~ /^#.*encoding.*/
      Ripper.lex(body).map {|token| token_class.new(token) }
    end
Lexes the given text.

 @param [String] body (self.body) The text to lex
 @return [Array<Array<Integer, Integer>, Symbol, String>] A set of tokens
   in Ripper's result format. Each token is an array of the form:
   [[1, token_position], token_type, token_text]. I'm not exactly clear on
   why the 1 is always there. At any rate - the result is an array of those
   tokens.