Package cssutils :: Module tokenize2 :: Class Tokenizer |
[hide private]
[frames] | no frames]

# Class Tokenizer

source code

    object --+
             |
            Tokenizer

generates a list of Token tuples:
  (Tokenname, value, startline, startcolumn)

Instance Methods | [hide private]
__init__(self, macros=None, productions=None) inits tokenizer with given macros and productions which default to cssutils own macros and productions | source code
_expand_macros(self, macros, productions) returns macro expanded productions, order of productions is kept | source code
_compile_productions(self, expanded_productions) compile productions into callable match objects, order is kept | source code
tokenize(self, text, linesep=None, fullsheet=False) (tokenname, tokenvalue, line, col) | source code
Inherited from `object`: `__delattr__`, `__getattribute__`, `__hash__`, `__new__`, `__reduce__`, `__reduce_ex__`, `__repr__`, `__setattr__`, `__str__`

Class Variables | [hide private]
| nl = ``'``\r\n``'``

Properties | [hide private]
Inherited from `object`: `__class__`

Method Details | [hide private]

### __init__(self, macros=None, productions=None) (Constructor) | source code
inits tokenizer with given macros and productions which default to cssutils own macros and productions Overrides: object.__init__

### tokenize(self, text, linesep=None, fullsheet=False) | source code
generator: tokenizes text and yiels tokens, each token is a tuple of (tokenname, tokenvalue, line, col) text to be tokenized linesep used to detect the linenumber, defaults to os.linesep fullsheet if `True` appends EOF token as last one and completes incomplete COMMENT tokens