Package cssutils :: Module tokenize2 :: Class Tokenizer |
[hide private]
[frames] | no frames]

# Class Tokenizer

source code

    object --+
             |
            Tokenizer

generates a list of Token tuples:
  (Tokenname, value, startline, startcolumn)

Instance Methods | [hide private]
_expand_macros(self, macros, productions) returns macro expanded productions, order of productions is kept | source code
_compile_productions(self, expanded_productions) compile productions into callable match objects, order is kept | source code
__init__(self, macros=None, productions=None) inits tokenizer with given macros and productions which default to cssutils own macros and productions | source code
tokenize(self, text, fullsheet=False) generator: tokenizes text and yiels tokens, each token is a tuple of: | source code
Inherited from `object`: `__delattr__`, `__getattribute__`, `__hash__`, `__new__`, `__reduce__`, `__reduce_ex__`, `__repr__`, `__setattr__`, `__str__`

Class Variables | [hide private]
| _linesep = ``u'``\n``'``

Properties | [hide private]
Inherited from `object`: `__class__`

Method Details | [hide private]

### __init__(self, macros=None, productions=None) (Constructor) | source code
inits tokenizer with given macros and productions which default to cssutils own macros and productions Overrides: object.__init__

### tokenize(self, text, fullsheet=False) | source code
generator: tokenizes text and yiels tokens, each token is a tuple of:
    (tokenname, tokenvalue, line, col)
The tokenvalue will contain a normal string, meaning CSS unicode escapes have been resolved to normal characters. The serializer escapes needed characters back to unicode escapes depending of the stylesheet target encoding. text to be tokenized fullsheet if `True` appends EOF token as last one and completes incomplete COMMENT tokens