Chapter Contents |
Previous |
Next |
SAS Macro Language: Reference |
The process that SAS uses to extract words and symbols from the input stack is called tokenization. Tokenization is performed by a component of SAS called the word scanner, as shown in The Sample Program before Tokenization. The word scanner starts at the first character in the input stack and examines each character in turn. In doing so, the word scanner assembles the characters into tokens. There are four general types of tokens:
* / + - ** ; $ ( ) . & % =
For more information on tokens, see Appendix 2, "SAS Tokens."
The Sample Program before Tokenization
The first SAS statement in the input stack (The Sample Program before Tokenization) contains eight tokens (four names and four special characters).
datasales(drop=lastyr) ;
When the word scanner finds a blank or the beginning of a new token, it removes a token from the input stack and transfers it to the bottom of the queue.
In this example, when the word scanner pulls the first token from the input stack, it recognizes the token as the beginning of a DATA step. The word scanner triggers the DATA step compiler, which begins to request more tokens. The compiler pulls tokens from the top of the queue, as shown in The Word Scanner Obtains Tokens.
The Word Scanner Obtains Tokens
The compiler continues to pull tokens until it recognizes the end of the DATA step (in this case, the RUN statement), which is called a DATA step boundary, as shown in The Word Scanner Sends Tokens to the Compiler. When the DATA step compiler recognizes the end of a step, the step is executed, and the DATA step is complete.
The Word Scanner Sends Tokens to the Compiler
In most SAS programs with no macro processor activity, all information that the compiler receives comes from the submitted program.
Chapter Contents |
Previous |
Next |
Top of Page |
Copyright 1999 by SAS Institute Inc., Cary, NC, USA. All rights reserved.