Note [exact print annotations]
Given a parse tree of a Haskell module, how can we reconstruct the original Haskell source code, retaining all whitespace and source code comments? We need to track the locations of all elements from the original source: this includes keywords such as 'let' / 'in' / 'do' etc as well as punctuation such as commas and braces, and also comments. We collectively refer to this metadata as the "exact print annotations". NON-COMMENT ELEMENTS Intuitively, every AST element directly contains a bag of keywords (keywords can show up more than once in a node: a semicolon i.e. newline can show up multiple times before the next AST element), each of which needs to be associated with its location in the original source code. These keywords are recorded directly in the AST element in which they occur, for the GhcPs phase. For any given element in the AST, there is only a set number of keywords that are applicable for it (e.g., you'll never see an 'import' keyword associated with a let-binding.) The set of allowed keywords is documented in a comment associated with the constructor of a given AST element, although the ground truth is in GHC.Parser and GHC.Parser.PostProcess (which actually add the annotations). COMMENT ELEMENTS We associate comments with the lowest (most specific) AST element enclosing them PARSER STATE There are three fields in PState (the parser state) which play a role with annotation comments. > comment_q :: [LEpaComment], > header_comments :: Maybe [LEpaComment], > eof_pos :: Maybe (RealSrcSpan, RealSrcSpan), -- pos, gap to prior token The 'comment_q' field captures comments as they are seen in the token stream, so that when they are ready to be allocated via the parser they are available. The 'header_comments' capture the comments coming at the top of the source file. They are moved there from the `comment_q` when comments are allocated for the first top-level declaration. The 'eof_pos' captures the final location in the file, and the location of the immediately preceding token to the last location, so that the exact-printer can work out how far to advance to add the trailing whitespace. PARSER EMISSION OF ANNOTATIONS The parser interacts with the lexer using the functions > getCommentsFor :: (MonadP m) => SrcSpan -> m EpAnnComments > getPriorCommentsFor :: (MonadP m) => SrcSpan -> m EpAnnComments > getFinalCommentsFor :: (MonadP m) => SrcSpan -> m EpAnnComments The 'getCommentsFor' function is the one used most often. It takes the AST element SrcSpan and removes and returns any comments in the 'comment_q' that are inside the span. 'allocateComments' in 'Lexer' is responsible for making sure we only return comments that actually fit in the 'SrcSpan'. The 'getPriorCommentsFor' function is used for top-level declarations, and removes and returns any comments in the 'comment_q' that either precede or are included in the given SrcSpan. This is to ensure that preceding documentation comments are kept together with the declaration they belong to. The 'getFinalCommentsFor' function is called right at the end when EOF is hit. This drains the 'comment_q' completely, and returns the 'header_comments', remaining 'comment_q' entries and the 'eof_pos'. These values are inserted into the 'HsModule' AST element. The wiki page describing this feature is https://gitlab.haskell.org/ghc/ghc/wikis/api-annotations
References 0
This Note does not link to any other.
Referenced by 6
- GHC call site
- GHC.Parser call site
- GHC.Parser.Lexer call site
- Language.Haskell.Syntax.Binds call site
- Language.Haskell.Syntax.Decls call site
- Language.Haskell.Syntax.Expr call site