[lyx.git] / lib / examples / listerrors.lyx

#LyX 1.2 created this file. For more info see http://www.lyx.org/
\lyxformat 220
\textclass literate-article
\begin_preamble
%
% ps2pdf stuff
%
\usepackage[ps2pdf,pdftitle={LyX listerrors re-implemented},urlcolor=blue,linktocpage,letterpaper,colorlinks=true]{hyperref}
\@savsf=1% This is to get around a hyperref+noweb interaction problem
\hyphenpenalty 10000

%
% This (from the noweb FAQ) relaxes the constraint that chunks are never broken across pages.
%
\def\nwendcode{\endtrivlist \endgroup \vfil\penalty10\vfilneg}
\let\nwdocspar=\smallbreak
\end_preamble
\language english
\inputencoding auto
\fontscheme pslatex
\graphics default
\paperfontsize default
\spacing single 
\papersize Default
\paperpackage a4
\use_geometry 0
\use_amsmath 0
\use_natbib 0
\use_numerical_citations 0
\paperorientation portrait
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\defskip medskip
\quotes_language english
\quotes_times 2
\papercolumns 1
\papersides 1
\paperpagestyle default

\layout Title

LyX listerrors:
\newline 
rewritten in Python
\layout Author

Kayvan A.
 Sylvan
\newline 

\begin_inset LatexCommand \url{mailto:kayvan@sylvan.com}

\end_inset 


\layout Date

3/15/2002
\layout Abstract

The listerrors program used to be compiled as a C program and installed
 as 
\emph on 
BINDIR/listerrors
\emph default 
 along with LyX in order to perform some simple re-formatting of noweb and
 GCC error messages.
 This document describes and implements the Python version of the same program.
\layout Standard


\begin_inset LatexCommand \tableofcontents{}

\end_inset 


\layout Section

Introduction
\layout Standard

The motivation for this program was Bugzilla bug 190
\begin_inset Foot
collapsed true

\layout Standard

Visit 
\begin_inset LatexCommand \url{http://bugzilla.lyx.org/show_bug.cgi?id=190}

\end_inset 

 for the details.
\end_inset 

 dealing with the 
\begin_inset Quotes eld
\end_inset 

listerrors
\begin_inset Quotes erd
\end_inset 

 executable.
\layout Standard

What is 
\begin_inset Quotes eld
\end_inset 

listerrors
\begin_inset Quotes erd
\end_inset 

? Usually, LyX has great support for parsing of error messages.
 For each error in the log file, LyX pops up an error box at that location
 in the LyX window.
 The error scanning routines expect these errors to be in a certain format
 (similar to LaTeX errors).
 When dealing with Literate Programs, you have 
\begin_inset Quotes eld
\end_inset 

noweb
\begin_inset Foot
collapsed true

\layout Standard

See 
\begin_inset LatexCommand \url{http://www.eecs.harvard.edu/~nr/noweb}

\end_inset 

 for more information about noweb.
\end_inset 


\begin_inset Quotes erd
\end_inset 

 as well as gcc error messages (and potentially others).
 The listerrors program attempts to standardize these error messages to
 a format that LyX can parse and react to.
\layout Standard

In a nutshell, the problems with the old implementation of listerrors that
 bug 190 refers to were::
\layout Enumerate

It was a C program and it was installed in the user path in the same directory
 as LyX.
 Having such a generically named binary in, for example, 
\emph on 
/usr/bin
\emph default 
, was potentially confusing.
\layout Enumerate

It required that noweb be installed on the compiling machine (the source
 was extracted by noweb from 
\emph on 
SRCDIR/examples/Literate.lyx
\emph default 
, compiled and installed by 
\begin_inset Quotes eld
\end_inset 

make install
\begin_inset Quotes erd
\end_inset 

).
\layout Standard

The new version deals with these problems in the following fashion:
\layout Enumerate

Both the example file (this document) and the program are to be added to
 the LyX CVS repository.
\layout Enumerate

The program itself will be installed in 
\emph on 
SHAREDIR/lyx/scripts
\emph default 
, along with other LyX-specific helper scripts.
\layout Standard

In the design and implementation of this new 
\begin_inset Quotes eld
\end_inset 

listerrors
\begin_inset Quotes erd
\end_inset 

, the Python
\begin_inset Foot
collapsed true

\layout Standard

See the Python home page (
\begin_inset LatexCommand \url{http://www.python.org}

\end_inset 

) for more information.
\end_inset 

 language was chosen since it is fully multi-platform and provides a very
 uniform and easy to read syntax.
 This re-write also simplifies the code for 
\begin_inset Quotes eld
\end_inset 

listerrors
\begin_inset Quotes erd
\end_inset 

 greatly.
 Python is installed by default on all modern Linux systems and is freely
 available for all other platforms.
\layout Scrap

<<listerrors>>=
\newline 
#!/usr/bin/python
\newline 
"""reformat noweb and compiler errors for LyX.
\newline 

\newline 
Expects to read from stdin and output to stdout.
\newline 
"""
\newline 

\newline 
__author__ = "Kayvan A.
 Sylvan <kayvan@sylvan.com>"
\newline 
__date__ = "$Date: 2002/03/19 21:42:48 $"
\newline 
__version__ = "$Revision: 1.1 $"
\newline 
__credits__ = """Edmar Wienskoski Jr.
 <edmar-w-jr@technologist.com>
\newline 
    original Literate support for LyX.
\newline 
Bernard Michael Hurley <berhardh@westherts.ac.uk>
\newline 
    modifications to original listerrors."""
\newline 
__copyright__ = "Copyright 2002 - The LyX team."
\newline 

\newline 
import sys
\newline 

\newline 
<<Function Bodies>>
\newline 

\newline 
if __name__ == "__main__":
\newline 
  main()
\newline 
@
\layout Section

LaTeX style error message
\layout Standard

The following function mimics the TeX error message format.
\layout Scrap

<<Function Bodies>>=
\newline 
def write_error(msg, tool = "noweb", line_number = 1):
\newline 
  """Write out the given message in TeX error style.
\newline 

\newline 
  called like: write_error(msg, tool, line_number)."""
\newline 
  print "! Build Error: ==> %s ==>
\backslash 
n" % (tool),
\newline 
  print " ...
\backslash 
n
\backslash 
nl.%d ...
\backslash 
n" % (line_number),
\newline 
  if type(msg) == type("str"): # simple string
\newline 
    print msg
\newline 
  else: # some kind of list (sequence or tuple)
\newline 
    for m in msg:
\newline 
        if m != "": print m,
\newline 
    print
\newline 

\newline 
@ %def write_error
\layout Section

Filtering errors
\layout Standard

The only complication in our filtering code is that some parsers might need
 to push back lines that are read in to be read again later.
 We solve this problem by implementing a 
\begin_inset Quotes eld
\end_inset 

getline
\begin_inset Quotes erd
\end_inset 

 and 
\begin_inset Quotes eld
\end_inset 

pushline
\begin_inset Quotes erd
\end_inset 

 set of functions:
\layout Scrap

<<Function Bodies>>=
\newline 
__lines = [] # lines pushed back
\newline 

\newline 
def getline(file = sys.stdin):
\newline 
  """read a line from internal stack or from file.
\newline 

\newline 
  optional file argument defaults to sys.stdin."""
\newline 
  global __lines
\newline 
  lines = __lines
\newline 
  if lines:
\newline 
    line = lines[-1]
\newline 
    lines = lines[:-1]
\newline 
  else:
\newline 
    line = file.readline()
\newline 
  return line
\newline 

\newline 
@ %def getline
\layout Standard

And now for the corresponding pushline function:
\layout Scrap

<<Function Bodies>>=
\newline 
def pushline(line):
\newline 
  "push a line onto the pushback stack."
\newline 
  global __lines
\newline 
  lines = __lines
\newline 
  lines += (line,) # push a list onto the stack, not individual letters
\newline 

\newline 
@ %def pushline
\layout Standard

The main() entry point function is extremely simple.
 Note that this version of 
\begin_inset Quotes eld
\end_inset 

listerrors
\begin_inset Quotes erd
\end_inset 

 takes no options and simply filters, attempting simply to match against
 the known error message patterns.
 The listerrors C program handled a single-character command-line argument
 that the current code no longer needs.
 
\layout Scrap

<<Function Bodies>>=
\newline 
def main():
\newline 
  """Entry point for listerrors.
 Takes no options.
\newline 

\newline 
  Reads stdin and writes to stdout.
 Filter errors"""
\newline 

\newline 
  while 1:
\newline 
    line = getline()
\newline 
    if line == "": break
\newline 
    <<Check line against patterns and take action>>
\newline 
@ %def main
\layout Standard

For each line read in, we need to find out if it matches any of our tools
 (noweb, gcc, etc.) and act accordingly.
\layout Scrap

<<Check line against patterns and take action>>=
\newline 
try_patterns_dispatch = [ noweb_try, gcc_try, xlc_try ]
\newline 
for predicate in try_patterns_dispatch:
\newline 
  if predicate(line): break
\newline 
@
\layout Section

Different Error Formats
\layout Standard

The following sections handle the various error message formats that we
 recognize in this program.
 
\layout Subsection

noweb errors
\layout Standard

Noweb errors are output on a single line, so examining just the current
 line is enough.
\layout Scrap

<<Function Bodies>>=
\newline 
def noweb_try(line):
\newline 
  """see if line is a noweb error.
\newline 

\newline 
  Returns 1 on success, 0 otherwise.
 Outputs on stdout."""
\newline 
  retval = 0
\newline 
  <<Look for the unescaped angle-brackets in documentation>>
\newline 
  <<Look for anything with double angle brackets>>
\newline 
  <<Last ditch effort scan for specific strings>>
\newline 
  return retval
\newline 

\newline 
@ %def noweb_try
\layout Standard

First, we look for the 
\begin_inset Quotes eld
\end_inset 

unescaped < < in documentation chunk
\begin_inset Quotes erd
\end_inset 

 message.
 This is the only message with an associated line number from noweb.
\layout Scrap

<<Look for the unescaped angle-brackets in documentation>>=
\newline 
if line.find(": unescaped << in documentation chunk") != -1:
\newline 
  line_parts = line.split(':')
\newline 
  num_str = line_parts[1]
\newline 
  num_len = len(num_str)
\newline 
  i = 0
\newline 
  while i < num_len and num_str[i].isdigit(): i += 1
\newline 
  if i == num_len:
\newline 
    write_error(":" + line_parts[2], "noweb", int(num_str))
\newline 
    retval = 1
\newline 
@
\layout Standard

Some noweb messages are simply about undefined scraps.
 These can be seen by looking for matching double-angle-brackets.
\layout Scrap

<<Look for anything with double angle brackets>>=
\newline 
if (not retval):
\newline 
  left = line.find("<<")
\newline 
  if (left != -1) and ((left + 2) < len(line)) and 
\backslash 

\newline 
     (line[left+2:].find(">>") != -1):
\newline 
    write_error(line, "noweb");
\newline 
    retval = 1;
\newline 
@
\layout Standard

Finally, here is an additional list of explicit strings to check for.
\layout Scrap

<<Last ditch effort scan for specific strings>>=
\newline 
if (not retval):
\newline 
  msgs_to_try = ("couldn't open file",
\newline 
    "couldn't open temporary file",
\newline 
    "error writing temporary file",
\newline 
    "ill-formed option",
\newline 
    "unknown option",
\newline 
    "Bad format sequence",
\newline 
    "Can't open output file",
\newline 
    "Can't open temporary file",
\newline 
    "Capacity exceeded:",
\newline 
    "Ignoring unknown option -",
\newline 
    "This can't happen:",
\newline 
    "non-numeric line number in")
\newline 
  for msg in msgs_to_try:
\newline 
    if line.find(msg) != -1:
\newline 
      write_error(line, "noweb")
\newline 
      retval = 1
\newline 
      break
\newline 
@
\layout Subsection

gcc errors
\layout Standard

The gcc errors can be multi-line, with the following format:
\layout LyX-Code

foo.c: In function `main': 
\newline 
foo.c:3: `bar' undeclared (first use in this function) 
\newline 
foo.c:3: (Each undeclared identifier is reported only once 
\newline 
foo.c:3: for each function it appears in.) 
\newline 
foo.c:3: parse error before `x'
\layout Standard

In order to parse this, the gcc error handler has to look ahead and return
 any and all lines that do not match the expected pattern.
\layout Scrap

<<Function Bodies>>=
\newline 
def gcc_try(line):
\newline 
  """See if line is a gcc error.
 Read ahead to handle all the lines.
\newline 

\newline 
  Returns 1 on success, 0 otherwise.
 Outputs on stdout."""
\newline 
  retval = 0
\newline 
  <<Handle the gcc error message>>
\newline 
  return retval
\newline 

\newline 
@ %def gcc_try
\layout Standard

The error message starts with a gcc header (as above) without an associated
 line number.
\layout Scrap

<<Handle the gcc error message>>= 
\newline 
first_space = line.find(' ')
\newline 
if first_space > 1: # The smallest would be "X: "
\newline 
  if line[first_space - 1] == ':':
\newline 
    header_to_see = line[:first_space - 1]
\newline 
    next_line = getline()
\newline 
    if next_line and next_line[:first_space - 1] == header_to_see:
\newline 
      num_end = first_space
\newline 
      while next_line[num_end].isdigit(): num_end += 1
\newline 
      if num_end > first_space: # good!
\newline 
        <<Accumulate gcc error lines and print it>>
\newline 
      else: # oops! Not a gcc error.
\newline 
        pushline(next_line)
\newline 
    elif next_line:
\newline 
      pushline(next_line) # return this line to input stream
\newline 
@
\layout Standard

At the point in the code that we know that we are in the middle of an error
 message, we do the following:
\layout Scrap

<<Accumulate gcc error lines and print it>>=
\newline 
num_str = next_line[first_space:num_end]
\newline 
msgs = []
\newline 
msgs += (line[first_space:],)
\newline 
msgs += (next_line[num_end + 1:],)
\newline 
header_to_see = next_line[:num_end]
\newline 
next_line = getline()
\newline 
while next_line and next_line[:num_end] == header_to_see:
\newline 
  msgs += (next_line[num_end + 1:],)
\newline 
  next_line = getline()
\newline 
if next_line: pushline(next_line)
\newline 
write_error(msgs, "gcc", int(num_str))
\newline 
retval = 1
\newline 
@
\layout Subsection

xlc errors
\layout Standard

A xlc error message is easy to identify.
 Every error message starts with a quoted string with no spaces, a comma,
 a space, the word 
\begin_inset Quotes eld
\end_inset 

line
\begin_inset Quotes erd
\end_inset 

, a space, and some variable text.
 The following routine tests if a given buffer line matches this criteria
 (this code would probably be simpler if I used the 
\begin_inset Quotes eld
\end_inset 

re
\begin_inset Quotes erd
\end_inset 

 regexp module, but we don't really need the full regular expression engine
 here).
 
\layout Scrap

<<Function Bodies>>=
\newline 
def xlc_try(line):
\newline 
  """see if line is an xlc error.
\newline 

\newline 
  Returns 1 on success, 0 otherwise.
 Outputs on stdout."""
\newline 
  retval = 0
\newline 
  if line[0] == '"': # This is the first character of all xlc errors
\newline 
    next_quote = line.find('"', 1)
\newline 
    first_space = line.find(' ')
\newline 
    if (next_quote != -1) and (first_space > next_quote): # no space inisde
 quotes
\newline 
      if line[first_space - 1:first_space + 6] == ", line ":
\newline 
        num_start = num_end = first_space + 6
\newline 
        while line[num_end].isdigit(): num_end += 1
\newline 
        if num_end > num_start:
\newline 
          write_error(line, "xlc", int(line[num_start : num_end]))
\newline 
          retval = 1
\newline 
  return retval
\newline 
  
\newline 
@ %def xlc_try
\layout Section

Extracting the code
\layout Standard

This project can be tangled from LyX if you set your 
\begin_inset Quotes eld
\end_inset 

Program
\begin_inset Quotes erd
\end_inset 

 convertor to call a generic script that always extracts a scrap named 
\family typewriter 
build-script
\family default 
 and executes it.
 Here is an example of such a generic script:
\layout LyX-Code

#!/bin/sh
\newline 
notangle -Rbuild-script $1 | env NOWEB_SOURCE=$1 sh
\layout Standard

This section defines our build-script, which extracts the code.
\layout Scrap

<<build-script>>=
\newline 
#!/bin/sh
\newline 
if [ -z "$NOWEB_SOURCE" ]; then NOWEB_SOURCE=listerrors.nw; fi
\newline 
notangle -Rlisterrors ${NOWEB_SOURCE} > listerrors
\newline 
chmod +x listerrors
\newline 
@
\layout Section

Indices
\layout Standard

This section provides cross-references into the rest of the program.
\layout Subsection

Macros
\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
nowebchunks
\end_inset 


\layout Subsection

Identifiers
\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
nowebindex
\end_inset 


\the_end