-#LyX 1.3 created this file. For more info see http://www.lyx.org/
-\lyxformat 221
+#LyX 1.5.0svn created this file. For more info see http://www.lyx.org/
+\lyxformat 276
+\begin_document
+\begin_header
\textclass literate-article
\begin_preamble
%
\end_preamble
\language english
\inputencoding auto
-\fontscheme pslatex
+\font_roman times
+\font_sans helvet
+\font_typewriter courier
+\font_default_family default
+\font_sc false
+\font_osf false
+\font_sf_scale 100
+\font_tt_scale 100
\graphics default
\paperfontsize default
\spacing single
-\papersize Default
-\paperpackage a4
-\use_geometry 0
+\papersize default
+\use_geometry false
\use_amsmath 0
-\use_natbib 0
-\use_numerical_citations 0
+\use_esint 0
+\cite_engine basic
+\use_bibtopic false
\paperorientation portrait
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\defskip medskip
\quotes_language english
-\quotes_times 2
\papercolumns 1
\papersides 1
\paperpagestyle default
+\tracking_changes false
+\output_changes false
+\end_header
-\layout Title
+\begin_body
+
+\begin_layout Title
LyX listerrors:
-\newline
+\newline
rewritten in Python
-\layout Author
+\end_layout
+
+\begin_layout Author
Kayvan A.
Sylvan
-\newline
+\newline
-\begin_inset LatexCommand \url{mailto:kayvan@sylvan.com}
+\begin_inset LatexCommand url
+target "mailto:kayvan@sylvan.com"
+\end_inset
-\end_inset
+\end_layout
-\layout Date
+\begin_layout Date
3/15/2002
-\layout Abstract
+\end_layout
+
+\begin_layout Abstract
The listerrors program used to be compiled as a C program and installed
- as
-\emph on
+ as
+\emph default
+
+\emph on
BINDIR/listerrors
-\emph default
+\emph default
along with LyX in order to perform some simple re-formatting of noweb and
GCC error messages.
This document describes and implements the Python version of the same program.
-\layout Standard
+\end_layout
+
+\begin_layout Standard
-\begin_inset LatexCommand \tableofcontents{}
+\begin_inset LatexCommand tableofcontents
+\end_inset
-\end_inset
+\end_layout
-\layout Section
+\begin_layout Section
Introduction
-\layout Standard
+\end_layout
+
+\begin_layout Standard
The motivation for this program was Bugzilla bug 190
\begin_inset Foot
-collapsed true
+status collapsed
-\layout Standard
+\begin_layout Standard
Visit
-\begin_inset LatexCommand \url{http://bugzilla.lyx.org/show_bug.cgi?id=190}
-
-\end_inset
+\begin_inset LatexCommand url
+target "http://bugzilla.lyx.org/show_bug.cgi?id=190"
+\end_inset
for the details.
-\end_inset
+\end_layout
+
+\end_inset
dealing with the
\begin_inset Quotes eld
-\end_inset
+\end_inset
listerrors
\begin_inset Quotes erd
-\end_inset
+\end_inset
executable.
-\layout Standard
+\end_layout
+
+\begin_layout Standard
What is
\begin_inset Quotes eld
-\end_inset
+\end_inset
listerrors
\begin_inset Quotes erd
-\end_inset
+\end_inset
? Usually, LyX has great support for parsing of error messages.
For each error in the log file, LyX pops up an error box at that location
(similar to LaTeX errors).
When dealing with Literate Programs, you have
\begin_inset Quotes eld
-\end_inset
+\end_inset
noweb
\begin_inset Foot
-collapsed true
+status collapsed
-\layout Standard
+\begin_layout Standard
See
-\begin_inset LatexCommand \url{http://www.eecs.harvard.edu/~nr/noweb}
-
-\end_inset
+\begin_inset LatexCommand url
+target "http://www.eecs.harvard.edu/~nr/noweb"
+\end_inset
for more information about noweb.
-\end_inset
+\end_layout
+
+\end_inset
\begin_inset Quotes erd
-\end_inset
+\end_inset
as well as gcc error messages (and potentially others).
The listerrors program attempts to standardize these error messages to
a format that LyX can parse and react to.
-\layout Standard
+\end_layout
+
+\begin_layout Standard
In a nutshell, the problems with the old implementation of listerrors that
bug 190 refers to were::
-\layout Enumerate
+\end_layout
+
+\begin_layout Enumerate
It was a C program and it was installed in the user path in the same directory
as LyX.
- Having such a generically named binary in, for example,
-\emph on
+ Having such a generically named binary in, for example,
+\emph default
+
+\emph on
/usr/bin
-\emph default
+\emph default
, was potentially confusing.
-\layout Enumerate
+\end_layout
+
+\begin_layout Enumerate
It required that noweb be installed on the compiling machine (the source
- was extracted by noweb from
-\emph on
+ was extracted by noweb from
+\emph default
+
+\emph on
SRCDIR/examples/Literate.lyx
-\emph default
+\emph default
, compiled and installed by
\begin_inset Quotes eld
-\end_inset
+\end_inset
make install
\begin_inset Quotes erd
-\end_inset
+\end_inset
).
-\layout Standard
+\end_layout
+
+\begin_layout Standard
The new version deals with these problems in the following fashion:
-\layout Enumerate
+\end_layout
+
+\begin_layout Enumerate
Both the example file (this document) and the program are to be added to
the LyX CVS repository.
-\layout Enumerate
+\end_layout
+
+\begin_layout Enumerate
-The program itself will be installed in
-\emph on
+The program itself will be installed in
+\emph default
+
+\emph on
SHAREDIR/lyx/scripts
-\emph default
+\emph default
, along with other LyX-specific helper scripts.
-\layout Standard
+\end_layout
+
+\begin_layout Standard
In the design and implementation of this new
\begin_inset Quotes eld
-\end_inset
+\end_inset
listerrors
\begin_inset Quotes erd
-\end_inset
+\end_inset
, the Python
\begin_inset Foot
-collapsed true
+status collapsed
-\layout Standard
+\begin_layout Standard
See the Python home page (
-\begin_inset LatexCommand \url{http://www.python.org}
-
-\end_inset
+\begin_inset LatexCommand url
+target "http://www.python.org"
+\end_inset
) for more information.
-\end_inset
+\end_layout
+
+\end_inset
language was chosen since it is fully multi-platform and provides a very
uniform and easy to read syntax.
This re-write also simplifies the code for
\begin_inset Quotes eld
-\end_inset
+\end_inset
listerrors
\begin_inset Quotes erd
-\end_inset
+\end_inset
greatly.
Python is installed by default on all modern Linux systems and is freely
available for all other platforms.
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<listerrors>>=
-\newline
-#!/usr/bin/python
-\newline
+\newline
+#!/usr/bin/python -tt
+\newline
"""reformat noweb and compiler errors for LyX.
-\newline
+\newline
-\newline
+\newline
Expects to read from stdin and output to stdout.
-\newline
+\newline
"""
-\newline
+\newline
-\newline
+\newline
__author__ = "Kayvan A.
Sylvan <kayvan@sylvan.com>"
-\newline
-__date__ = "$Date: 2003/01/17 13:50:10 $"
-\newline
-__version__ = "$Revision: 1.3 $"
-\newline
+\newline
+__date__ = "$Date: 2005/07/18 09:42:26 $"
+\newline
+__version__ = "$Revision: 1.5 $"
+\newline
__credits__ = """Edmar Wienskoski Jr.
<edmar-w-jr@technologist.com>
-\newline
+\newline
original Literate support for LyX.
-\newline
+\newline
Bernard Michael Hurley <berhardh@westherts.ac.uk>
-\newline
+\newline
modifications to original listerrors."""
-\newline
-__copyright__ = "Copyright 2002 - The LyX team."
-\newline
+\newline
+__copyright__ = "Copyright 2002 - Kayvan Sylvan."
+\newline
-\newline
+\newline
import sys, string
-\newline
+\newline
-\newline
+\newline
<<Function Bodies>>
-\newline
+\newline
-\newline
+\newline
if __name__ == "__main__":
-\newline
+\newline
main()
-\newline
+\newline
@
-\layout Section
+\end_layout
+
+\begin_layout Section
LaTeX style error message
-\layout Standard
+\end_layout
+
+\begin_layout Standard
The following function mimics the TeX error message format.
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Function Bodies>>=
-\newline
+\newline
def write_error(msg, tool = "noweb", line_number = 1):
-\newline
+\newline
"""Write out the given message in TeX error style.
-\newline
+\newline
-\newline
+\newline
called like: write_error(msg, tool, line_number)."""
-\newline
+\newline
print "! Build Error: ==> %s ==>
-\backslash
+\backslash
n" % (tool),
-\newline
+\newline
print " ...
-\backslash
+\backslash
n
-\backslash
+\backslash
nl.%d ...
-\backslash
+\backslash
n" % (line_number),
-\newline
+\newline
if type(msg) == type("str"): # simple string
-\newline
+\newline
print msg
-\newline
+\newline
else: # some kind of list (sequence or tuple)
-\newline
+\newline
for m in msg:
-\newline
+\newline
if m != "": print m,
-\newline
+\newline
print
-\newline
+\newline
-\newline
+\newline
@ %def write_error
-\layout Section
+\end_layout
+
+\begin_layout Section
Filtering errors
-\layout Standard
+\end_layout
+
+\begin_layout Standard
The only complication in our filtering code is that some parsers might need
to push back lines that are read in to be read again later.
We solve this problem by implementing a
\begin_inset Quotes eld
-\end_inset
+\end_inset
getline
\begin_inset Quotes erd
-\end_inset
+\end_inset
and
\begin_inset Quotes eld
-\end_inset
+\end_inset
pushline
\begin_inset Quotes erd
-\end_inset
+\end_inset
set of functions:
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Function Bodies>>=
-\newline
+\newline
__lines = [] # lines pushed back
-\newline
+\newline
-\newline
+\newline
def getline(file = sys.stdin):
-\newline
+\newline
"""read a line from internal stack or from file.
-\newline
+\newline
-\newline
+\newline
optional file argument defaults to sys.stdin."""
-\newline
+\newline
global __lines
-\newline
+\newline
lines = __lines
-\newline
+\newline
if lines:
-\newline
+\newline
line = lines.pop()
-\newline
+\newline
else:
-\newline
+\newline
line = file.readline()
-\newline
+\newline
return line
-\newline
+\newline
-\newline
+\newline
@ %def getline
-\layout Standard
+\end_layout
+
+\begin_layout Standard
And now for the corresponding pushline function:
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Function Bodies>>=
-\newline
+\newline
def pushline(line):
-\newline
+\newline
"push a line onto the pushback stack."
-\newline
+\newline
global __lines
-\newline
+\newline
lines = __lines
-\newline
+\newline
lines.append(line)
-\newline
+\newline
-\newline
+\newline
@ %def pushline
-\layout Standard
+\end_layout
+
+\begin_layout Standard
The main() entry point function is extremely simple.
Note that this version of
\begin_inset Quotes eld
-\end_inset
+\end_inset
listerrors
\begin_inset Quotes erd
-\end_inset
+\end_inset
takes no options and simply filters, attempting simply to match against
the known error message patterns.
The listerrors C program handled a single-character command-line argument
that the current code no longer needs.
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Function Bodies>>=
-\newline
+\newline
def main():
-\newline
+\newline
"""Entry point for listerrors.
Takes no options.
-\newline
+\newline
-\newline
+\newline
Reads stdin and writes to stdout.
Filter errors"""
-\newline
+\newline
-\newline
+\newline
while 1:
-\newline
+\newline
line = getline()
-\newline
+\newline
if line == "": break
-\newline
+\newline
<<Check line against patterns and take action>>
-\newline
+\newline
@ %def main
-\layout Standard
+\end_layout
+
+\begin_layout Standard
For each line read in, we need to find out if it matches any of our tools
(noweb, gcc, etc.) and act accordingly.
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Check line against patterns and take action>>=
-\newline
+\newline
try_patterns_dispatch = [ noweb_try, gcc_try, xlc_try ]
-\newline
+\newline
for predicate in try_patterns_dispatch:
-\newline
+\newline
if predicate(line): break
-\newline
+\newline
@
-\layout Section
+\end_layout
+
+\begin_layout Section
Different Error Formats
-\layout Standard
+\end_layout
+
+\begin_layout Standard
The following sections handle the various error message formats that we
recognize in this program.
-\layout Subsection
+\end_layout
+
+\begin_layout Subsection
noweb errors
-\layout Standard
+\end_layout
+
+\begin_layout Standard
Noweb errors are output on a single line, so examining just the current
line is enough.
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Function Bodies>>=
-\newline
+\newline
def noweb_try(line):
-\newline
+\newline
"""see if line is a noweb error.
-\newline
+\newline
-\newline
+\newline
Returns 1 on success, 0 otherwise.
Outputs on stdout."""
-\newline
+\newline
retval = 0
-\newline
+\newline
<<Look for the unescaped angle-brackets in documentation>>
-\newline
+\newline
<<Look for anything with double angle brackets>>
-\newline
+\newline
<<Last ditch effort scan for specific strings>>
-\newline
+\newline
return retval
-\newline
+\newline
-\newline
+\newline
@ %def noweb_try
-\layout Standard
+\end_layout
+
+\begin_layout Standard
First, we look for the
\begin_inset Quotes eld
-\end_inset
+\end_inset
unescaped < < in documentation chunk
\begin_inset Quotes erd
-\end_inset
+\end_inset
message.
This is the only message with an associated line number from noweb.
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Look for the unescaped angle-brackets in documentation>>=
-\newline
+\newline
if string.find(line, ": unescaped << in documentation chunk") != -1:
-\newline
+\newline
line_parts = string.split(line, ':')
-\newline
+\newline
num_str = line_parts[1]
-\newline
+\newline
num_len = len(num_str)
-\newline
+\newline
i = 0
-\newline
+\newline
while i < num_len and (num_str[i] in string.digits): i = i + 1
-\newline
+\newline
if i == num_len:
-\newline
+\newline
write_error(":" + line_parts[2], "noweb", int(num_str))
-\newline
+\newline
retval = 1
-\newline
+\newline
@
-\layout Standard
+\end_layout
+
+\begin_layout Standard
Some noweb messages are simply about undefined scraps.
These can be seen by looking for matching double-angle-brackets.
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Look for anything with double angle brackets>>=
-\newline
+\newline
if (not retval):
-\newline
+\newline
left = string.find(line, "<<")
-\newline
+\newline
if (left != -1) and ((left + 2) < len(line)) and
-\backslash
+\backslash
-\newline
+\newline
(string.find(line[left+2:], ">>") != -1):
-\newline
+\newline
write_error(line, "noweb");
-\newline
+\newline
retval = 1;
-\newline
+\newline
@
-\layout Standard
+\end_layout
+
+\begin_layout Standard
Finally, here is an additional list of explicit strings to check for.
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Last ditch effort scan for specific strings>>=
-\newline
+\newline
if (not retval):
-\newline
+\newline
msgs_to_try = ("couldn't open file",
-\newline
+\newline
"couldn't open temporary file",
-\newline
+\newline
"error writing temporary file",
-\newline
+\newline
"ill-formed option",
-\newline
+\newline
"unknown option",
-\newline
+\newline
"Bad format sequence",
-\newline
+\newline
"Can't open output file",
-\newline
+\newline
"Can't open temporary file",
-\newline
+\newline
"Capacity exceeded:",
-\newline
+\newline
"Ignoring unknown option -",
-\newline
+\newline
"This can't happen:",
-\newline
+\newline
"non-numeric line number in")
-\newline
+\newline
for msg in msgs_to_try:
-\newline
+\newline
if string.find(line, msg) != -1:
-\newline
+\newline
write_error(line, "noweb")
-\newline
+\newline
retval = 1
-\newline
+\newline
break
-\newline
+\newline
@
-\layout Subsection
+\end_layout
+
+\begin_layout Subsection
gcc errors
-\layout Standard
+\end_layout
+
+\begin_layout Standard
The gcc errors can be multi-line, with the following format:
-\layout LyX-Code
+\end_layout
+
+\begin_layout LyX-Code
foo.c: In function `main':
-\newline
+\newline
foo.c:3: `bar' undeclared (first use in this function)
-\newline
+\newline
foo.c:3: (Each undeclared identifier is reported only once
-\newline
+\newline
foo.c:3: for each function it appears in.)
-\newline
+\newline
foo.c:3: parse error before `x'
-\layout Standard
+\end_layout
+
+\begin_layout Standard
In order to parse this, the gcc error handler has to look ahead and return
any and all lines that do not match the expected pattern.
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Function Bodies>>=
-\newline
+\newline
def gcc_try(line):
-\newline
+\newline
"""See if line is a gcc error.
Read ahead to handle all the lines.
-\newline
+\newline
-\newline
+\newline
Returns 1 on success, 0 otherwise.
Outputs on stdout."""
-\newline
+\newline
retval = 0
-\newline
+\newline
<<Handle the gcc error message>>
-\newline
+\newline
return retval
-\newline
+\newline
-\newline
+\newline
@ %def gcc_try
-\layout Standard
+\end_layout
+
+\begin_layout Standard
The error message starts with a gcc header (as above) without an associated
line number.
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Handle the gcc error message>>=
-\newline
+\newline
first_space = string.find(line, ' ')
-\newline
+\newline
if first_space > 1: # The smallest would be "X: "
-\newline
+\newline
if line[first_space - 1] == ':':
-\newline
+\newline
header_to_see = line[:first_space - 1]
-\newline
+\newline
next_line = getline()
-\newline
+\newline
if next_line and next_line[:first_space - 1] == header_to_see:
-\newline
+\newline
num_end = first_space
-\newline
+\newline
while next_line[num_end] in string.digits: num_end = num_end + 1
-\newline
+\newline
if num_end > first_space: # good!
-\newline
+\newline
<<Accumulate gcc error lines and print it>>
-\newline
+\newline
else: # oops! Not a gcc error.
-\newline
+\newline
pushline(next_line)
-\newline
+\newline
elif next_line:
-\newline
+\newline
pushline(next_line) # return this line to input stream
-\newline
+\newline
@
-\layout Standard
+\end_layout
+
+\begin_layout Standard
At the point in the code that we know that we are in the middle of an error
message, we do the following:
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Accumulate gcc error lines and print it>>=
-\newline
+\newline
num_str = next_line[first_space:num_end]
-\newline
+\newline
msgs = [line[first_space:]]
-\newline
+\newline
msgs.append(next_line[num_end + 1:])
-\newline
+\newline
header_to_see = next_line[:num_end]
-\newline
+\newline
next_line = getline()
-\newline
+\newline
while next_line and next_line[:num_end] == header_to_see:
-\newline
+\newline
msgs.append(next_line[num_end + 1:])
-\newline
+\newline
next_line = getline()
-\newline
+\newline
if next_line: pushline(next_line)
-\newline
+\newline
write_error(msgs, "gcc", int(num_str))
-\newline
+\newline
retval = 1
-\newline
+\newline
@
-\layout Subsection
+\end_layout
+
+\begin_layout Subsection
xlc errors
-\layout Standard
+\end_layout
+
+\begin_layout Standard
A xlc error message is easy to identify.
Every error message starts with a quoted string with no spaces, a comma,
a space, the word
\begin_inset Quotes eld
-\end_inset
+\end_inset
line
\begin_inset Quotes erd
-\end_inset
+\end_inset
, a space, and some variable text.
The following routine tests if a given buffer line matches this criteria
(this code would probably be simpler if I used the
\begin_inset Quotes eld
-\end_inset
+\end_inset
re
\begin_inset Quotes erd
-\end_inset
+\end_inset
regexp module, but we don't really need the full regular expression engine
here).
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<Function Bodies>>=
-\newline
+\newline
def xlc_try(line):
-\newline
+\newline
"""see if line is an xlc error.
-\newline
+\newline
-\newline
+\newline
Returns 1 on success, 0 otherwise.
Outputs on stdout."""
-\newline
+\newline
retval = 0
-\newline
+\newline
if line[0] == '"': # This is the first character of all xlc errors
-\newline
+\newline
next_quote = string.find(line, '"', 1)
-\newline
+\newline
first_space = string.find(line, ' ')
-\newline
+\newline
if (next_quote != -1) and (first_space > next_quote): # no space inisde
quotes
-\newline
+\newline
if line[first_space - 1:first_space + 6] == ", line ":
-\newline
+\newline
num_start = num_end = first_space + 6
-\newline
+\newline
while line[num_end] in string.digits: num_end = num_end + 1
-\newline
+\newline
if num_end > num_start:
-\newline
+\newline
write_error(line, "xlc", int(line[num_start : num_end]))
-\newline
+\newline
retval = 1
-\newline
+\newline
return retval
-\newline
+\newline
-\newline
+\newline
@ %def xlc_try
-\layout Section
+\end_layout
+
+\begin_layout Section
Extracting the code
-\layout Standard
+\end_layout
+
+\begin_layout Standard
This project can be tangled from LyX if you set your
\begin_inset Quotes eld
-\end_inset
+\end_inset
Program
\begin_inset Quotes erd
-\end_inset
+\end_inset
- convertor to call a generic script that always extracts a scrap named
-\family typewriter
+ convertor to call a generic script that always extracts a scrap named
+\family default
+
+\family typewriter
build-script
-\family default
+\family default
and executes it.
Here is an example of such a generic script:
-\layout LyX-Code
+\end_layout
+
+\begin_layout LyX-Code
#!/bin/sh
-\newline
+\newline
notangle -Rbuild-script $1 | env NOWEB_SOURCE=$1 sh
-\layout Standard
+\end_layout
+
+\begin_layout Standard
This section defines our build-script, which extracts the code.
-\layout Scrap
+\end_layout
+
+\begin_layout Scrap
<<build-script>>=
-\newline
+\newline
#!/bin/sh
-\newline
+\newline
if [ -z "$NOWEB_SOURCE" ]; then NOWEB_SOURCE=listerrors.nw; fi
-\newline
+\newline
notangle -Rlisterrors ${NOWEB_SOURCE} > listerrors
-\newline
+\newline
chmod +x listerrors
-\newline
+\newline
@
-\layout Section
+\end_layout
+
+\begin_layout Section
Indices
-\layout Standard
+\end_layout
+
+\begin_layout Standard
This section provides cross-references into the rest of the program.
-\layout Subsection
+\end_layout
+
+\begin_layout Subsection
Macros
-\layout Standard
+\end_layout
+
+\begin_layout Standard
\begin_inset ERT
-status Collapsed
+status collapsed
-\layout Standard
+\begin_layout Standard
-\backslash
+\backslash
nowebchunks
-\end_inset
+\end_layout
+
+\end_inset
-\layout Subsection
+\end_layout
+
+\begin_layout Subsection
Identifiers
-\layout Standard
+\end_layout
+
+\begin_layout Standard
\begin_inset ERT
-status Collapsed
+status collapsed
-\layout Standard
+\begin_layout Standard
-\backslash
+\backslash
nowebindex
-\end_inset
+\end_layout
+
+\end_inset
+
+\end_layout
-\the_end
+\end_body
+\end_document