1 #LyX 1.6.0svn created this file. For more info see http://www.lyx.org/
5 \textclass literate-article
8 % This relaxes the noweb constraint that chunks are
9 % never broken across pages.
11 % This is from the noweb FAQ
13 \def\nwendcode{\endtrivlist \endgroup}
14 \let\nwdocspar=\smallbreak
17 \inputencoding default
20 \font_typewriter default
21 \font_default_family default
28 \paperfontsize default
37 \paperorientation portrait
40 \paragraph_separation indent
42 \quotes_language english
45 \paperpagestyle default
46 \tracking_changes false
61 Sylvan <kayvan@sylvan.com>
68 \begin_layout Abstract
69 This document describes and implements a perl script for importing noweb
73 \begin_layout Standard
74 \begin_inset CommandInset toc
75 LatexCommand tableofcontents
82 \begin_layout Standard
83 \begin_inset Newpage newpage
93 \begin_layout Standard
94 Since version 1.0.1, LyX now supports Literate Programming using
99 This addition to LyX made it very pleasant to write programs in the literate
100 style (like this one).
101 In addition to being able to write new literate programs, it would be quite
106 code could be imported into LyX in some fashion.
107 That's where this program comes in.
110 \begin_layout Standard
124 \begin_inset Newline newline
128 \begin_inset Newline newline
132 \begin_inset Newline newline
135 # Copyright (C) 1999 Kayvan A.
136 Sylvan <kayvan@sylvan.com>
137 \begin_inset Newline newline
140 # You are free to use and modify this code under the terms of
141 \begin_inset Newline newline
144 # the GNU General Public Licence version 2 or later.
145 \begin_inset Newline newline
149 \begin_inset Newline newline
152 # Written with assistance from:
153 \begin_inset Newline newline
156 # Edmar Wienskoski Jr.
157 <edmar-w-jr@technologist.com>
158 \begin_inset Newline newline
161 # Amir Karger <karger@post.harvard.edu>
162 \begin_inset Newline newline
166 \begin_inset Newline newline
169 # $Id: noweb2lyx.lyx,v 1.5 2005/07/18 09:42:27 jamatos Exp $
170 \begin_inset Newline newline
174 \begin_inset Newline newline
177 # NOTE: This file was automatically generated from noweb2lyx.lyx using noweb.
178 \begin_inset Newline newline
182 \begin_inset Newline newline
185 <<Setup variables from user supplied args>>
186 \begin_inset Newline newline
190 \begin_inset Newline newline
193 <<Convert noweb to LyX>>
194 \begin_inset Newline newline
200 \begin_layout Section
201 The Noweb file defined
204 \begin_layout Standard
209 file is a collection of documentation and code chunks.
210 Documentation chunks simply start with an ``@'' and have no name:
213 \begin_layout LyX-Code
214 @ Here is some documentation.
215 \begin_inset Newline newline
218 We can do arbitrary LaTeX code here.
219 \begin_inset Newline newline
226 \begin_layout Standard
227 Code chunks look like this:
230 \begin_layout LyX-Code
235 \begin_layout Plain Layout
246 \begin_layout Plain Layout
254 \begin_inset Newline newline
258 code for the chunk goes here ...}
259 \begin_inset Newline newline
265 \begin_layout Standard
266 The ``@'' is a necessary delimiter to end the code chunk.
267 The other form that the ``@'' line takes is as follows:
270 \begin_layout LyX-Code
275 \begin_layout Plain Layout
286 \begin_layout Plain Layout
294 \begin_inset Newline newline
298 code for the chunk ...}
299 \begin_inset Newline newline
302 @ %def identifier1 identifier2
305 \begin_layout Standard
306 In the latter form, we are declaring to
310 that this code chunk defines identifier1, identifier2, etc.
313 \begin_layout Standard
314 When first tackling this problem, I spoke with members of the LyX team that
315 knew about the literate programming extensions and reLyX (the LaTeX importing
319 \begin_layout Standard
320 One of the first ideas was to extend the reLyX code to understand the
325 This proved to be too hard and presents other problems
329 \begin_layout Plain Layout
330 Not the least of these problems is the fact that << is a quote in French.
336 On the other hand, it turns out that reLyX contains a very useful literal
338 If the input file contains the construct
341 \begin_layout LyX-Code
345 \begin_inset Newline newline
350 \begin_inset Newline newline
358 \begin_layout Standard
359 then reLyX will copy the surrounded code to the output file verbatim.
360 Given this, the first part of the translation is easy; we simply have to
361 copy the code chunks into an intermediate file that surrounds them with
377 \begin_layout Standard
378 Once reLyX is done with the input file, the problem is reduced to changing
379 the code chunks from LyX's LaTeX layout to the Scrap layout.
382 \begin_layout Standard
383 There is one final constraint on
388 We want to be able to run it as a simple pre-processor and post-processor
390 We can accomplish this by setting the flags
403 \begin_layout Plain Layout
434 \begin_layout Plain Layout
451 before we reach the main conversion code.
454 \begin_layout Standard
455 With all that preamble out of the way, we now have the basic high-level
456 outline for our code:
460 <<Convert noweb to LyX>>=
461 \begin_inset Newline newline
465 \begin_inset Newline newline
468 <<Transform noweb for reLyX>>
469 \begin_inset Newline newline
473 \begin_inset Newline newline
476 if ((!$pre_only) && (!$post_only)) {
477 \begin_inset Newline newline
480 <<Run reLyX on intermediate file>>
481 \begin_inset Newline newline
485 \begin_inset Newline newline
489 \begin_inset Newline newline
493 \begin_inset Newline newline
497 \begin_inset Newline newline
501 \begin_inset Newline newline
507 \begin_layout Section
508 Making a file that reLyX can process
511 \begin_layout Standard
512 In this section, we present the code that performs the task of creating
513 the intermediate file that reLyX can process, using the algorithm that
515 This algorithm is outlined in the code that follows:
519 <<Transform noweb for reLyX>>=
520 \begin_inset Newline newline
523 <<Setup INPUT and OUTPUT>>
524 \begin_inset Newline newline
527 inputline: while(<INPUT>)
528 \begin_inset Newline newline
532 \begin_inset Newline newline
545 >=/) { # Beginning of a noweb scrap
546 \begin_inset Newline newline
549 <<Read in and output the noweb code chunk>>
550 \begin_inset Newline newline
555 s+(.*)/) { # Beginning of a documentation chunk
556 \begin_inset Newline newline
559 print OUTPUT $1; # We do not need the ``@'' part
560 \begin_inset Newline newline
571 ]/) { # noweb quoted code
572 \begin_inset Newline newline
575 <<Perform special input quoting of [[var]]>>
576 \begin_inset Newline newline
580 \begin_inset Newline newline
583 print OUTPUT; # Just let the line pass through
584 \begin_inset Newline newline
588 \begin_inset Newline newline
592 \begin_inset Newline newline
595 <<Close INPUT and OUTPUT>>
596 \begin_inset Newline newline
602 \begin_layout Standard
603 In the code above, we do some pre-processing of the noweb ``[[...]]'' construct.
604 This avoids some problems with reLyX confusing lists composed of ``[[...]]''
609 <<Perform special input quoting of [[var]]>>=
610 \begin_inset Newline newline
620 \begin_inset Newline newline
624 \begin_inset Newline newline
630 \begin_layout Standard
644 \begin_layout Plain Layout
661 file, once we have identified a
665 code chunk, we transform it into a form that is usable by reLyX.
669 <<Read in and output the noweb code chunk>>=
670 \begin_inset Newline newline
673 <<Save the beginning of the scrap to savedScrap>>
674 \begin_inset Newline newline
677 <<Concatenate the rest of the scrap>>
678 \begin_inset Newline newline
681 <<print out the scrap in a reLyXskip block>>
682 \begin_inset Newline newline
688 \begin_layout Subsection
689 File input and output for the pre-processing step
692 \begin_layout Standard
710 \begin_layout Plain Layout
741 \begin_layout Plain Layout
758 to read and write files.
759 In the code fragment above, we need to read from the input file and write
760 to a file that will be later transformed by reLyX.
761 If we are being called only to pre-process the input file, then there is
762 no need to create a temporary file.
766 <<Setup INPUT and OUTPUT>>=
767 \begin_inset Newline newline
771 \begin_inset Newline newline
774 &setup_files($input_file, $output_file);
775 \begin_inset Newline newline
779 \begin_inset Newline newline
782 $relyx_file = "temp$$";
783 \begin_inset Newline newline
786 &setup_files($input_file, $relyx_file);
787 \begin_inset Newline newline
791 \begin_inset Newline newline
797 \begin_layout Standard
798 This code uses a small perl subroutine,
811 \begin_layout Plain Layout
827 , which we define below:
832 \begin_inset Newline newline
836 \begin_inset Newline newline
840 \begin_inset Newline newline
843 open(INPUT, "<$in") || die "Cannot read $in: $!
846 \begin_inset Newline newline
849 open(OUTPUT, ">$out") || die "Cannot write $out: $!
852 \begin_inset Newline newline
856 \begin_inset Newline newline
862 \begin_layout Subsection
870 \begin_layout Standard
871 After we see the beginning of the scrap, we need to read in and save the
872 rest of the scrap for output.
876 <<Save the beginning of the scrap to savedScrap>>=
877 \begin_inset Newline newline
881 \begin_inset Newline newline
885 \begin_inset Newline newline
892 <<Concatenate the rest of the scrap>>=
893 \begin_inset Newline newline
896 scrapline: while (<INPUT>) {
897 \begin_inset Newline newline
900 last scrapline if /^@
903 \begin_inset Newline newline
907 \begin_inset Newline newline
911 \begin_inset Newline newline
915 \begin_inset Newline newline
920 s+$/) {$savedScrap .= $_; last switch; }
921 \begin_inset Newline newline
926 s+%def.*$/) {$savedScrap .= $_; last switch; }
927 \begin_inset Newline newline
932 s+(.*)$/) {$savedScrap .= "@
937 \begin_inset Newline newline
941 \begin_inset Newline newline
947 \begin_layout Subsection
948 Printing out the scrap
951 \begin_layout Standard
952 The final piece of the first pass of the conversion is done by this code.
956 <<print out the scrap in a reLyXskip block>>=
957 \begin_inset Newline newline
967 \begin_inset Newline newline
970 print OUTPUT $savedScrap;
971 \begin_inset Newline newline
983 \begin_inset Newline newline
986 print OUTPUT "$endLine";
987 \begin_inset Newline newline
993 \begin_layout Standard
994 Finally, we need to close the
1007 \begin_layout Plain Layout
1038 \begin_layout Plain Layout
1059 <<Close INPUT and OUTPUT>>=
1060 \begin_inset Newline newline
1064 \begin_inset Newline newline
1068 \begin_inset Newline newline
1074 \begin_layout Section
1078 \begin_layout Standard
1079 In this section, we describe and implement the code that runs reLyX on the
1093 \begin_layout Plain Layout
1113 \begin_layout Subsection
1114 Selecting the document class
1117 \begin_layout Standard
1118 In order to run reLyX, we need to know the article class of the input document
1119 (to choose the corresponding literate document layout).
1120 For this, we need to parse the intermediate file.
1124 <<Run reLyX on intermediate file>>=
1125 \begin_inset Newline newline
1128 <<Parse for document class>>
1129 \begin_inset Newline newline
1132 <<Run reLyX with document class>>
1133 \begin_inset Newline newline
1139 \begin_layout Standard
1140 In the code below, you'll see a strange regular expression to search for
1142 The reason for this kludge is that without it, we can't run
1150 file that is generated by LyX
1154 \begin_layout Plain Layout
1172 \begin_layout Plain Layout
1201 \begin_layout Plain Layout
1217 class and gets confused, so we have to obfuscate it slightly.
1223 With the regular expression as it is, we can actually run
1227 on itself and a produce a quite reasonable LyX file.
1231 <<Parse for document class>>=
1232 \begin_inset Newline newline
1235 open(INPUT, "<$relyx_file") ||
1236 \begin_inset Newline newline
1239 die "Cannot read $relyx_file: $!
1242 \begin_inset Newline newline
1245 $class = "article"; # default if none found
1246 \begin_inset Newline newline
1249 parse: while(<INPUT>) {
1250 \begin_inset Newline newline
1257 docu[m]entclass{(.*)}/) {
1258 \begin_inset Newline newline
1262 \begin_inset Newline newline
1266 \begin_inset Newline newline
1270 \begin_inset Newline newline
1274 \begin_inset Newline newline
1278 \begin_inset Newline newline
1284 \begin_layout Subsection
1285 Running reLyX with the corresponding literate document layout
1288 \begin_layout Standard
1289 Now that we know what the document class ought to be, we do:
1293 <<Run reLyX with document class>>=
1294 \begin_inset Newline newline
1297 $doc_class = "literate-" .
1299 \begin_inset Newline newline
1302 die "reLyX returned non-zero: $!
1305 \begin_inset Newline newline
1308 if (system("reLyX -c $doc_class $relyx_file"));
1309 \begin_inset Newline newline
1315 \begin_layout Standard
1316 reLyX performs the main bulk of the translation work.
1317 Note that if the ``literate-
1321 '' document layout is not found, then reLyX will fail with an error.
1322 In that case, you may need to modify your
1326 input file to a supported document type.
1329 \begin_layout Section
1330 Fixing the reLyX output
1333 \begin_layout Standard
1334 We need to perform some post-processing of what reLyX produces in order
1335 to have the best output for our literate document.
1336 The outline of the post-processing steps are:
1340 <<Fix up LyX file>>=
1341 \begin_inset Newline newline
1344 <<Setup INPUT and OUTPUT for the final output>>
1345 \begin_inset Newline newline
1348 line: while(<INPUT>)
1349 \begin_inset Newline newline
1353 \begin_inset Newline newline
1356 <<Fix code chunks in latex layout>>
1357 \begin_inset Newline newline
1360 <<Fix [[var]] noweb construct>>
1361 \begin_inset Newline newline
1364 print OUTPUT; # default
1365 \begin_inset Newline newline
1369 \begin_inset Newline newline
1372 <<Close INPUT and OUTPUT>>
1373 \begin_inset Newline newline
1379 \begin_layout Standard
1380 Note that in the perl code that is contained in the
1393 \begin_layout Plain Layout
1410 loop above, the perl construct
1424 \begin_layout Plain Layout
1441 is sufficient to restart the loop.
1442 We can use this construct to do some relatively complex parsing of the
1443 reLyX generated file.
1446 \begin_layout Subsection
1447 File input and output for the post-processing
1450 \begin_layout Standard
1464 \begin_layout Plain Layout
1495 \begin_layout Plain Layout
1512 is taken care of by this code:
1516 <<Setup INPUT and OUTPUT for the final output>>=
1517 \begin_inset Newline newline
1521 \begin_inset Newline newline
1524 &setup_files("$input_file", "$output_file");
1525 \begin_inset Newline newline
1529 \begin_inset Newline newline
1532 &setup_files("$relyx_file.lyx", "$output_file");
1533 \begin_inset Newline newline
1537 \begin_inset Newline newline
1543 \begin_layout Subsection
1544 Making sure the code chunks are in the Scrap layout
1547 \begin_layout Standard
1548 Now, as we outlined before, the final step is transforming the code-chunks
1549 which have been put into a LaTeX layout by LyX into the scrap layout.
1553 <<Fix code chunks in latex layout>>=
1554 \begin_inset Newline newline
1561 latex latex/) { # Beginning of some latex code
1562 \begin_inset Newline newline
1565 if (($line = <INPUT>) =~ /^
1567 s*<</) { # code scrap
1568 \begin_inset Newline newline
1571 <<Transform this chunk into layout scrap>>
1572 \begin_inset Newline newline
1576 \begin_inset Newline newline
1581 latex latex line + next line
1582 \begin_inset Newline newline
1585 print OUTPUT "$_$line";
1586 \begin_inset Newline newline
1590 \begin_inset Newline newline
1594 \begin_inset Newline newline
1598 \begin_inset Newline newline
1604 \begin_layout Standard
1605 When we are sure that we are in a code chunk, we must read in the rest of
1606 the code chunk and output a scrap layout for it:
1610 <<Transform this chunk into layout scrap>>=
1611 \begin_inset Newline newline
1623 \begin_inset Newline newline
1626 codeline: while (<INPUT>) {
1627 \begin_inset Newline newline
1631 \begin_inset Newline newline
1634 last codeline if /^@
1637 \begin_inset Newline newline
1641 \begin_inset Newline newline
1644 print OUTPUT $savedScrap;
1645 \begin_inset Newline newline
1648 <<Slurp up to the end of the latex layout>>
1649 \begin_inset Newline newline
1655 \begin_layout Standard
1656 Okay, now we just need to eat the rest of the latex layout.
1657 There should only be a few different types of lines for us to match:
1661 <<Slurp up to the end of the latex layout>>=
1662 \begin_inset Newline newline
1665 slurp: while (<INPUT>) {
1666 \begin_inset Newline newline
1674 \begin_inset Newline newline
1682 \begin_inset Newline newline
1688 \begin_inset Newline newline
1691 warn "confused by line: $_";
1692 \begin_inset Newline newline
1696 \begin_inset Newline newline
1702 \begin_layout Subsection
1714 \begin_layout Standard
1719 allows the user to use a special code quoting mechanism in documentation
1721 Fixing this ``[[quoted-code]]''
1725 syntax means putting the ``[[quoted-code]]'' in a LaTeX layout in the LyX
1727 Otherwise, LyX will backslash-quote the brackets, creating ugly output.
1728 The quoted-code is transformed by
1732 when it generates the final LaTeX code.
1736 <<Fix [[var]] noweb construct>>=
1737 \begin_inset Newline newline
1748 ]/) { # special code for [[var]]
1749 \begin_inset Newline newline
1775 \begin_inset Newline newline
1779 \begin_inset Newline newline
1783 \begin_inset Newline newline
1787 \begin_inset Newline newline
1793 \begin_layout Section
1794 Cleaning up intermediate files
1797 \begin_layout Standard
1798 The cleanup code is very simple:
1803 \begin_inset Newline newline
1806 system("rm -f $relyx_file*") unless ($post_only || $pre_only);
1807 \begin_inset Newline newline
1813 \begin_layout Section
1814 User supplied arguments
1817 \begin_layout Standard
1822 script understands two arguments, input-file and output-file.
1823 It is also set up to be used internally by reLyX to pre-process or postprocess
1824 files in the import pipeline.
1828 <<Setup variables from user supplied args>>=
1829 \begin_inset Newline newline
1832 &usage() if ($#ARGV < 1); # zero or one argument
1833 \begin_inset Newline newline
1836 if ($ARGV[0] eq "-pre") {
1837 \begin_inset Newline newline
1840 &usage unless ($#ARGV == 2);
1841 \begin_inset Newline newline
1844 $input_file = $ARGV[1]; $output_file = $ARGV[2]; $pre_only = 1;
1845 \begin_inset Newline newline
1848 } elsif ($ARGV[0] eq "-post") {
1849 \begin_inset Newline newline
1852 &usage unless ($#ARGV == 2);
1853 \begin_inset Newline newline
1856 $input_file = $ARGV[1]; $output_file = $ARGV[2]; $post_only = 1;
1857 \begin_inset Newline newline
1861 \begin_inset Newline newline
1864 &usage unless ($#ARGV == 1);
1865 \begin_inset Newline newline
1868 $input_file = $ARGV[0]; $output_file = $ARGV[1];
1869 \begin_inset Newline newline
1872 $pre_only = 0; $post_only = 0;
1873 \begin_inset Newline newline
1877 \begin_inset Newline newline
1880 @ %def input_file output_file pre_only post_only
1885 \begin_inset Newline newline
1889 \begin_inset Newline newline
1892 print "Usage: noweb2lyx [-pre | -post] input-file output-file
1893 \begin_inset Newline newline
1897 \begin_inset Newline newline
1900 If -pre is specified, only pre-processes the input-file for reLyX.
1901 \begin_inset Newline newline
1904 Similarly, in the case of -post, post-processes reLyX output.
1905 \begin_inset Newline newline
1908 In case of bugs, Email Kayvan Sylvan <kayvan
1913 \begin_inset Newline newline
1917 \begin_inset Newline newline
1921 \begin_inset Newline newline
1927 \begin_layout Section
1935 \begin_layout Standard
1936 The noweb2lyx script can be tangled from LyX if you set
1942 to call a generic script that always extracts a scrap named
1947 Here is an example of such a script:
1950 \begin_layout LyX-Code
1952 \begin_inset Newline newline
1955 notangle -Rbuild-script $1 | sh
1960 \begin_inset Newline newline
1964 \begin_inset Newline newline
1967 notangle -Rnoweb2lyx.in noweb2lyx.nw > noweb2lyx.in
1968 \begin_inset Newline newline
1971 sed -e "s=@PERL@=$PREFIX/bin/perl=" noweb2lyx.in > noweb2lyx
1972 \begin_inset Newline newline
1976 \begin_inset Newline newline
1982 \begin_layout Standard
1983 \begin_inset Newpage newpage
1989 \begin_layout Section*
1993 \begin_layout Standard
2006 \begin_layout Plain Layout
2018 \begin_layout Section*
2022 \begin_layout Standard
2035 \begin_layout Plain Layout