1 #LyX 2.1 created this file. For more info see http://www.lyx.org/
8 % This relaxes the noweb constraint that chunks are
9 % never broken across pages.
11 % This is from the noweb FAQ
13 \def\nwendcode{\endtrivlist \endgroup}
14 \let\nwdocspar=\smallbreak
16 \use_default_options false
20 \maintain_unincluded_children false
22 \language_package default
23 \inputencoding default
27 \font_typewriter default
29 \font_default_family default
30 \use_non_tex_fonts false
36 \default_output_format default
38 \bibtex_command default
39 \index_command default
40 \paperfontsize default
45 \use_package amsmath 0
46 \use_package amssymb 0
49 \use_package mathdots 1
50 \use_package mathtools 0
52 \use_package stackrel 0
53 \use_package stmaryrd 0
54 \use_package undertilde 0
56 \cite_engine_type default
60 \paperorientation portrait
70 \paragraph_separation indent
71 \paragraph_indentation default
72 \quotes_language english
75 \paperpagestyle default
76 \tracking_changes false
93 Sylvan <kayvan@sylvan.com>
100 \begin_layout Abstract
101 This document describes and implements a perl script for importing noweb
105 \begin_layout Standard
106 \begin_inset CommandInset toc
107 LatexCommand tableofcontents
114 \begin_layout Standard
115 \begin_inset Newpage newpage
121 \begin_layout Section
125 \begin_layout Standard
126 Since version 1.0.1, LyX now supports Literate Programming using
131 This addition to LyX made it very pleasant to write programs in the literate
132 style (like this one).
133 In addition to being able to write new literate programs, it would be quite
138 code could be imported into LyX in some fashion.
139 That's where this program comes in.
142 \begin_layout Standard
154 \begin_layout Standard
155 \begin_inset Flex Chunk
158 \begin_layout Plain Layout
160 \begin_inset Argument 1
163 \begin_layout Plain Layout
172 \begin_layout Plain Layout
177 \begin_layout Plain Layout
179 # Copyright (C) 1999 Kayvan A.
180 Sylvan <kayvan@sylvan.com>
183 \begin_layout Plain Layout
185 # You are free to use and modify this code under the terms of
188 \begin_layout Plain Layout
190 # the GNU General Public Licence version 2 or later.
193 \begin_layout Plain Layout
198 \begin_layout Plain Layout
200 # Written with assistance from:
203 \begin_layout Plain Layout
205 # Edmar Wienskoski Jr.
206 <edmar-w-jr@technologist.com>
209 \begin_layout Plain Layout
211 # Amir Karger <karger@post.harvard.edu>
214 \begin_layout Plain Layout
219 \begin_layout Plain Layout
221 # $Id: noweb2lyx.lyx,v 1.5 2005/07/18 09:42:27 jamatos Exp $
224 \begin_layout Plain Layout
229 \begin_layout Plain Layout
231 # NOTE: This file was automatically generated from noweb2lyx.lyx using noweb.
234 \begin_layout Plain Layout
239 \begin_layout Plain Layout
241 <<Setup variables from user supplied args>>
244 \begin_layout Plain Layout
249 \begin_layout Plain Layout
251 <<Convert noweb to LyX>>
259 \begin_layout Section
260 The Noweb file defined
263 \begin_layout Standard
268 file is a collection of documentation and code chunks.
269 Documentation chunks simply start with an ``@'' and have no name:
272 \begin_layout LyX-Code
273 @ Here is some documentation.
276 \begin_layout Standard
280 \begin_layout Plain Layout
282 We can do arbitrary LaTeX code here.
285 \begin_layout Plain Layout
296 \begin_layout Standard
297 Code chunks look like this:
300 \begin_layout LyX-Code
305 \begin_layout Plain Layout
316 \begin_layout Plain Layout
324 \begin_inset Newline newline
328 code for the chunk goes here ...}
329 \begin_inset Newline newline
335 \begin_layout Standard
336 The ``@'' is a necessary delimiter to end the code chunk.
337 The other form that the ``@'' line takes is as follows:
340 \begin_layout LyX-Code
345 \begin_layout Plain Layout
356 \begin_layout Plain Layout
364 \begin_inset Newline newline
368 code for the chunk ...}
369 \begin_inset Newline newline
372 @ %def identifier1 identifier2
375 \begin_layout Standard
376 In the latter form, we are declaring to
380 that this code chunk defines identifier1, identifier2, etc.
383 \begin_layout Standard
384 When first tackling this problem, I spoke with members of the LyX team that
385 knew about the literate programming extensions and reLyX (the LaTeX importing
389 \begin_layout Standard
390 One of the first ideas was to extend the reLyX code to understand the
395 This proved to be too hard and presents other problems
399 \begin_layout Plain Layout
400 Not the least of these problems is the fact that << is a quote in French.
406 On the other hand, it turns out that reLyX contains a very useful literal
408 If the input file contains the construct
411 \begin_layout LyX-Code
415 \begin_inset Newline newline
420 \begin_inset Newline newline
428 \begin_layout Standard
429 then reLyX will copy the surrounded code to the output file verbatim.
430 Given this, the first part of the translation is easy; we simply have to
431 copy the code chunks into an intermediate file that surrounds them with
447 \begin_layout Standard
448 Once reLyX is done with the input file, the problem is reduced to changing
449 the code chunks from LyX's LaTeX layout to the Chunk layout.
452 \begin_layout Standard
453 There is one final constraint on
458 We want to be able to run it as a simple pre-processor and post-processor
460 We can accomplish this by setting the flags
473 \begin_layout Plain Layout
504 \begin_layout Plain Layout
521 before we reach the main conversion code.
524 \begin_layout Standard
525 With all that preamble out of the way, we now have the basic high-level
526 outline for our code:
529 \begin_layout Standard
530 \begin_inset Flex Chunk
533 \begin_layout Plain Layout
535 \begin_inset Argument 1
538 \begin_layout Plain Layout
547 \begin_layout Plain Layout
549 <<Transform noweb for reLyX>>
552 \begin_layout Plain Layout
557 \begin_layout Plain Layout
559 if ((!$pre_only) && (!$post_only)) {
562 \begin_layout Plain Layout
564 <<Run reLyX on intermediate file>>
567 \begin_layout Plain Layout
572 \begin_layout Plain Layout
577 \begin_layout Plain Layout
582 \begin_layout Plain Layout
587 \begin_layout Plain Layout
597 \begin_layout Section
598 Making a file that reLyX can process
601 \begin_layout Standard
602 In this section, we present the code that performs the task of creating
603 the intermediate file that reLyX can process, using the algorithm that
605 This algorithm is outlined in the code that follows:
608 \begin_layout Standard
609 \begin_inset Flex Chunk
612 \begin_layout Plain Layout
616 \begin_layout Plain Layout
618 <<Transform noweb for reLyX>>=
621 \begin_layout Plain Layout
623 <<Setup INPUT and OUTPUT>>
626 \begin_layout Plain Layout
628 inputline: while(<INPUT>)
631 \begin_layout Plain Layout
636 \begin_layout Plain Layout
648 >=/) { # Beginning of a noweb chunk
651 \begin_layout Plain Layout
653 <<Read in and output the noweb code chunk>>
656 \begin_layout Plain Layout
660 s+(.*)/) { # Beginning of a documentation chunk
663 \begin_layout Plain Layout
665 print OUTPUT $1; # We do not need the ``@'' part
668 \begin_layout Plain Layout
678 ]/) { # noweb quoted code
681 \begin_layout Plain Layout
683 <<Perform special input quoting of [[var]]>>
686 \begin_layout Plain Layout
691 \begin_layout Plain Layout
693 print OUTPUT; # Just let the line pass through
696 \begin_layout Plain Layout
701 \begin_layout Plain Layout
706 \begin_layout Plain Layout
708 <<Close INPUT and OUTPUT>>
716 \begin_layout Standard
717 In the code above, we do some pre-processing of the noweb ``[[...]]'' construct.
718 This avoids some problems with reLyX confusing lists composed of ``[[...]]''
722 \begin_layout Standard
726 \begin_layout Plain Layout
728 <<Perform special input quoting of [[var]]>>=
731 \begin_layout Plain Layout
742 \begin_layout Plain Layout
747 \begin_layout Plain Layout
757 \begin_layout Standard
771 \begin_layout Plain Layout
788 file, once we have identified a
792 code chunk, we transform it into a form that is usable by reLyX.
795 \begin_layout Standard
796 \begin_inset Flex Chunk
799 \begin_layout Plain Layout
801 \begin_inset Argument 1
804 \begin_layout Plain Layout
805 Read in and output the noweb code chunk
810 <<Save the beginning of the chunk to savedchunk>>
813 \begin_layout Plain Layout
815 <<Concatenate the rest of the chunk>>
818 \begin_layout Plain Layout
820 <<print out the chunk in a reLyXskip block>>
828 \begin_layout Subsection
829 File input and output for the pre-processing step
832 \begin_layout Standard
850 \begin_layout Plain Layout
881 \begin_layout Plain Layout
898 to read and write files.
899 In the code fragment above, we need to read from the input file and write
900 to a file that will be later transformed by reLyX.
901 If we are being called only to pre-process the input file, then there is
902 no need to create a temporary file.
905 \begin_layout Standard
906 \begin_inset Flex Chunk
909 \begin_layout Plain Layout
911 \begin_inset Argument 1
914 \begin_layout Plain Layout
915 Setup INPUT and OUTPUT
923 \begin_layout Plain Layout
925 &setup_files($input_file, $output_file);
928 \begin_layout Plain Layout
933 \begin_layout Plain Layout
935 $relyx_file = "temp$$";
938 \begin_layout Plain Layout
940 &setup_files($input_file, $relyx_file);
943 \begin_layout Plain Layout
953 \begin_layout Standard
954 This code uses a small perl subroutine,
967 \begin_layout Plain Layout
983 , which we define below:
986 \begin_layout Standard
990 \begin_layout Plain Layout
995 \begin_layout Plain Layout
1000 \begin_layout Plain Layout
1005 \begin_layout Plain Layout
1007 open(INPUT, "<$in") || die "Cannot read $in: $!
1012 \begin_layout Plain Layout
1014 open(OUTPUT, ">$out") || die "Cannot write $out: $!
1019 \begin_layout Plain Layout
1024 \begin_layout Plain Layout
1034 \begin_layout Subsection
1042 \begin_layout Standard
1043 After we see the beginning of the chunk, we need to read in and save the
1044 rest of the chunk for output.
1047 \begin_layout Standard
1048 \begin_inset Flex Chunk
1051 \begin_layout Plain Layout
1053 \begin_inset Argument 1
1056 \begin_layout Plain Layout
1057 Save the beginning of the chunk to savedchunk
1065 \begin_layout Plain Layout
1075 \begin_layout Standard
1079 \begin_layout Plain Layout
1081 <<Concatenate the rest of the chunk>>=
1084 \begin_layout Plain Layout
1086 chunkline: while (<INPUT>) {
1089 \begin_layout Plain Layout
1091 last chunkline if /^@
1096 \begin_layout Plain Layout
1101 \begin_layout Plain Layout
1106 \begin_layout Plain Layout
1111 \begin_layout Plain Layout
1115 s+$/) {$savedchunk .= $_; last switch; }
1118 \begin_layout Plain Layout
1122 s+%def.*$/) {$savedchunk .= $_; last switch; }
1125 \begin_layout Plain Layout
1129 s+(.*)$/) {$savedchunk .= "@
1136 \begin_layout Plain Layout
1141 \begin_layout Plain Layout
1151 \begin_layout Subsection
1152 Printing out the chunk
1155 \begin_layout Standard
1156 The final piece of the first pass of the conversion is done by this code.
1159 \begin_layout Standard
1160 \begin_inset Flex Chunk
1163 \begin_layout Plain Layout
1165 \begin_inset Argument 1
1168 \begin_layout Plain Layout
1169 print out the chunk in a reLyXskip block
1183 \begin_layout Plain Layout
1185 print OUTPUT $savedchunk;
1188 \begin_layout Plain Layout
1201 \begin_layout Plain Layout
1203 print OUTPUT "$endLine";
1211 \begin_layout Standard
1212 Finally, we need to close the
1225 \begin_layout Plain Layout
1256 \begin_layout Plain Layout
1276 \begin_layout Standard
1277 \begin_inset Flex Chunk
1280 \begin_layout Plain Layout
1282 \begin_inset Argument 1
1285 \begin_layout Plain Layout
1286 Close INPUT and OUTPUT
1294 \begin_layout Plain Layout
1304 \begin_layout Section
1308 \begin_layout Standard
1309 In this section, we describe and implement the code that runs reLyX on the
1323 \begin_layout Plain Layout
1343 \begin_layout Subsection
1344 Selecting the document class
1347 \begin_layout Standard
1348 In order to run reLyX, we need to know the article class of the input document
1349 (to choose the corresponding literate document layout).
1350 For this, we need to parse the intermediate file.
1353 \begin_layout Standard
1354 \begin_inset Flex Chunk
1357 \begin_layout Plain Layout
1359 \begin_inset Argument 1
1362 \begin_layout Plain Layout
1363 Run reLyX on intermediate file
1368 <<Parse for document class>>
1371 \begin_layout Plain Layout
1373 <<Run reLyX with document class>>
1381 \begin_layout Standard
1382 In the code below, you'll see a strange regular expression to search for
1384 The reason for this kludge is that without it, we can't run
1392 file that is generated by LyX
1396 \begin_layout Plain Layout
1414 \begin_layout Plain Layout
1443 \begin_layout Plain Layout
1459 class and gets confused, so we have to obfuscate it slightly.
1465 With the regular expression as it is, we can actually run
1469 on itself and a produce a quite reasonable LyX file.
1472 \begin_layout Standard
1473 \begin_inset Flex Chunk
1476 \begin_layout Plain Layout
1478 \begin_inset Argument 1
1481 \begin_layout Plain Layout
1482 Parse for document class
1487 open(INPUT, "<$relyx_file") ||
1490 \begin_layout Plain Layout
1492 die "Cannot read $relyx_file: $!
1497 \begin_layout Plain Layout
1499 $class = "article"; # default if none found
1502 \begin_layout Plain Layout
1504 parse: while(<INPUT>) {
1507 \begin_layout Plain Layout
1513 docu[m]entclass{(.*)}/) {
1516 \begin_layout Plain Layout
1521 \begin_layout Plain Layout
1526 \begin_layout Plain Layout
1531 \begin_layout Plain Layout
1536 \begin_layout Plain Layout
1546 \begin_layout Subsection
1547 Running reLyX with the corresponding literate document layout
1550 \begin_layout Standard
1551 Now that we know what the document class ought to be, we do:
1554 \begin_layout Standard
1558 \begin_layout Plain Layout
1560 <<Run reLyX with document class>>=
1563 \begin_layout Plain Layout
1565 $doc_class = "literate-" .
1569 \begin_layout Plain Layout
1571 die "reLyX returned non-zero: $!
1576 \begin_layout Plain Layout
1578 if (system("reLyX -c $doc_class $relyx_file"));
1581 \begin_layout Plain Layout
1591 \begin_layout Standard
1592 reLyX performs the main bulk of the translation work.
1593 Note that if the ``literate-
1597 '' document layout is not found, then reLyX will fail with an error.
1598 In that case, you may need to modify your
1602 input file to a supported document type.
1605 \begin_layout Section
1606 Fixing the reLyX output
1609 \begin_layout Standard
1610 We need to perform some post-processing of what reLyX produces in order
1611 to have the best output for our literate document.
1612 The outline of the post-processing steps are:
1615 \begin_layout Standard
1616 \begin_inset Flex Chunk
1619 \begin_layout Plain Layout
1621 \begin_inset Argument 1
1624 \begin_layout Plain Layout
1630 <<Setup INPUT and OUTPUT for the final output>>
1633 \begin_layout Plain Layout
1635 line: while(<INPUT>)
1638 \begin_layout Plain Layout
1643 \begin_layout Plain Layout
1645 <<Fix code chunks in latex layout>>
1648 \begin_layout Plain Layout
1650 <<Fix [[var]] noweb construct>>
1653 \begin_layout Plain Layout
1655 print OUTPUT; # default
1658 \begin_layout Plain Layout
1663 \begin_layout Plain Layout
1665 <<Close INPUT and OUTPUT>>
1673 \begin_layout Standard
1674 Note that in the perl code that is contained in the
1687 \begin_layout Plain Layout
1704 loop above, the perl construct
1718 \begin_layout Plain Layout
1735 is sufficient to restart the loop.
1736 We can use this construct to do some relatively complex parsing of the
1737 reLyX generated file.
1740 \begin_layout Subsection
1741 File input and output for the post-processing
1744 \begin_layout Standard
1758 \begin_layout Plain Layout
1789 \begin_layout Plain Layout
1806 is taken care of by this code:
1809 \begin_layout Standard
1810 \begin_inset Flex Chunk
1813 \begin_layout Plain Layout
1815 \begin_inset Argument 1
1818 \begin_layout Plain Layout
1819 Setup INPUT and OUTPUT for the final output
1827 \begin_layout Plain Layout
1829 &setup_files("$input_file", "$output_file");
1832 \begin_layout Plain Layout
1837 \begin_layout Plain Layout
1839 &setup_files("$relyx_file.lyx", "$output_file");
1842 \begin_layout Plain Layout
1852 \begin_layout Subsection
1853 Making sure the code chunks are in the Chunk layout
1856 \begin_layout Standard
1857 Now, as we outlined before, the final step is transforming the code-chunks
1858 which have been put into a LaTeX layout by LyX into the Chunk layout.
1861 \begin_layout Standard
1862 \begin_inset Flex Chunk
1865 \begin_layout Plain Layout
1867 \begin_inset Argument 1
1870 \begin_layout Plain Layout
1871 Fix code chunks in latex layout
1880 latex latex/) { # Beginning of some latex code
1883 \begin_layout Plain Layout
1885 if (($line = <INPUT>) =~ /^
1887 s*<</) { # code chunk
1890 \begin_layout Plain Layout
1892 <<Transform this chunk into layout chunk>>
1895 \begin_layout Plain Layout
1900 \begin_layout Plain Layout
1904 latex latex line + next line
1907 \begin_layout Plain Layout
1909 print OUTPUT "$_$line";
1912 \begin_layout Plain Layout
1917 \begin_layout Plain Layout
1922 \begin_layout Plain Layout
1932 \begin_layout Standard
1933 When we are sure that we are in a code chunk, we must read in the rest of
1934 the code chunk and output a chunk layout for it:
1937 \begin_layout Standard
1941 \begin_layout Plain Layout
1943 <<Transform this chunk into layout chunk>>=
1946 \begin_layout Plain Layout
1950 \begin_layout Plain Layout
1963 \begin_layout Plain Layout
1965 codeline: while (<INPUT>) {
1968 \begin_layout Plain Layout
1973 \begin_layout Plain Layout
1975 last codeline if /^@
1980 \begin_layout Plain Layout
1985 \begin_layout Plain Layout
1987 print OUTPUT $savedchunk;
1990 \begin_layout Plain Layout
1992 <<Slurp up to the end of the latex layout>>
1995 \begin_layout Plain Layout
2005 \begin_layout Standard
2006 Okay, now we just need to eat the rest of the latex layout.
2007 There should only be a few different types of lines for us to match:
2010 \begin_layout Standard
2011 \begin_inset Flex Chunk
2014 \begin_layout Plain Layout
2016 \begin_inset Argument 1
2019 \begin_layout Plain Layout
2020 Slurp up to the end of the latex layout
2025 slurp: while (<INPUT>) {
2028 \begin_layout Plain Layout
2037 \begin_layout Plain Layout
2046 \begin_layout Plain Layout
2053 \begin_layout Plain Layout
2055 warn "confused by line: $_";
2058 \begin_layout Plain Layout
2068 \begin_layout Subsection
2080 \begin_layout Standard
2085 allows the user to use a special code quoting mechanism in documentation
2087 Fixing this ``[[quoted-code]]''
2091 syntax means putting the ``[[quoted-code]]'' in a LaTeX layout in the LyX
2093 Otherwise, LyX will backslash-quote the brackets, creating ugly output.
2094 The quoted-code is transformed by
2098 when it generates the final LaTeX code.
2101 \begin_layout Standard
2102 \begin_inset Flex Chunk
2105 \begin_layout Plain Layout
2107 \begin_inset Argument 1
2110 \begin_layout Plain Layout
2111 Fix [[var]] noweb construct
2124 ]/) { # special code for [[var]]
2127 \begin_layout Plain Layout
2154 \begin_layout Plain Layout
2159 \begin_layout Plain Layout
2164 \begin_layout Plain Layout
2174 \begin_layout Section
2175 Cleaning up intermediate files
2178 \begin_layout Standard
2179 The cleanup code is very simple:
2182 \begin_layout Standard
2183 \begin_inset Flex Chunk
2186 \begin_layout Plain Layout
2188 \begin_inset Argument 1
2191 \begin_layout Plain Layout
2197 system("rm -f $relyx_file*") unless ($post_only || $pre_only);
2205 \begin_layout Section
2206 User supplied arguments
2209 \begin_layout Standard
2214 script understands two arguments, input-file and output-file.
2215 It is also set up to be used internally by reLyX to pre-process or postprocess
2216 files in the import pipeline.
2219 \begin_layout Standard
2223 \begin_layout Plain Layout
2225 <<Setup variables from user supplied args>>=
2228 \begin_layout Plain Layout
2230 &usage() if ($#ARGV < 1); # zero or one argument
2233 \begin_layout Plain Layout
2235 if ($ARGV[0] eq "-pre") {
2238 \begin_layout Plain Layout
2240 &usage unless ($#ARGV == 2);
2243 \begin_layout Plain Layout
2245 $input_file = $ARGV[1]; $output_file = $ARGV[2]; $pre_only = 1;
2248 \begin_layout Plain Layout
2250 } elsif ($ARGV[0] eq "-post") {
2253 \begin_layout Plain Layout
2255 &usage unless ($#ARGV == 2);
2258 \begin_layout Plain Layout
2260 $input_file = $ARGV[1]; $output_file = $ARGV[2]; $post_only = 1;
2263 \begin_layout Plain Layout
2268 \begin_layout Plain Layout
2270 &usage unless ($#ARGV == 1);
2273 \begin_layout Plain Layout
2275 $input_file = $ARGV[0]; $output_file = $ARGV[1];
2278 \begin_layout Plain Layout
2280 $pre_only = 0; $post_only = 0;
2283 \begin_layout Plain Layout
2288 \begin_layout Plain Layout
2290 @ %def input_file output_file pre_only post_only
2293 \begin_layout Plain Layout
2298 \begin_layout Plain Layout
2303 \begin_layout Plain Layout
2305 print "Usage: noweb2lyx [-pre | -post] input-file output-file
2308 \begin_layout Plain Layout
2312 \begin_layout Plain Layout
2314 If -pre is specified, only pre-processes the input-file for reLyX.
2317 \begin_layout Plain Layout
2319 Similarly, in the case of -post, post-processes reLyX output.
2322 \begin_layout Plain Layout
2324 In case of bugs, Email Kayvan Sylvan <kayvan
2331 \begin_layout Plain Layout
2336 \begin_layout Plain Layout
2341 \begin_layout Plain Layout
2351 \begin_layout Section
2359 \begin_layout Standard
2360 The noweb2lyx script can be tangled from LyX if you set
2366 to call a generic script that always extracts a chunk named
2371 Here is an example of such a script:
2374 \begin_layout LyX-Code
2376 \begin_inset Newline newline
2379 notangle -Rbuild-script $1 | env NOWEB_SOURCE=$1 NOWEB_OUTPUT_DIR=$2 sh
2382 \begin_layout Standard
2383 \begin_inset Flex Chunk
2386 \begin_layout Plain Layout
2388 \begin_inset Argument 1
2391 \begin_layout Plain Layout
2400 \begin_layout Plain Layout
2402 notangle -Rnoweb2lyx.in noweb2lyx.nw > noweb2lyx.in
2405 \begin_layout Plain Layout
2407 sed -e "s=@PERL@=$PREFIX/bin/perl=" noweb2lyx.in > noweb2lyx
2410 \begin_layout Plain Layout
2420 \begin_layout Standard
2421 \begin_inset Newpage newpage
2427 \begin_layout Section*
2431 \begin_layout Standard
2444 \begin_layout Plain Layout
2456 \begin_layout Section*
2460 \begin_layout Standard
2473 \begin_layout Plain Layout