Fix a few edge-cases which in the lyx2lyx conversion to format 249

(multi-encoding -> utf8); specifically, the language was being
incorrectly identified in certain insets, which of course led to
encoding problems.
This fixes part of bug 3613 (http://bugzilla.lyx.org/show_bug.cgi?id=3613).

git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@19151 a592a061-630c-0410-9148-cb99ea01b6c8

diff --git a/RELEASE-NOTES b/RELEASE-NOTES
index 20a37a1f2eb19a5f26e054ff69724f751347d6cc..428b4fe58adf66c97ce127ceca4e27b4ec011be1 100644 (file)
--- a/RELEASE-NOTES
+++ b/RELEASE-NOTES
@@ -51,6 +51,26 @@ developers mailing list, we do have some possible solutions for this.
 The effects of this will be more pronounced for RTL (Hebrew, Arabic, Farsi) 
 users --- though they affect users of other languages as well.
 
+- Inset encodings and Conversion from earlier LyX versions
+
+One of the main new features in version 1.5.0 is Unicode. As part of the
+transition, lyx2lyx (the scripts used for converting back and forth between
+different versions of the lyx files) converts old .lyx files, which may use 
+a number of different encodings, to UTF-8. This conversion depends on
+correctly identifying the language of the text. There were previously some
+edge-cases (insets embedded in different-language text type scenarios) in 
+which the language was incorrectly identified, which caused some text to 
+appear incorrectly after having upgraded from older versions. This has now been
+fixed. Unfortunately, however, the fix cannot be applied to files which have
+already been converted past format 249.  So if you have already converted 
+your old files (using a development version or release candidate), this fix
+won't help, unless you still have the originals lying around (and haven't 
+yet made too many changes to the newer versions ;) ).
+
+Generally, it is probably wise to keep a backup of the old version of your 
+files, at least until you are sure that the upgrade went smoothly (which it 
+almost always will).
+
 
 Note: There may later be an updated list of known issues online at
        http://wiki.lyx.org/LyX/ReleaseNotes

diff --git a/development/FORMAT b/development/FORMAT
index 1aa2e5b795659165f4cb3df3ed5b1d934720ff33..b39f184f795cc26f17b20f0fa5902045e46f6a8f 100644 (file)
--- a/development/FORMAT
+++ b/development/FORMAT
@@ -1,6 +1,11 @@
 LyX file-format changes
 -----------------------
 
+2007-07-20 Dov Feldstern <dov@lyx.org>
+
+       * format *not* incremented; fixed format 249 conversion, so that it now
+               correctly deals with encodings in footnotes (part of bug 3613)
+
 2007-06-26 Uwe Stöhr <uwestoehr@web.de> and Dov Feldstern <dov@lyx.org>
 
        * format incremented to 276: switching exsting language 'arabic' to 

diff --git a/lib/lyx2lyx/lyx_1_5.py b/lib/lyx2lyx/lyx_1_5.py
index f784f8476899c1ffa2faa5108fce9e371759e4f9..d874ef3101bb27a4b9c574833ecb1aaeecec3b34 100644 (file)
--- a/lib/lyx2lyx/lyx_1_5.py
+++ b/lib/lyx2lyx/lyx_1_5.py
@@ -246,10 +246,13 @@ document.encoding must be set to the old value (format 248) in both cases.
 We do this here and not in LyX.py because it is far easier to do the
 necessary parsing in modern formats than in ancient ones.
 """
+    inset_types = ["Foot", "Note"]
     if document.cjk_encoding != '':
         return
     encoding_stack = [document.encoding]
+    inset_stack = []
     lang_re = re.compile(r"^\\lang\s(\S+)")
+    inset_re = re.compile(r"^\\begin_inset\s(\S+)")
     if document.inputencoding == "auto" or document.inputencoding == "default":
         for i in range(len(document.body)):
             result = lang_re.match(document.body[i])
@@ -264,7 +267,11 @@ necessary parsing in modern formats than in ancient ones.
                     encoding_stack[-1] = lang[language][3]
             elif find_token(document.body, "\\begin_layout", i, i + 1) == i:
                 document.warning("Adding nested encoding %s." % encoding_stack[-1], 3)
-                encoding_stack.append(encoding_stack[-1])
+                if len(inset_stack) > 0 and inset_stack[-1] in inset_types:
+                    from lyx2lyx_lang import lang
+                    encoding_stack.append(lang[document.language][3])
+                else:
+                    encoding_stack.append(encoding_stack[-1])
             elif find_token(document.body, "\\end_layout", i, i + 1) == i:
                 document.warning("Removing nested encoding %s." % encoding_stack[-1], 3)
                 if len(encoding_stack) == 1:
@@ -272,6 +279,15 @@ necessary parsing in modern formats than in ancient ones.
                     document.warning("Malformed LyX document: Unexpected `\\end_layout'.")
                 else:
                     del encoding_stack[-1]
+            elif find_token(document.body, "\\begin_inset", i, i + 1) == i:
+                inset_result = inset_re.match(document.body[i])
+                if inset_result:
+                    inset_type = inset_result.group(1)
+                    inset_stack.append(inset_type)
+                else: 
+                    inset_stack.append("")
+            elif find_token(document.body, "\\end_inset", i, i + 1) == i:
+                del inset_stack[-1]
             if encoding_stack[-1] != document.encoding:
                 if forward:
                     # This line has been incorrectly interpreted as if it was