Generate EPUB3 documents

[ccheraa]

This adds an option to generate EPUB3 documents using https://github.com/eeditiones/tei-publisher-app

These changes use tei-publisher-lib to generate epub documents, while the current epub generation method uses the deprecated tei-simple, since both packages use the same namespaces, which causes conflicts, we are now using twi-publisher-lib only.

But this package relies on hsg-shell which uses tei-simple so in order to test these changes you ned to make a few adjustments to the document /db/apps/hsg-shell/resources/odd/compiled/frus-web.xql:

First, change:

import module namespace css="http://www.tei-c.org/tei-simple/xquery/css" at "xmldb:exist://embedded-eXist-server/db/apps/tei-simple/content/css.xql";
import module namespace html="http://www.tei-c.org/tei-simple/xquery/functions" at "xmldb:exist://embedded-eXist-server/db/apps/tei-simple/content/html-functions.xql";

to:

import module namespace css="http://www.tei-c.org/tei-simple/xquery/css";
import module namespace html="http://www.tei-c.org/tei-simple/xquery/functions";

Second, add these extra arguments to the following calls:
html:listItem: ()
html:list: ()
html:cit: ()
html:alternate: @when
html:link: (), map {}

i.e.

- html:listItem($config, ., ("tei-item"), .)
+ html:listItem($config, ., ("tei-item"), ., ())

- html:alternate($config, ., ("tei-choice1"), ., corr[1], sic[1])
+ html:alternate($config, ., ("tei-choice1"), ., corr[1], sic[1], @when)

This open source contribution to the release project was commissioned by the Office of the Historian, U.S. Department of State, https://history.state.gov/.

[joewiz]

@ccheraa To ensure I don't overlook any of the required changes that you described, could you please open a PR in hsg-shell (against master) with the necessary changes?

[joewiz]

@ccheraa Alternatively, it might be faster to upload the /db/apps/hsg-shell/resources/odd/compiled/frus-web.xql with your changes as an attachment to this thread. We just will need to remember that hsg-shell will need a PR with these changes, unless they're already present.

[ccheraa]

@joewiz here is the updated frus-web.xql:
https://gist.github.com/ccheraa/24241af17e1f252b32d9a53b62f66909

[joewiz]

@ccheraa Thank you! When I installed the supplied frus-web.xql into /db/apps/hsg-shell/resources/odd/compiled/frus-web.xql and called http://localhost:8080/exist/apps/release/ebook-batch.xq I got an error:

<exception>
    <path>/db/apps/release/ebook-batch.xq</path>
    <message>err:XPST0003 error found while loading module epub: Error while loading module
        modules/epub.xql: error found while loading module render: Error while loading module
        tei-render.xql: error found while loading module frusx: Error while loading module frus.xql:
        error found while loading module hsg-config: Error while loading module
        /db/apps/hsg-shell/modules/config.xqm: error found while loading module pm-frus: Error while
        loading module ../resources/odd/compiled/frus-web-module.xql: error found while loading
        module m: Error while loading module /db/apps/hsg-shell/resources/odd/compiled/frus-web.xql:
        Syntax error within user defined function model:apply: unexpected token: when [at line 117,
        column 90]</message>
</exception>

Looking at line 117:90 I saw a typeswitch expression for a TEI element that we don't support in frus.odd (tei:choice), so I tried deleting it, and this cleared the error.

But when I tried generating an EPUB3 for a volume, no output file was saved to the destination folder. The request I issued was: http://localhost:8080/exist/apps/release/ebook-batch.xq?volumes=frus1969-76v18&format=epub3. I found this error in monex's console log:

Error while generating epub3 version of vol frus1969-76v18: java:org.exist.xquery.XPathException module: /db/apps/release/modules/epub3.xql(299,38) exerr:ERROR The actual return type does not match the sequence type declared in the function's signature: frus:document-number(element(tei:div)) xs:string. Expected cardinality: exactly one, got 0. [at line 299, column 38, source: /db/apps/release/modules/epub3.xql] In function: frus:document-number(element(tei:div)) [651:59:/db/apps/release/modules/frus.xql] local:update-text-node(item(), item(), item()) [454:5:/db/apps/release/modules/epub3.xql] local:update-text-node(item(), item()) [713:36:/db/apps/release/modules/epub3.xql] local:update-text-node(item(), item(), item()) [454:5:/db/apps/release/modules/epub3.xql] local:update-text-node(item(), item()) [447:41:/db/apps/release/modules/epub3.xql] local:update-text(item(), item()) [428:94:/db/apps/release/modules/epub3.xql] local:update-document(item(), item()) [56:21:/db/apps/release/modules/epub3.xql] local:generate-epub(item(), item(), item(), item(), item(), item(), item(), item(), item(), item()) [881:26:/db/apps/release/modules/epub3.xql] epub3:save-frus-epub-to-disk(xs:string, xs:string*, xs:string) [99:44:/db/apps/release/modules/epub3.xql]. The actual return type does not match the sequence type declared in the function's signature: frus:document-number(element(tei:div)) xs:string. Expected cardinality: exactly one, got 0.

Looking at release/modules/epub3.xql I found a call to the frus:document-number function on line 651 - here:

release/modules/epub3.xql

Line 651 in e97bc38

let $docnumber := frus:document-number($document)

. It appears this function is called on a chapter div, when it was intended for document divs.

(Looking at the return clause here, I notice some mixing of namespace prefixes: some elements have the tei namespace prefix and some lack them.)

Lastly, are some or all of the TEI Publisher PRs required for the custom frus-web.xql you supplied to be generated automatically when the ODD is processed? I think the full list of these PRs is as follows:

• [fix] notes in document title should not appear in navigation entry eeditiones/tei-publisher-app#106
• added option to choose navigation root eeditiones/tei-publisher-app#107
• added option to skip title page eeditiones/tei-publisher-app#108
• added an option to choose images collection eeditiones/tei-publisher-app#109
• added an option to choose cover image eeditiones/tei-publisher-app#110

[joewiz]

p.s. I just realized that the first error in my post above may have been caused by my being on the https://github.com/evolvedbinary/exist/commits/feature/tumbling-sliding-window branch of eXist (which exhibits a parser bug when referring to an attribute with the name when; I have just reported this issue to Adam). I believe that switching back to the develop branch (or an earlier stable release) should cause that problem to go away. But I will need to investigate further.

[joewiz]

After switching to eXist develop, the first error I had reported above ("unexpected token: when") went away. The remaining aspects of my response are all still valid though.

[ccheraa]

@joewiz It should work now 👍

[joewiz]

I generated EPUB 2 (original) and EPUB 3 (this PR) versions of frus1969-76v18:

• EPUB 2: frus1969-76v18.epub.zip
• EPUB 3: frus1969-76v18.3.epub.zip

(Pardon the extra .zip, but without this GitHub wouldn't permit the upload.)

After a brief side-by-side review of this volume, I noticed several issues with the resulting EPUB 3. Here, screenshots are taken with Apple Books, but I would guess the same issues are present in any ebook reader app:

1. The cover image is still missing.

   

2. The resources are all renamed to random filenames instead of the original naming scheme, which used the div @xml:id; the random names make looking up resources in the archive for troubleshooting purposes very difficult.

   

3. The signature at the end of the preface is marked by a missing line break and a missing date:

   

4. The styling of person names in the List of Persons is styled with bold + italics instead of just bold (the source TEI is <hi rend="strong">).

   

5. The EPUB viewer's table of contents interface shows (1) a table of contents entry div[@xml:id eq "toc"] that should be suppressed and (2) extra information in document titles, such as [1] and the footnote contents.

   

6. The generated table of contents for comp1 contains extra information that makes chapters look like they are documents (the links are :

   

7. The heading of Tab A in document 1 is styled with bold + italics instead of just bold (the source TEI is <hi rend="strong">):

   

8. The right-aligned signature is styled with bold + italics instead of just bold (the source TEI is <hi rend="strong">):

   

I am guessing some of these might be issues introduced by the move to TEI Publisher rather than the specific addition of EPUB 3 footnote capabilities. But they are serious flaws/regressions, and in the current state we could not distribute these EPUB 3s - which is a shame, since the pop-up footnotes work so nicely.

@racheltommasino Perhaps we should call this task complete and defer ironing out the unforeseen TEI Publisher EPUB 3 issues in a future task?

[joewiz]

@ccheraa One more thing. What is the plan for getting your custom modifications to frus-web.xql into hsg-shell (resources/odd/compiled/frus-web.xql)? That's a compiled file (from resources/odd/source/frus.odd), not a hand-edited one.