Background

Currently, Arcalibre has about 27k lines of code inherited from Calibre written in RapydScript-NG (ext *.pyj), a compile-to-JavaScript language intended to be close enough to Python to allow for copying and pasting some code between the two.

RapydScript-NG itself is Kovid Goyal’s fork of RapydScript, and as far as I am able to tell, is used exclusively in Calibre and its forks. The original Calibre build includes a snapshot of the RapydScript-NG compile as an xz-archive, itself minified from the compiler by the bin/web-repl-export script in the RapydScript-NG repo as called by calibre.utils.rapydscript.update_rapydscript.

Calls into RapydScript-NG language code currently use the QtWebEngine set of Qt components backed by the "headless" QPA plugin bundled with Calibre, which adds a significant additional chain of dependencies above and beyond the RapydScript-NG compiler itself. Notably, this chain of dependencies is responsible for some Arcalibre failures and other failures downstream of the original Calibre codebase wherein QtWebEngine fails to initialize a Vulkan context.

Moving Forward

Given that RapydScript-NG is a large additional dependency and is less needed given the state of modern JavaScript (it’s a much nicer language now than it was in 2012, when version 0.1.0 of RapydScript was first released!), if Arcalibre is going to be a maintainable fork of Calibre and not only an archival (see https://forums.rereading.space/post/6), we’ll likely need to rewrite RapydScript-NG code in pure JavaScript, or at most some well-established compile-to-JS language like TypeScript. That in turn raises a question: how do we do that? The contents of src/pyj currently contain a number of interdependent module imports, making it difficult to switch to more conventional JavaScript.

Footnote about TypeScript

While I don’t necessarily advocate for TypeScript, it’s worth noting that that would still reduce the weight of external dependencies, as it can be obtained and executed using npx rather than via a bespoke compiler and a toolchain that includes an implicit Vulkan dependency. Pure JavaScript requires even less in the way of dependencies, and may be better on that basis, to be sure.

Questions

  • Should we remove RapydScript-NG code?
  • What should replace it? Pure JavaScript, TypeScript, or something else?
  • If we do remove RapydScript-NG code, how should we go about doing so?
  • tempestEnglish
    2·
    1 month ago

    that makes sense — once i finish up some stuff for work i’m going to do a preliminary test of what it would take to start with this (setting up typescript, maybe getting tests working with it, etc) and we can go from there

      • tempestEnglish
        1·
        13 days ago

        I did take a look at that, I don’t think that pytest specifically is going to be that helpful for this as ideally I would like to test these modules after being compiled to Javascript — so then the same tests can be run on each as they’re replaced

        With that in mind I think it might make sense to make a subdirectory inside the tests/ directory for JS tests, and then add a separate Justfile command for running jest or vitest or another Javascript test runner (I haven’t looked into what would be the best for this yet, those are just ones I have used previously)

    • tempestEnglish
      1·
      13 days ago

      Okay, so we expected horrors and I found horrors, getting ES6 module resolution to interop with Rapydscript’s module resolution is going to be . . . for lack of a better word, “fun”.

      The way the Rapydscript compiler works is to handle all module resolution at compile time, and then to produce a single Javascript file that neatly wraps up all of its inter-module resolution internally, and doesn’t even do us the favor of exposing any exports for if that file is imported elsewhere — it entirely assumes that this file will be executed in a top-level page script context, so introspecting the modules at all for testing will be a bit of an issue.

      I think I can solve this by adding a new entrypoint module that uses Rapydscript’s literal inline JS syntax to specifically export the libraries I want to test:

      import initialize
import utils

v'module.exports.utils = utils'
v'module.exports._modules = ρσ_modules'

      and then compiling that with the --bare option gives a module that can at the very least be imported into a non-page context and call the functions in like a test environment.

      This does still present an issue that mocking out other modules for testing is going to be a mess, as they all exist on this ρσ_modules object Rapydscript inserts rather than going through the normal JS module system. This also will be an issue if we start rewriting some of the modules, as we’ll need to change the way things are imported depending on if they’re in JS or Rapydscript

      None of this is insurmountable, I’m going to start working on rewriting one of the smaller utility modules, and then I’ll see if I can get the module loading to work between them. (As well as start verifying whether I can modify these and see those changes in the running application)