Background

Currently, Arcalibre has about 27k lines of code inherited from Calibre written in RapydScript-NG (ext *.pyj), a compile-to-JavaScript language intended to be close enough to Python to allow for copying and pasting some code between the two.

RapydScript-NG itself is Kovid Goyal’s fork of RapydScript, and as far as I am able to tell, is used exclusively in Calibre and its forks. The original Calibre build includes a snapshot of the RapydScript-NG compile as an xz-archive, itself minified from the compiler by the bin/web-repl-export script in the RapydScript-NG repo as called by calibre.utils.rapydscript.update_rapydscript.

Calls into RapydScript-NG language code currently use the QtWebEngine set of Qt components backed by the "headless" QPA plugin bundled with Calibre, which adds a significant additional chain of dependencies above and beyond the RapydScript-NG compiler itself. Notably, this chain of dependencies is responsible for some Arcalibre failures and other failures downstream of the original Calibre codebase wherein QtWebEngine fails to initialize a Vulkan context.

Moving Forward

Given that RapydScript-NG is a large additional dependency and is less needed given the state of modern JavaScript (it’s a much nicer language now than it was in 2012, when version 0.1.0 of RapydScript was first released!), if Arcalibre is going to be a maintainable fork of Calibre and not only an archival (see https://forums.rereading.space/post/6), we’ll likely need to rewrite RapydScript-NG code in pure JavaScript, or at most some well-established compile-to-JS language like TypeScript. That in turn raises a question: how do we do that? The contents of src/pyj currently contain a number of interdependent module imports, making it difficult to switch to more conventional JavaScript.

Footnote about TypeScript

While I don’t necessarily advocate for TypeScript, it’s worth noting that that would still reduce the weight of external dependencies, as it can be obtained and executed using npx rather than via a bespoke compiler and a toolchain that includes an implicit Vulkan dependency. Pure JavaScript requires even less in the way of dependencies, and may be better on that basis, to be sure.

Questions

  • Should we remove RapydScript-NG code?
  • What should replace it? Pure JavaScript, TypeScript, or something else?
  • If we do remove RapydScript-NG code, how should we go about doing so?
  • tempestEnglish
    2·
    1 month ago

    Okay that makes sense. I don’t know if i want to weigh in directly on the “should we remove RapydScript” question, as that feels more like a dependent part of the broader “is the goal just an archive or adjusting for maintainability” question being sorted out elsewhere. But if the decision is made that maintainability is a project priority, then I think removing RapydScript and replacing it with something more maintainable is a pretty easy goal to get behind.

    My recommendation would probably be Typescript. It is a well-supported compiler that as you mentioned would already reduce the maintenance burden a lot compared to RapydScript, and I don’t personally think the reduction in dependencies from using regular Javascript would be a good choice given how much there is to replace (at least going off the number of .pyj files in the repo). The type-checking and interface definitions would be helpful for keeping that code relatively approachable, even for folks who may not have as much JS experience.

    As for how, I still need to poke at the Rapydscript-ng compiler a bit to sort out some of the details of how this could be implemented, but it seems like the files in the repo are already organized into isolated modules that we could potentially rewrite in Javascript or Typescript one at a time. I would suggest setting up JS unit tests based on the compiled .pyj versions of each module, so we have a behavioral baseline to test against while doing so (and for testing any additional changes made to this code in the future). I’m not entirely sure how the modules are loaded yet but theoretically according to Rapydscript’s docs it can interop with JS just fine, so we could even merge in the rewritten modules as we go so that this doesn’t become a huge change that needs to all be resolved at once.

    • Cassandra GranadeOPMAEnglish
      2·
      1 month ago

      Yeah, no, a lot of this comes back to the “what should Arcalibre be” question, but I’m increasingly getting to the point where an archival fork doesn’t make as much sense as I’d have hoped. The Calibre codebase is really fragile, and in ways that make it difficult to snapshot without making significant changes. I’ll say more in the other thread shortly.

      In any case, yeah, TypeScript seems like a good path forward. It’s a much lighter-weight and better-maintained dependency (even if it is Microsoft). There’s a lot of JS code in the form of .pyj modules, such that having some compile-time guarantees makes a lot of sense.

      Similarly, I like the idea of replacing modules one at a time. I’ve not gotten the tests down to a point where I can run them, but hopefully the existing test suite should help in porting RS → TS?

      • tempestEnglish
        2·
        1 month ago

        that makes sense — once i finish up some stuff for work i’m going to do a preliminary test of what it would take to start with this (setting up typescript, maybe getting tests working with it, etc) and we can go from there

          • tempestEnglish
            1·
            13 days ago

            I did take a look at that, I don’t think that pytest specifically is going to be that helpful for this as ideally I would like to test these modules after being compiled to Javascript — so then the same tests can be run on each as they’re replaced

            With that in mind I think it might make sense to make a subdirectory inside the tests/ directory for JS tests, and then add a separate Justfile command for running jest or vitest or another Javascript test runner (I haven’t looked into what would be the best for this yet, those are just ones I have used previously)

        • tempestEnglish
          1·
          13 days ago

          Okay, so we expected horrors and I found horrors, getting ES6 module resolution to interop with Rapydscript’s module resolution is going to be . . . for lack of a better word, “fun”.

          The way the Rapydscript compiler works is to handle all module resolution at compile time, and then to produce a single Javascript file that neatly wraps up all of its inter-module resolution internally, and doesn’t even do us the favor of exposing any exports for if that file is imported elsewhere — it entirely assumes that this file will be executed in a top-level page script context, so introspecting the modules at all for testing will be a bit of an issue.

          I think I can solve this by adding a new entrypoint module that uses Rapydscript’s literal inline JS syntax to specifically export the libraries I want to test:

          import initialize
import utils

v'module.exports.utils = utils'
v'module.exports._modules = ρσ_modules'

          and then compiling that with the --bare option gives a module that can at the very least be imported into a non-page context and call the functions in like a test environment.

          This does still present an issue that mocking out other modules for testing is going to be a mess, as they all exist on this ρσ_modules object Rapydscript inserts rather than going through the normal JS module system. This also will be an issue if we start rewriting some of the modules, as we’ll need to change the way things are imported depending on if they’re in JS or Rapydscript

          None of this is insurmountable, I’m going to start working on rewriting one of the smaller utility modules, and then I’ll see if I can get the module loading to work between them. (As well as start verifying whether I can modify these and see those changes in the running application)

    • tempestEnglish
      1·
      1 month ago

      As an addendum: because this is a project that is largely Python, it also could make sense to look into Python-in-wasm tools as an alternative to Javascript/Typescript. I’m not going to make a definite suggestion here as I am not really a Python developer (nor have I used any of these) but depending on the skillsets of folks who end up involved in this, that could be a more maintainable path forward

      An upside of that approach is that it might be possible to run the existing .pyj modules with minimal alterations, although how feasible that is / how much work it is to get the module loading to behave when used this way would depend on the Python runtime used. I’m also a bit worried that (again depending on how the runtime handles loading) this route would maybe have to be done as a much larger all-at-once change, and would have fewer opportunities to identify / test against changes introduced by the runtime’s access to web APIs differing from how Rapydscript wraps them. This would probably be a lot clearer and easier to weigh pros/cons of if someone with more Python experience wanted to try their hand at testing out the feasibility of loading a wasm runtime in the QT web context

      (I should say: my personal feeling is that a Python-wasm runtime is probably a more risky dependency to add than Typescript, given that the ones I’ve encountered seem relatively new, and it’s harder to know that any one we select will be maintained long-term . . . but that might be a worthwhile trade-off if having that portion of code be in Python makes it easier for folks to contribute across the project without learning another language)

      • Cassandra GranadeOPMAEnglish
        2·
        1 month ago

        I’ve looked into Python/WASM as a possible plugin mechanism to allow either non-Python plugins, or decoupling plugins from the venv used by Calibre itself, and it definitely seems feasible? That said, some of the RS code seems to go into web contexts that may or may not yet support WASM, I don’t know how well QtWebEngine plays with WASM at moment, most of what I’ve seen goes (weirdly enough) the other way, embedding Qt in WASM contexts.

        I really like the idea, but I think I fall more towards what you note, that it’s more risky at this point than TypeScript, and a higher bar to contribution.