Way forward

[donalffons]

WebIDL / v0.x

This was the first iteration of OpenCascade.js. The build system was mainly based on AmmoJS's build system and it was using the WebIDL syntax to generate bindings between C++ and JavaScript.

Auto-Generated Embind / v1.x

The approach of the v0.x version had some significant drawbacks, mainly:

1. The limitations of the WebIDL syntax make it impossible to expose certain overloaded functions, methods and constructors. Since OpenCascade heavily uses overloads throughout the library, this means that some major parts of OpenCascade would not be accessible from JavaScript.
2. Opencascade is a large library with more than 1M source lines of code. Having to write out all bindings manually was not only a lot of work, it also meant that it would be extremely hard to maintain OpenCascade.js and keep it in sync with updates to the parent project. Since OpenCascade.js v0.x (using OpenCascade v7.4.0), OpenCascade received 4 updates with the current version being v7.5.0.
3. Esspecially for larger projects, Typescript definitions are extremely important for the developer experience. Although there are tools that make it very easy to generate typescript bindings from WebIDL, a lot of helpful information would be lost in this process. Most importantly, comments (i.e. on classes and functions) would not be available in OpenCascade.js

Because of the point 1, I decided to use Embind instead of WebIDL, which resolved this issue. This also introduced the (not so pretty) syntax of having to use _1, _2 etc to select the respective overload. But I think completeness and correctness beats aesthetics in this case.

Because of points 2 and 3, I developed a system for automatically generating the bindings. Using libclang (which are clang bindings for Python), the C++ code is parsed into an AST. This AST can then be traversed and several rules are used to generate the Embind bindings and typescript definitions. This worked very well to the point, that currently no manual bindings are required.

These changes meant that now a large portion of OpenCascade was available from JavaScript: Probably more than 70% of OpenCascade's API (although there are certainly some bindings, which won't work, e.g. references to built-in types etc.).

Modules / v2.0.x (unreleased)

The approach of the v1.x version braught some improvements, but also more drawbacks, mainly:

1. The library is now very large and has been growing since v1.x. This is of course not great for developing small applications, e.g. when all this data must be shipped via the network.
2. The large amount of WASM code must be compiled to the client's target architecture. The more WASM code, the longer this compilation process takes. This also increases memory consumption.
3. The large amount of bindings come with an additional cost, as Emscripten requires significant efforts during WASM instantiation per binding.
4. In most browsers, there is an upper limit of 100.000 to the number of exports a WASM file can have. OpenCascade.js manages to break this limit.

To find a solution to these problems, I looked into making modularized builds of the library, as opposed to the monolithic approach of v1.x. With v2.x, a user of the library can now choose which OpenCascade modules to use when initializing the library.

Using modules improves some of the abovementioned issues. However, file sizes and instantiation times remain quite long. Therefore, I think to achieve an "optimal" result, it will be necessary to build custom versions of the library per each specific use case.

See here for some preliminary documentation.

Custom Builds / v2.1.x (unreleased)

For developing and prototyping using OpenCascade.js, I think the v2.0.x approach is a good balance between ease of use (i.e. being able to conveniently access any part of the OpenCascade API without much hassle) and speed (i.e. bundle size, startup time and memory consumption). However, the resulting builds will still not be optimal for most use cases: Most of the time, the modules will contain too much code and too many bindings, thereby "wasting" bandwitdth and resources.

To fix this issue, there must be a way to create custom builds. For those custom builds, the following information has to be provided:

1. Which parts of the OpenCascade library should be included in the WASM files
2. Which bindings should be generated (to allow access from JavaScript)
3. Which Emscripten compiler flags should be used (Debug vs Release? Browser vs NodeJS vs Worker? Memory settings?)
4. (Probably a few more settings, e.g. wether to generate typescript definitions etc.)

For this purpose, there will be an option to generate custom builds to produce minimal / optimal builds for any specific use case. Custom builds are defined via YAML files, which contain all required information for a build. The the pre-built modules of v2.0.x are using the exact same representation as these YAML files.

To create a custom build, it should not be necessary to fork this repository. Instead, I am publishing a Docker Image, which can be pulled and then run against a YAML file to create the desired build, similarly to how Emscripten's Docker Image can be used to compile C++ projects on the Host machine.

See here for some preliminary documentation.

Full Typescript Support / v2.2.x? (unreleased)

Typescript support is currently in a beta stage. Many type definitions are missing (instead, the any type is used). The foundations are there, but this topic needs more attention.

Improved Embind Interfaces / v2.3.x? (unreleased)

Currently, the bindings lack support for these two important use-cases

1. There is no support for references (and pointers) to built-in types as function arguments.
2. There is currently no support for default function arguments.

Both issues can probably resolved in a non-breaking way.

Issue 1 can be resolved with clever use of Emscripten's val class (and probably some C++ lambda functions to make the link).

/* C++ */
#include <emscripten/bind.h>

using namespace emscripten;

void increment(int& i) {
    i++;
}

EMSCRIPTEN_BINDINGS(my_module) {
    function(
        "increment",
        (void (*)(emscripten::val)) []( emscripten::val c ) {
            if(c.typeOf().as<std::string>() == "number") {
                auto i = c.as<int>();
                increment(i);
            } else if(c.typeOf().as<std::string>() == "object" && c["current"].typeOf().as<std::string>() == "number") {
                auto i = c["current"].as<int>();
                increment(i);
                c.set("current", i);
            } else {
                throw("unsupported type");
            }
        }
    );
}

/* JS */
const intRef = { current: 41 };
o.increment(123);
o.increment(intRef);
console.log("after", intRef.current);

Issue 2 can also be resolved with Embind. Embind supports generating overloads by number of arguments.

/* C++ */
#include <emscripten/bind.h>

#include <iostream>

using namespace emscripten;

void somefunction(bool test = false) {
    std::cout << "somefunction: " << test << std::endl;
}

EMSCRIPTEN_BINDINGS(my_module) {
    function(
        "somefunction",
        (void (*)()) []() {
            somefunction();
        }
    );
    function(
        "somefunction",
        (void (*)(bool)) [](bool test) {
            somefunction(test);
        }
    );
}

/* JS */
o.somefunction();
o.somefunction(true);

This will increase the complexity in the code-generation code.

EDIT: (Memo to myself) Emscripten's optional_override could help to achieve these modified signatures emscripten-core/emscripten#4475

[donalffons]

@zalo, @Irev-Dev, @bitbybit-dev, this might be of interest to you. If you (or anyone else) have feedback or questions, please share them! 🙂.

[bitbybit-dev]

Hey guys. @donalffons thank you for this information and for taking this project seriously. I understand that there are many challenges to tackle when dealing with such a huge kernel as OCCT. Breaking through standard wasm export limits says a lot. Automation is indeed the only answer here and it's great to see you understand all the intricacies of such migration very well.

I had to read some things a few times and will not pretend to understand it all, but I thought I'll share my thoughts anyway.

v1.x

Having used the library I share your concern about the file size and loading times, but even as it is now in v1.x I think it's a price worth paying for having the super rich feature set. In my prod build gzipped file weighs around 13mb and is cached on the client's machine, so network traffic is expensive, but paid once. Deflating/compiling/loading of course costs time, but it's still 'ok-ish'. The fact that loading of opencascade.js is asynchronous helps a lot, users get a nice indication that things are being prepared.

By the way, my poor iPhone 6s is capable of loading it in Safari and running simple scripts. For larger projects browser kills the app and restarts the webpage as memory footprint is too big to handle, but at this stage it might very well be due to the lack of memory management on my side.

Having to deal with _1 _2 in overloads is a bit annoying, but I've seen worse things in my life... :)  Also what's nice is that they match the documentation of OCCT. The same could be said about default function arguments as they could be seen in OCCT docs and provided by the end user in some generic way.

v2.x

I think you went through all the abstraction levels possible to allow users of opencascade.js choose the best way for their apps. I mean, you kept the big bang approach to load pre-built modules, this might be fine for the lazy folk - like me ;) when starting out and experimenting. I guess this will still be faster than the v1.x approach. 
And you allow us to build the library with exactly the things people need through docker, which I think is great. I will need to experiment with that to see what works best. 
I must admit that at this stage pre-built modules will help enormously as for the custom build I will have to learn a bit more about OCCT itself. And also I guess there's quite some room between these two approaches. Not sure if it would make sense but there could be domain specific pre-built collections of libraries. On the other hand at exactly that point it's probably best to just go and do custom build :)

TypeScript

I might be biased on that one as I love TypeScript, but in this case I think OCCT & TypeScript match is literally made in heaven. I've seen @zalo make some moves toward that, but I didn't dig deep enough to learn more about that. Having pre generated type definitions will be a huge thing and I wouldn't even care if they are incomplete. If some parts of the api will be marked as 'any', so be it. Currently I still have to have OCCT docs open constantly to get things going. 

Just a wild thought, if TypeScript definitions would have comments with 'link' attributes to OCCT docs it would be amazing. Most editors that support TypeScript will interpret that correctly. But yeah, reverse engineering doc algorithms to get a good link might not be fun...

v2.3.x

I hope I understand the increment example, passing those objects as references could work out I guess. Not sure how widespread such methods are in OCCT and how many arguments or what types they usually have. Do you need to change function signatures for standard OCCT function calls then, so that all such arguments would be provided with a single object like in the increment example? Or would there need to be a separate object for each argument?

When performance is not a big concern, if possible, I would even consider generating immutable alternatives to those void methods returning answers as properties in the simple object. Increment method contains the state and I suppose this is very important when dealing with functions doing difficult calculations, but for simpler computations, like in the case of GetFaceUVBounds following the increment example I thought maybe this could be possible:

instead of this:

const bounds = { Umin: 0, Umax: 0, Vmin: 0, Vmax: 0 }
ShapeAnalysis.GetFaceUVBounds(face, bounds);
console.log(bounds.Umin) // overwritten value

Immutable variant could be:

const bounds =  ShapeAnalysis.GetFaceUVBounds_Immutable(face);
console.log(bounds.Umin); // new object new value

I'm not familiar with reflection in C++ or Emscripten val, so forgive me if I missed some problems with this approach. 

P.S

This text definitely turned out to be longer than I expected, but it's super nice to have this discussion. It also helped to learn the latest things which will come in v2.x. I'll give v2. beta a try and will share my experiences later.

[donalffons]

By the way, my poor iPhone 6s is capable of loading it in Safari and running simple scripts. For larger projects browser kills the app and restarts the webpage as memory footprint is too big to handle, but at this stage it might very well be due to the lack of memory management on my side.

I think this kind of problem would improve when building a custom version of the library. I made some tests with an older tablet which was not able to run OpenCascade.js with the "default" builds, but was when using small custom builds. Memory management will definitely also help.

Not sure if it would make sense but there could be domain specific pre-built collections of libraries.

Yeah, having some kind of small standard library would be super useful. However, I'm also not yet experienced enough in OpenCascade to know what should be included and what shouldn't.

if TypeScript definitions would have comments with 'link' attributes to OCCT docs it would be amazing.

Yeah, that sounds like a good idea. OpenCascade is using Doxygen to generate their docs. At least generating URLs to the class level should be straightforward: BRep_PointOnSurface => https://dev.opencascade.org/doc/refman/html/class_b_rep___point_on_surface.html (i.e. X => _x, _ => __). Links to class member seem more difficult.

Not sure how widespread such methods are in OCCT and how many arguments or what types they usually have. Do you need to change function signatures for standard OCCT function calls then, so that all such arguments would be provided with a single object like in the increment example? Or would there need to be a separate object for each argument?

With the approach I shared, it would not be neccesary to change the OCCT function signatures (I think). I was thinking about a separate object for each argument, similar how ReactJS implements references with their useRef hook. A reference to a built-in type would then always need to look like this (number would map to float, double, int, ... in C++):

type refType = {
  current: number | boolean;
};

With that pattern, the code would look like this:

const Umin = { current: 0 };
const Umax = { current: 0 };
const Vmin = { current: 0 };
const Vmax = { current: 0 };
ShapeAnalysis.GetFaceUVBounds(face, Umin, Umax, Vmin, Vmax);

...maybe there would be an opportunity for a helper function, so this could be simplified.... (Would need to make sure that this also works with Typescript):

const [Umin, Umax, Vmin, Vmax] = makeRefs([0, 0, 0, 0]);
ShapeAnalysis.GetFaceUVBounds(face, Umin, Umax, Vmin, Vmax);

if possible, I would even consider generating immutable alternatives to those void methods returning answers as properties in the simple object [...]

Yeah, I agree this would be much nicer code. But there is no guarantee that these reference type are return values only . From a C++ standpoint, they could be both input and outputs at the same time. So, I think your suggestion would require manual patches (to make sure that the C++ functions don't read the value of those references). I think there are a lot of functions using this pattern in OpenCascade.

[bitbybit-dev]

Nice, thanks for the detailed answers, didn't know about refType approach, if that's the most generic and has a chance of working on most of the methods then it's the best solution. Users can have their own ways of abstracting these things away if they don't like it. I imagine there's enough on your plate already and because of OCCT scale keep focusing on what you find important to open up API surface. All those nice to have things will come later... ;)

[Irev-Dev]

@donalffons I mostly want to express appreciation. Your approach seems very well thought out and a good compromise considering.

I think yml config plus docker image is a great solution. One low-priority concern I have here is the build story might be less than ideal for apps using services of the like vercel or netlify as I doubt they have the ability to run a docker image as part of the app build (I haven't checked the docs so I might be wrong). Not a big concern because the benefits of having this customisation available out weights this downside and the built files can always committed to version control or hosted separately on a CDN.

[donalffons]

One low-priority concern I have here is the build story might be less than ideal for apps using services of the like vercel or netlify as I doubt they have the ability to run a docker image as part of the app build (I haven't checked the docs so I might be wrong).

Yeah, I was pretty disappointed that the approach of using modules really didn't yield good enough results in terms of file size and loading times. It actually seems to be worse than having one big standalone module in many cases. I think I need to investigate this more...

Having people create custom builds is certainly not ideal, but I don't have any better ideas at the moment.

If you use GH actions to build your app (you could still use vercel etc for hosting, just don't have them build the app), you should be able to integrate the custom build process using docker into your pipeline. But yes, this does introduce additional complexity.

Mabye if Emscripten would behave a bit differently, when resolving imports from other modules, it might improve the situation of loading times: Instead of resolving all external imports when a library is loaded, resolve them when WASM functions are called. But I really don't know enough about Emscripten to be confident if this would really be possible and it might also have a big impact on runtime performance.

[Irev-Dev]

@donalffons Yup makes sense.

[Irev-Dev]

@donalffons Hope you don't mind. I linked to this discussion from the emscripten discord, just in case there are any Gurus that wanted to weigh in on this.

[donalffons]

I started this issue over at the Emscripten repository with some benchmarks and a request for feedback.