Can we avoid CLI Mod altogether

[sravankorumilli]

With the support for defining plugins through just Yamls we can avoid CLI mod but Compile Assets is the GRPC method that has to be moved out of this to make this happen. Lets analyse on this.

[sravankorumilli]

Can we avoid this by moving the responsibility to executor instead of plugin server. @kushsharma do you have more inputs here?

[kushsharma]

I guess we can, just need to make sure how we can make compile assets logic generic enough to support all of our current use cases. It is part of plugin because logic could vary based on plugins. Last resort could be to extract this as a mod as well.

[sravankorumilli]

it will be part of plugin instead of plugin server will move to plugin executor. That means given start and end dates it is the responsibility of bq2bq plugin to generate various replace queries as that is the only scope of why this was introduced in plugin. Rest of the scenarios compilation happens in optimus itself.

[deryrahman]

As of now, there're 3 approaches where we can move compile assets part:

1. Move to executor. In this case, admin build instance is no longer executing CompileAssets() through bq2bq plugin. The api only return the raw asset (not compiled yet). Bq2bq executor has responsibility to compile this raw assets, so that we need to rewrite it in python code.
2. Move to optimus server. This is the easiest one, admin build instance is executing CompileAssets() as part of the optimus server (no plugin needed). The rest will follow the same.
3. Move to init container. Similar with approach 1. The difference is the compile assets is executed by init container so that we can use golang instead, no need rewrite into python code.

cc: @sravankorumilli @smarch-int

[deryrahman]

I was creating interaction diagram for our existing flow, so that it easier to follow what's going on, and where we can safely move compile assets functionality. cmiiw
https://user-images.githubusercontent.com/3104624/183061075-3890d2fd-0053-4466-85e7-5491a58ab7c7.png

[deryrahman]

Second option is not preferable (part of bq2bq instead, not optimus server)