Contributing to Netbeans

[mytskine]

Hi.

I've implemented the feature requested in issue #5010, syntax coloring of PHP heredoc blocks named "...SQL...". It would be a great help in a PHP project I work for, which has many long SQL queries (dozens of lines for each). The branch is https://github.com/mytskine/netbeans/tree/5010_php_heredoc_sql

But since I'm not a Java developer, my branch is not ready for production. It's a hack, a personal toy written with plain vim and barely tested. Yet I have a few questions after struggling with Netbeans' code and build process.

• Is there an up to date documentation for contributing and developping Netbeans? The wiki parts I read were obsolete.
• Is there a hierarchy convention or a naming convention? E.g. I had to extract the SQL lexer from the SQL editor module into its own module and hesitated on the right place.
• In php/php.editor/.../lexer/, the files PHPTop*.java looked promising but are apparently unused. Is it dead code or did I miss something?
• Would a low quality pull request be useful? I'm afraid it will required more work from the maintainers to fix it than to write it themselves.

Many thanks for maintaining Netbeans, especially to the PHP maintainer!

[matthiasblaesing]

For small fixes, test them, ensure your commit has a good description and valid author information, just open a PR. For larger contributions please make sure an ICLA is on file at the Apache foundation (https://www.apache.org/licenses/contributor-agreements.html).

It's a hack, a personal toy written with plain vim and barely tested. Yet I have a few questions after struggling with Netbeans' code and build process.

I don't understand why you don't use NetBeans to work on the modules. The basic idea:

1. Checkout NB from github
2. Build it once following the hints here: https://github.com/apache/netbeans/blob/master/README.md
3. Create a branch for your modifications
4. Open the module(s) you want to modify in NetBeans
5. If the changes only affect a single module, you can directly test the module from the IDE (Contextmenu -> run/debug)
6. If multiple modules are changed, run a "Clean and build" on all of them and then you can do the same as in (5)

I had to extract the SQL lexer from the SQL editor module into its own module and hesitated on the right place.

I doubt, that this is the right approach and it will not scale. I admit, that I have not yet looked to deeply into this, but NetBeans has the infrastructure to handle embeddings. I.e. situations where several languages are interleved/nested into each other.

Examples are:

• JS in HTML
• HTML in PHP
• CSS in HTML
• CSS in HTML in PHP
• JS in HTML in PHP
• Java in (X)HTML (aka JSP)

For this there is an EmbeddingProvider:

• https://bits.netbeans.org/23/javadoc/org-netbeans-modules-parsing-api/org/netbeans/modules/parsing/spi/EmbeddingProvider.html
• https://bits.netbeans.org/23/javadoc/org-netbeans-modules-parsing-api/org/netbeans/modules/parsing/spi/ParserBasedEmbeddingProvider.html

Examples are:

• https://github.com/apache/netbeans/blob/master/php/php.editor/src/org/netbeans/modules/php/editor/PhpEmbeddingProvider.java
• https://github.com/apache/netbeans/blob/master/ide/html.editor/src/org/netbeans/modules/html/editor/embedding/CssEmbeddingProvider.java
• https://github.com/apache/netbeans/blob/master/ide/html.editor/src/org/netbeans/modules/html/editor/embedding/JsEmbeddingProvider.java

[mytskine]

Thank you for this detailed answer. I'll investigate if reaching for an EmbeddingProvider is possible from within the php-editor module. That's not the way its syntax coloring deals with HTML, which is what inspired my patch.

I don't understand why you don't use NetBeans to work on the modules.

Two reasons:

1. I did as you suggested, "following the hints" in Netbeans' readme. So I used ant which is the only tool documented there.
2. My Netbeans instance is configured for working with PHP, and I don't want to alter it. I'll try to switch to an alternative config directory and enable the java modules.

[matthiasblaesing]

Thank you for this detailed answer. I'll investigate if reaching for an EmbeddingProvider is possible from within the php-editor module. That's not the way its syntax coloring deals with HTML, which is what inspired my patch.

Where do you see the HTML coloring in PHP? The PhpEmbeddingProvider would be the first place where I'd look.

I don't understand why you don't use NetBeans to work on the modules.

Two reasons:

1. I did as you suggested, "following the hints" in Netbeans' readme. So I used `ant` which is the only tool documented there.

Ok, I think everybody working on that document came from the direction of the IDE and then it is a no brainer, that a Java IDE can work on its own modules. Want to suggested changes?

2. My Netbeans instance is configured for working with PHP, and I don't want to alter it. I'll try to switch to an alternative config directory and enable the java modules.

I routinely run with all clusters enabled (+ rust cluster) and am mostly happy with the IDE (yeah, this will come back to haunt me).

[mytskine]

Where do you see the HTML coloring in PHP? The PhpEmbeddingProvider would be the first place where I'd look.

I started with the class PHP5ColoringLexer. In some cases, it creates a LanguageEmbedding instance for HTML. I have not checked if this is related to EmbeddingProvider.

As far as I understand, PhpEmbeddingProvider provides a PHP lexer. I was interested in SQL inside PHP, so the other way. There is no SqlEmbeddingProvider class, but it would probably make sense to create it. I suppose, it would still require to factorize the SQL lexer out of the SQL editor, but that's just a hich without reding the code.

[matthiasblaesing]

I had a look at your code. For someone that "hacked" it together it looks pretty sane and I should have taken a look directly ;-)

I try to summarize. You want to support that this gets SQL highlighting:

<?php 
$a = <<<SQL
    SELECT * FROM dummy
SQL;

You can drop the "factor sql.lexer out" part of the changes. I removed your entire first commit and then applied these changes:

diff --git a/php/php.editor/nbproject/project.xml b/php/php.editor/nbproject/project.xml
index 70aeb1c18000..f1dd294cdc38 100644
--- a/php/php.editor/nbproject/project.xml
+++ b/php/php.editor/nbproject/project.xml
@@ -360,15 +360,6 @@
                         <specification-version>1.12</specification-version>
                     </run-dependency>
                 </dependency>
-                <dependency>
-                    <code-name-base>org.netbeans.modules.sql.lexer</code-name-base>
-                    <build-prerequisite/>
-                    <compile-dependency/>
-                    <run-dependency>
-                        <release-version>1</release-version>
-                        <specification-version>1.34</specification-version>
-                    </run-dependency>
-                </dependency>
                 <dependency>
                     <code-name-base>org.netbeans.modules.web.common</code-name-base>
                     <build-prerequisite/>
diff --git a/php/php.editor/src/org/netbeans/modules/php/editor/lexer/PHPTokenId.java b/php/php.editor/src/org/netbeans/modules/php/editor/lexer/PHPTokenId.java
index 32f04f924899..38c024132bbc 100644
--- a/php/php.editor/src/org/netbeans/modules/php/editor/lexer/PHPTokenId.java
+++ b/php/php.editor/src/org/netbeans/modules/php/editor/lexer/PHPTokenId.java
@@ -28,7 +28,6 @@ import org.netbeans.api.lexer.Language;
 import org.netbeans.api.lexer.LanguagePath;
 import org.netbeans.api.lexer.Token;
 import org.netbeans.api.lexer.TokenId;
-import org.netbeans.api.sql.lexer.SQLTokenId;
 import org.netbeans.modules.php.api.util.FileUtils;
 import org.netbeans.spi.lexer.LanguageEmbedding;
 import org.netbeans.spi.lexer.LanguageHierarchy;
@@ -235,7 +234,10 @@ public enum PHPTokenId implements TokenId {
             if (id == T_INLINE_HTML) {
                 return LanguageEmbedding.create(HTMLTokenId.language(), 0, 0, true);
             } else if (id == T_EMBEDDED_SQL) {
-                return LanguageEmbedding.create(SQLTokenId.language(), 0, 0, false);
+                Language<?> sqlLanguage = Language.find("text/x-sql");
+                if (sqlLanguage != null) {
+                    return LanguageEmbedding.create(sqlLanguage, 0, 0, false);
+                }
             } else if (id == PHPDOC_COMMENT) {
                 return LanguageEmbedding.create(PHPDocCommentTokenId.language(), 0, 0);
             }

The idea here is that you only need the language instance. Of course you can directly get it from the implementation (done for HTML and PHPDoc here), but that is not necessary. The runtime gives you access to the language by mimetype. You'll need db.sql.editor at runtime for the feature, but if the module is missing, this will degrade gracefully.

This is the result:

I think other embeddings "shade" the embedded sequence (if I remember correctly JS in HTML) with a background color, this might be a good idea here also.

For getting completion support there is PHPSQLStatement, which might need adjusting too.

It might be a good idea to get @junichi11 and @tmysik into the boat early as @junichi11 is the main PHP maintainer and @tmysik the main reviewer, so from my POV this looks like a good feature, but these two should have the last word.

[tmysik]

@matthiasblaesing Thanks a lot for your great analysis and letting @junichi11 and me know about this interesting discussion.

@mytskine I like this feature and I totally agree with Matthias - if you are able to create a PR, I will be more than happy to look in it. I believe that the same applies for Junichi.

Thank you!