New directory for third party tools of interest for GrimoireLab #230
Conversation
For now, it inclues Dockerfiles and other related files to produce FOSSology Debian packages, via a Docker container, and a Docker container like grimoirelab/full with nomos also installed. Signed-off-by: Jesus M. Gonzalez-Barahona <[email protected]>
jgbarah
force-pushed
the
fosology-docker
branch
from
4fcabfa to
e03e890
Compare
There was a problem hiding this comment.
I have tested the PR and suggested some changes. After the changes, it was possible to produce colic and cocom data. Please find below the configuration files, the docker command and some screenshots of the result.
projects.json
{
"grimoirelab": {
"meta": {
"title": "GrimoireLab"
},
"git": [
"https://github.com/chaoss/grimoirelab-toolkit"
],
"cocom": [
"https://github.com/chaoss/grimoirelab-toolkit"
],
"colic": [
"https://github.com/chaoss/grimoirelab-toolkit"
],
"github": [
"https://github.com/chaoss/grimoirelab-toolkit"
]
}
}
credentials.cfg
[github]
api-token = ccf50...
dashboard.cfg
[general]
# Update incrementally, forever
update = true
# Don't start a new update earlier than (since last update, seconds)
min_update_delay = 300
# Produce debugging data for the logs
debug = true
[es_enrichment]
# Refresh identities and projects for all items after enrichment
autorefresh = true
[sortinghat]
# Run affilation
affiliate = True
# How to match to unify
matching = [email]
# How long to sleep before running again, for identities tasks
sleep_for = 100
[panels]
# Dashboard: default time frame
kibiter_time_from = "now-1y"
# Dashboard: default index pattern
kibiter_default_index = "git"
# GitHub repos panels
code-complexity = true
code-license = true
[phases]
collection = true
identities = true
enrichment = true
panels = true
[git]
# Names for raw and enriched indexes
raw_index = git_grimoirelab-raw
enriched_index = git_grimoirelab
studies = [enrich_demography:git, enrich_areas_of_code:git, enrich_onion:git]
[github]
# Names for raw and enriched indexes
raw_index = github_grimoirelab-raw
enriched_index = github_grimoirelab
# Sleep it GitHub API rate is exhausted, waited until it is recovered
sleep-for-rate = true
[cocom]
raw_index = cocom_chaoss
enriched_index = cocom_chaoss_enrich
category = code_complexity_lizard_file
studies = [enrich_cocom_analysis]
branches = master
git-path = /tmp/git-cocom
worktree-path = /tmp/cocom/
[enrich_cocom_analysis]
out_index = cocom_chaoss_study
interval_months = [3]
[colic]
raw_index = colic_chaoss
enriched_index = colic_chaoss_enrich
category = code_license_nomos
studies = [enrich_colic_analysis]
exec-path = /usr/share/fossology/nomos/agent/nomossa
branches = master
git-path = /tmp/git-colic
worktree-path = /tmp/colic
[enrich_colic_analysis]
out_index = colic_chaoss_study
interval_months = [6]
[enrich_demography:git]
#no_incremental = true # default: false
[enrich_areas_of_code:git]
in_index = git_grimoirelab-raw
out_index = git_aoc_grimoirelab-enriched
#sort_on_field = metadata__timestamp
#no_incremental = false
[enrich_onion:git]
in_index = git_grimoirelab
out_index = git_onion_grimoirelab-enriched
#data_source = git
#contribs_field = hash
#timeframe_field = grimoire_creation_date
#sort_on_field = metadata__timestamp
#no_incremental = false
docker command
docker run -p 5601:5601 -p 9000:9200 -v $(pwd)/projects.json:/projects.json -v $(pwd)/dashboard.cfg:/dashboard.cfg -v $(pwd)/credentials.cfg:/override.cfg -t grimoirelab/full-nomos
| sudo find /var/lib/apt/lists -type f -delete && \ | ||
| sudo rm /tmp/fossology-common_3.6.0-1_amd64.deb \ | ||
| /tmp/fossology-nomos_3.6.0-1_amd64.deb | ||
|
|
There was a problem hiding this comment.
Nomos needs to be set up by running the conf script pgsql-conf-fix.sh located in /usr/share/fossology/setup.
We could add also other packages needed by Graal, a possible modification could be:
@@ -15,12 +15,25 @@ ADD build/fossology-nomos_3.6.0-1_amd64.deb /tmp
RUN sudo apt-get update && \
sudo apt-get -y install /tmp/fossology-common_3.6.0-1_amd64.deb \
/tmp/fossology-nomos_3.6.0-1_amd64.deb \
+ python3 \
+ python3-pip \
+ cloc \
&& \
sudo apt-get clean && \
sudo find /var/lib/apt/lists -type f -delete && \
sudo rm /tmp/fossology-common_3.6.0-1_amd64.deb \
/tmp/fossology-nomos_3.6.0-1_amd64.deb
+RUN pip3 install setuptools
+RUN pip3 install lizard>=1.16.3
+RUN pip3 install pylint>=1.8.4
+RUN pip3 install flake8>=3.7.7
+RUN pip3 install networkx>=2.1
+RUN pip3 install pydot>=1.2.4
+RUN pip3 install bandit>=1.4.0
+
+CMD /usr/share/fossology/setup/pgsql-conf-fix.sh <--- setup command needed by Nomos
+
# Entrypoint
ENTRYPOINT [ "/entrypoint.sh" ]
There was a problem hiding this comment.
I think Python3 is already in the container (it is used to run GrimoireLab tools). Same for setuptools, I think. Why are you proposing several different pip install commands? Can't we just install the main ones, and the rest would come in as dependencies? Some of them seem to be related to testing, not to the running packages (eg pylint or flake8): do we really need them in the container?
There was a problem hiding this comment.
I think Python3 is already in the container (it is used to run GrimoireLab tools). Same for setuptools, I think.
OK
Why are you proposing several different pip install commands? Can't we just install the main ones, and the rest would come in as dependencies?
Yes, you are right, it's better to get them as dependencies of graal.
Some of them seem to be related to testing, not to the running packages (eg pylint or flake8): do we really need them in the container?
Pylint and flake8 are used by some graal analyzers (https://github.com/chaoss/grimoirelab-graal/tree/master/graal/backends/core/analyzers), but they can be installed as deps of graal
There was a problem hiding this comment.
OK. Then, can I understand that from your diff, only the following two lines are needed?:
+ cloc \
...
+CMD /usr/share/fossology/setup/pgsql-conf-fix.sh
For the second one, are we really needing it, even when we're using MySQL and not PostgreSQL?
There was a problem hiding this comment.
I tried the PR again, from a fresh env. I included only cloc and eveything worked smoothly.
It would be possible to change the name of the image with something not related to nomos (maybe in the future there will be more things inside), for instance full-thirdparties WDYT?
|
Nice progress folks! Can I try this myself? The |
|
sure @svdo ! To create the image, please follow the info at third-party/README.md. Don't hesitate to write if something isn't clear, thanks! |
|
I got the build to work and created the docker image. When running it I encounter some issues though. First is that I get errors about The second is that Elasticsearch seems to die after a while. Not sure if related to the first issue, and didn't find more info about what could be causing this... |
|
Did you modify the third-party/Dockerfile-grimoirelab-nomos with the suggested changes at #230 (comment)? |
|
I did now 😉 That solves the issue about |
|
You're welcome @svdo ! WRT the second issue, maybe it's just a matter of giving more resources to Elasticsearch: If you can, please share the logs of Elasticsearch, thanks |
|
That seems to work (or more specifically the Mac version of that: https://stackoverflow.com/a/41251595). It's indexing... |
|
thanks for the feeback @svdo ! I should have pointed you to: https://github.com/chaoss/grimoirelab/tree/master/docker-compose#mac, sorry If you want to check the evolution of your indexes, you can use: |
|
Hmm, it failed again. Tail of |
I'm not totally sure but this error should pop up either:
If this is the case, you should delete the folders If it's possible, could you share your projects.json and dashboard.cfg/setup.cfg? I'll try to replicate your issue tomorrow. |
|
Ok I'm a step further now. I think giving more memory to docker (4GB instead of 2GB) seems to make elasticsearch happier. I have taken a small project and indexed it. The Problem is now that I don't see any dashboards about this. The top navigation bar does have the "Code Complexity" and "Code License" entries, but the submenus of both are empty. Any ideas?
|
Great!
This is a bug (I guess), it will be fixed in the next release. Please click on the Dashboard button and search for colic (or cocom).
Thanks! |
|
Cool that works. A couple of things that stand out to me:
|
|
Thank you for your feedback @svdo
Just to make sure I got it right, you would like to know if a repository has a license file and if so, which license type, right?
This requires some changes in the code, but it shouldn't be difficult. Can you open an issue in ELK about this feature request? |
Exactly. Something else that is probably a lot more complicated but also very valuable would be all licenses of dependencies that are used, either directly or indirectly... This is a big thing for enterprises in terms of open source license compliance.
Done! |
That sounds really interesting. Can you open an issue in graal repository about it?
Thanks |
|
|
thanks @svdo ! |
Signed-off-by: Valerio Cosentino <[email protected]>
[third-party] Rename image and add cloc dep
This code showcases the 3p image via an example, which has been developed at chaoss#230 Signed-off-by: Valerio Cosentino <[email protected]>
This code showcases the 3p image via an example, which has been developed at #230 Signed-off-by: Valerio Cosentino <[email protected]>
For now, it inclues Dockerfiles and other related files to produce FOSSology Debian packages, via a Docker container, and a Docker container like grimoirelab/full with nomos also installed.