How to build without fontawesome pro licence?

[wizofaus]

I'm keen to build this extension myself to get a better idea of how it works and possibly add some additional features, but it seems I need an auth token for the fontawesome library? Do I actually have a to purchase it to build this extension? Why do I even need fontawesome, not clear why this extension would require something like that...

[wizofaus]

BTW I removed the references to fontawesome pro (it seems you're only using it for some icons), but now when running npm run build I get

...\github\blmage\duolingo-solution-viewer\node_modules\rollup\dist\shared\loadConfigFile.js:480
? (await import(url.pathToFileURL(fileName).href)).default
^^^^^^

SyntaxError: Unexpected token import
at createScript (vm.js:80:10)

Seems this might be a nodejs version issue though, will upgrade to the latest.

[wizofaus]

Installed latest nodejs and now it gets stuck on

src/ui.js → dist/src/ui.js...

It was there a good 15 minutes, I killed it and reran "npm run build" but is just stuck there again.

The process is node.exe with the command line
"[...]\github\blmage\duolingo-solution-viewer\node_modules.bin\..\rollup\dist\bin\rollup" -c --environment production

When run with NODE_DEBUG=cluster,net,http,fs,tls,module,timers node app.js the last line in the output is

MODULE 21764: Module._load REQUEST [...]\github\blmage\duolingo-solution-viewer\node_modules\param.macro\dist\index.js parent: [...]\github\blmage\duolingo-solution-viewer\node_modules\babel-plugin-macros\dist\index.js

[wizofaus]

Left it running a couple of hours and finally got this:

<--- Last few GCs --->

[13572:03E2FCA8] 5547895 ms: Mark-sweep (reduce) 1012.4 (1025.0) -> 1011.4 (1026.0) MB, 3411.1 / 0.0 ms (average mu = 0.102, current mu = 0.027) task scavenge might not succeed
[13572:03E2FCA8] 5551672 ms: Mark-sweep (reduce) 1012.4 (1025.0) -> 1011.4 (1026.0) MB, 3684.3 / 0.0 ms (average mu = 0.063, current mu = 0.025) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00FCE5C9 v8::internal::Heap::PageFlagsAreConsistent+2169
2: 00FC4651 v8::internal::Heap::CollectGarbage+1985
3: 00FC2D03 v8::internal::Heap::AllocateExternalBackingStore+1155
npm verb lifecycle [email protected]~build: unsafe-perm in lifecycle true
...

[blmage]

Do I actually have a to purchase it to build this extension?

I must admit that I completely overlooked this when switching to the Font Awesome Pro packages, given all the benefits, and the fact that I did not really expect contributions (thanks for considering to!).

Sorry, I don't have an answer right now, but I'll be thinking about it to try and find the best solution.

It was there a good 15 minutes, I killed it and reran "npm run build" but is just stuck there again.

There is probably an unresolved import that remains somewhere.

[wizofaus]

Why would an unresolved import cause that? E.g. before commenting out the references to FontAwesome pro I simply got errors that the import couldn't be found... Do you have instructions for building at all? Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: blmage ***@***.***> Sent: Thursday, May 6, 2021 10:06:41 PM To: blmage/duolingo-solution-viewer ***@***.***> Cc: wizofaus ***@***.***>; Author ***@***.***> Subject: Re: [blmage/duolingo-solution-viewer] How to build without fontawesome pro licence? (#155) Do I actually have a to purchase it to build this extension? I must admit that I completely overlooked this when switching to the Font Awesome Pro packages, given all the benefits, and the fact that I did not really expect contributions (thanks for considering to!). Sorry, I don't have an answer right now, but I'll be thinking about it to try and find the best solution. It was there a good 15 minutes, I killed it and reran "npm run build" but is just stuck there again. There is probably an unresolved import that remains somewhere. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#155 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABI5UAKB5TKGXFFUXDFFY3TTMKA5DANCNFSM44GBTGKQ>.

[wizofaus]

It looks like faKey, faThumbtack, faArrowFromLeft and faArrowToRight, are the only icons you needed from fontawesome pro - for now I've assigned this with existing free icons, cleaned out node_modules, rerun npm install and npm run build and it seems to have built successfully...now just need to work out how to install into Chrome...(tried npm run watch but not working, I gather I can zip the 'dist' folder, will let you know)

[wizofaus]

Alright, got it building and installed using npm run build-debug and adding the dist folder as an unpacked extension to Chrome - it's definitely doing something (some of my console.log statements are executing) but it's not actually working, there's no solution lists getting loaded etc.
This is some of the additional debug output I've added:

eventListener {"type":"_duo-sv_-action_request_","action":"get_current_translation_challenge","value":{"result":"correct","statement":"There are many trees on top of the mountain.","userAnswer":"山の上 に たくさん 木 が あり ます"}}
post ACTION_RESULT_FAILURE
eventListener {"type":"_duo-sv_-action_result_","action":"get_current_translation_challenge","result":"failure"}

[wizofaus]

It's failing in sendMessageToBackground, chrome.runtime.sendMessage is setting chrome.runtime.lastError to { "message": "The message port closed before a response was received." }.

I restored background.js back to what it was prior to my modifications and it seems OK now, looks like it had got corrupted/replaced when I was trying to reinitialize the project as a new chrome extension (hint: don't do that!)

So I've finally got to where I wanted to get to, which is get all the raw data that's used to generate the solutions list dumped to the console. Definitely noticing some major issues for some lessons where there's 50MB events to parse, but I'm figuring it out as best I can - you're obviously far more patient than me trying to figure it all out from scratch!

[wizofaus]

BTW one thing I did notice is that it seems that from
"compactTranslations":["朝、鳥の声がしました。","[あさ/朝][、/は][とり/鳥]が鳴いて[い/][ました/た]。","[あさ/朝][、/は][とり/鳥]の[こえ/声/鳴き声]が[しました/した]。","[あさ/朝][、/は][とり/鳥]が鳴[きました/いた]。"],
it generated 264 "accepted solutions" - it's not clear on what basis it decides to split them up. But obviously it's using additional information as at least some of the listed accepted solutions used Japanese characters not in that list.

(I can see now you're not using 'compactTranslations' at all, but the 'grader' information, which seems inordinately convoluted!)

[blmage]

Why would an unresolved import cause that?

I'm not sure, but it's probably due to one of the Rollup plugins. Anyway, every time the build process hangs, it's because I forgot to npm i or to remove an obsolete import.

I can see now you're not using 'compactTranslations' at all, but the 'grader' information, which seems inordinately convoluted!

This seems to be especially true for solutions written in non-Latin alphabets, but so far it's been hard for me to figure out why / if it contains mostly irrelevant or redundant values / if there's anything we can do about it (I don't know or learn any of the corresponding languages) - your feedback would be very valuable here!

For example, I had not yet considered this until now, but would parsing the compactTranslations strings give solution lists that are sufficiently exhaustive? Knowing that this could be a perfect fallback for when a solution graph contains too many vertices.

Also, I remember finding "normalization data" in the localStorage, which seems to correspond to pairs of equivalent strings for non-Latin alphabets (again). Could most of the solutions actually correspond to different but equivalent combinations of (non-)normalized tokens?

You can run this code in the console if you want to take a look at it for Japanese e.g. :

JSON.parse(window.localStorage.getItem('duo.clientSideGradingConfig')).language_data['ja'].normalization_data;

[wizofaus]

From what I've seen, you can't get all accepted solutions from compactTranslations - but the ones you can't get are generally rubbish anyway that it makes no sense to use, I'm pretty sure it's all automatically generated, which is why some exercises have 100s of 1000s of accepted solutions. I'll do more investigation when I have time. Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: blmage ***@***.***> Sent: Friday, May 7, 2021 8:49:48 PM To: blmage/duolingo-solution-viewer ***@***.***> Cc: wizofaus ***@***.***>; Author ***@***.***> Subject: Re: [blmage/duolingo-solution-viewer] How to build without fontawesome pro licence? (#155) Why would an unresolved import cause that? I'm not sure, but it's probably due to one of the Rollup plugins. Anyway, every time the build process hangs, it's because I forgot to npm i or to remove an obsolete import. I can see now you're not using 'compactTranslations' at all, but the 'grader' information, which seems inordinately convoluted! This seems to be especially true for solutions written in non-Latin alphabets, but so far it's been hard for me to figure out why / if it contains mostly irrelevant or redundant values / if there's anything we can do about it (I don't know or learn any of the corresponding languages) - your feedback would be very valuable here! For example, I had not yet considered this until now, but would parsing the compactTranslations strings give solution lists that are sufficiently exhaustive? Knowing that this could be a perfect fallback for when a solution graph contains too many vertices. Also, I remember finding "normalization data" in the localStorage, which seems to correspond to pairs of equivalent strings for non-Latin alphabets (again). Could most of the solutions actually correspond to different but equivalent combinations of (non-)normalized tokens? You can run this code in the console if you want to take a look at it for Japanese e.g. : JSON.parse(window.localStorage.getItem('duo.clientSideGradingConfig')).language_data['ja'].normalization_data; — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#155 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABI5UAOZG3QJQYKY2CXY2LTTMPAUZANCNFSM44GBTGKQ>.

[wizofaus]

Anyway, to anyone else wondering the same thing, yes, it's fine to build without using fontawesome-pro, you just need to remove the imports from package.json:

• "@fortawesome/fontawesome-pro": "^5.15.1",
• "@fortawesome/pro-regular-svg-icons": "^5.15.2",
• "@fortawesome/pro-solid-svg-icons": "^5.15.2",

Then in ui.js, remove or comment out

// import { faKey, faThumbtack } from 'react-icons/fa';
and
//import { faArrowFromLeft, faArrowToRight } from '@fortawesome/pro-solid-svg-icons'

Then add

window.faKey = faCheck
window.faThumbtack = faCheck
window.faArrowFromLeft = faCheck
window.faArrowToRight = faCheck

just before library.add(...)

So far I haven't noticed anywhere that this negatively affects the UI, but the obviously parts of this solution viewer that I haven't quite figured out yet (just from looking at the code it seems to have a lot more functionality than the bits I use).

[wizofaus]

BTW don't suppose you've figured out a way to detect the URL of the discussion page? Each exercise seems to have an ID associated with it but I can never find it searching with DevTools among any of the source/network traffic/console output etc.

[blmage]

The extension used to send requests to this URL (the same that is used when the "Discuss" button is clicked) to fetch the comment IDs:

https://www.duolingo.com/sentence/[discussion_id]?learning_language=[locale]&ui_language=[locale]

in which discussion_id corresponds to the id value that can be found in the exercises, but to prevent being detected as a bot by the API, it now detects requests to this URL instead (used in forum comments):

https://forum-api.duolingo.com/comments/[comment_id]

knowing that it can then use the sentence_id (= discussion_id) and sentence_language from the result to find back the right exercise.

[wizofaus]

Ok I have the extension logging out the discussionId value now but that's a long hex value like f58f2b8ab6198f612ed1b6c3b12b86bd. So far I haven't seen commentId anywhere, even though I added in logging to all the functions that work with it.

[wizofaus]

Ok I get "commentId" logged out when I load the comments page. In fact it seems Duo doesn't even know what the "commentId" is when showing an exercise in a lesson, but uses some sort of redirection mechanism to bring up the forum page, which I'm trying to work out (actually it's not working for me at all at the moment for some reason).

[ssieb]

Which information do you have and what are you trying to find?

[wizofaus]

Actually I tried with Edge (which doesn't support the plugin obviously), and I can see from its network traffic that when it fetches https://www.duolingo.com/sentence/[discussion_id]?learning_language=[locale]&ui_language=[locale] the response does include the comment_id (in property id of the returned JSON object). Odd that searching network traffic doesn't find it though.

[wizofaus]

Basically I was just hoping to enhance the plugin to dump out useful summary information to the log about each lesson as I did it, and the forum discussion URL was one of the ones I definitely wanted, as often the "discuss" link doesn't even work.
Currently it seems it only queries that https://www.duolingo.com/sentence/... URL when you click "discuss", so I'd have to add code to the plugin to query it automatically I guess. Problem is in Chrome currently it's always returning 502. I confirmed that the same URL loads fine in Edge (logged in as the same Duo user - it doesn't work if you're not logged in, so obviously relies on some cookie value)

[wizofaus]

BTW deleting the _pxhd cookie in Chrome solved the 502 issue, Discuss button is now working.
However it doesn't change the fact that you need to click on "discuss" before Duo attempts to find out the associated commentId (for the forum URL).

[wizofaus]

Interestingly if I click on the 'discuss' icon for an exercise that has no comments yet, it navigates to this URL instead:

https://www.duolingo.com/new-comment?fromLanguage=en&learningLanguage=ja&sentenceId=7ff8ee49b6b07b8976362d0c7052ae4e

Visible in the browser address bar. And it seems there is no commentId assigned.