Wrong path picked up when execution occurs through script not in same directory

[mauro3]

When generating and evaluating notebooks with https://github.com/fredrikekre/Literate.jl via a scrip which is not in the output directory, then @nbinclude generates a path which contains the path of the Literate script and not the path of the notebook-directory. I'm pretty sure that this is a NBInclude issue, but happy to move to Literate.jl if that is not the case.

To reproduce I setup a repo: https://github.com/mauro3/NBinclude-issue-tmp.jl

After cloning that repo, reproduce with:

$ julia --project

julia> using Pkg; Pkg.instantiate()

julia> include("gen-notebook.jl")

Backtrace:

julia> include("gen-notebook.jl")
[ Info: generating notebook from `~/julia/issues/literate-nbinclude/scripts/00.jl`
[ Info: executing notebook `00.ipynb`
[ Info: writing result to `~/julia/issues/literate-nbinclude/notebooks/00.ipynb`
[ Info: generating notebook from `~/julia/issues/literate-nbinclude/scripts/lit-script.jl`
[ Info: executing notebook `lit-script.ipynb`
┌ Error: error when executing notebook based on input file: `~/julia/issues/literate-nbinclude/scripts/lit-script.jl`
└ @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:748
ERROR: LoadError: LoadError: SystemError: opening file "~/julia/issues/literate-nbinclude/00.jl": No such file or directory
in expression starting at ~/julia/issues/literate-nbinclude/notebooks/lit-script.ipynb:2
when executing the following code block from inputfile `~/julia/issues/literate-nbinclude/scripts/lit-script.jl`

```julia
using NBInclude
@nbinclude("00.jl")
```


Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] execute_block(sb::Module, block::String; inputfile::String, fake_source::String)
    @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:866
  [3] execute_notebook(nb::Dict{Any, Any}; inputfile::String, fake_source::String)
    @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:764
  [4] (::Literate.var"#38#40"{Dict{String, Any}})()
    @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:744
  [5] cd(f::Literate.var"#38#40"{Dict{String, Any}}, dir::String)
    @ Base.Filesystem ./file.jl:112
  [6] jupyter_notebook(chunks::Vector{Literate.Chunk}, config::Dict{String, Any})
    @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:743
  [7] notebook(inputfile::String, outputdir::String; config::Dict{Any, Any}, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:execute,), Tuple{Bool}}})
    @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:680
  [8] top-level scope
    @ ~/julia/issues/literate-nbinclude/gen-notebook.jl:6
  [9] include(fname::String)
    @ Base.MainInclude ./client.jl:476
 [10] top-level scope
    @ REPL[7]:1
in expression starting at ~/julia/issues/literate-nbinclude/gen-notebook.jl:6

julia>

What happens:

• Literate.jl tries to evaluate the generated notebook's cell via https://github.com/fredrikekre/Literate.jl/blob/60c08dc23e3f6dc6fb2ae04571f9f6e8a0c6d2fa/src/Literate.jl#L862 Of note is that it gets executed within the right directory, namely notebooks for this example, see https://github.com/fredrikekre/Literate.jl/blob/60c08dc23e3f6dc6fb2ae04571f9f6e8a0c6d2fa/src/Literate.jl#L743
• this then goes to NBInclude.jl to evaluate the @nbinclude call, at https://github.com/stevengj/NBInclude.jl/blob/d1cff61be648d2f8400f3eb4bc0509f622917088/src/NBInclude.jl#L65 the bug occurs. The path variable changes from "00.jl" to ".../literate-nbinclude/00.jl". I'm not sure what Base._include_dependency(m, path) is supposed to do but here it should return ".../literate-nbinclude/notebooks/00.jl"

[stevengj]

Does it work if you do include? nbinclude is supposed to pick the path the same way.

[mauro3]

It's a bit tricky to do with include because the only other backend of Literate which allows execution is markdown, which does not produce executable files, i.e. the markdown file cannot be included then.

Anyway, I did a bit more digging and found that if I execute @__DIR__ in the script being processed by Literate, it will pick up the output folder (i.e. the notebooks/ folder). This seems to indicate that Literate is changing the current directory to the correct place but this is then not picked up by @nbinclude.

Note that I have this work-around: https://github.com/eth-vaw-glaciology/GlacioTeaching.jl/blob/ce04d95629c73df2f3d23337a1a9aaba93770239/src/GlacioTeaching.jl#L14

[mauro3]

I now updated the linked repo. The script which shows the issue is now scripts/lit-script-include.jl and scripts/lit-script.jl just does the @__DIR__ thingy.

Also, the script is now more realistic with https://github.com/mauro3/NBinclude-issue-tmp.jl/blob/caeb185732acaee10ddf99c7e7e39f4b56e3c1fd/scripts/lit-script-include.jl#L3-L4 being used for notebook creation and https://github.com/mauro3/NBinclude-issue-tmp.jl/blob/caeb185732acaee10ddf99c7e7e39f4b56e3c1fd/scripts/lit-script-include.jl#L6 otherwise.

[stevengj]

When you do @nbinclude, it acts like include — it doesn't change the current working directory, but it should change @__FILE__ and @__DIR__ to that of the notebook file. And the path of the .ipynb file itself should be interpreted relative to the path of the file that is calling @nbinclude.

I'm a little confused about whether one of these behaviors is not happening in your case, or if it is happening but you want a different behavior.