Developing Local Packages

Introduction

At some point you might want to split your work from a single monolithic project into a series of independent and smaller projects.

For example, you might want to have a folder somewhere in your system called “MyPackages” or something similar, that contains a series of folders, each one for a different packages. Also, each one of your packages could contain a different git repo, or all of them could live within a single mono-repo.

That’s actually the easy part, as it just involves moving some files around.

But how can you now interact with your code? You need to tell Julia that these projects exists, or otherwise Julia won’t know what to do.

So let’s dig into this.

For simplicity, let’s assume we only have two projects: myLib and myApp, and myApp depends on myLib.

Basic workflow with just include

If you are lazy, just include myLib into myApp with a long include command, that could look like this:

myBase = "/home/martin/Desktop/MyPackages/"
include(myBase * "myLib/src/myLib.jl")

Or, alternatively, use a relative path like

include("../myLib.jl")

This will include all the code in myLib.jl into your current project.

However, even if myLib.jl defines a module and exports, exports won’t work properly, and inside myApp you will need to do myLib.foo().

In order to get the exports right, you might also want to do:

using .myLib

Now this is fine, this will work with Revise and allow you to change things quickly.

This is perfectly fine as a first step towards better organizing your code. All the code from myLib will be included into myApp.

Limitations of just including everthing

Now, is there any problem with this approach? Are there any limitations of simply including all the code?

Actually, this is actually the same way that a C or C++ programmer would work, using their #include statement. So it’s nothing to be terrible ashamed of.

The only problem with this is that you might be missing out on some modern tooling. For example, older languages like C/C++ and even Python gave rise to a notion called “dependency hell”, something unheard of in Julia — where the only “problem” we have is managing to understanding how a very well designed tool like Pkg actually works for various scenarios.

For example, let’s say that you stop developing myApp for a while, and when you go back to it, it turns out that you made some breaking changes to myLib and now myApp doesn’t work. If you want to be able to run myApp inmmediately without any refactoring, you would need to get the exact version of myLib that you were using before.

This is something you could easily accompish if you were using Pkg: just add [email protected] and you’re set! Then you can slowly work up to the new version of myLib without losing access to your existing functionality.

What about testing in this setting? Let’s say that you continue developing myLib and some tests of new functionality begin to fail. Now the tests in myApp which don’t rely on the new functionality will fail, too, as automated testing is a feature of Pkg.

What about sharing code with others, and integrating their changes? Everything is doable, but just more difficult without Pkg.

Basic workflows with local packages

Ok, for some projects where we intend to make changes every now and then, or share code with others who might in turn make changes, getting some help from the package managers sounds like a good deal.

We will then need to learn how to use it with local packages.

Option 1: add your local package

If LibA is it’s own Git repo, we can then add it as an unregistered package in the same way that we would add an unregistered package hosted on Github, for example. Just give Pkg a local path where you have a Git repo.

Remeber this could be an absolute or a relative path. Assuming myApp also lives in the myPackages folder, you can do something like:

]add ../myLib

Now, what happens if you change something in myLib? The changes won’t be reflected immediately on myApp. You would need to commit the changes on myLib repo, and then update myLib from myApp.

So, this approach will only work for code that you are not expecting to change quickly.

Option 2: dev your local package

Now, this is similar to the previous approach

]dev ../myLib

Should you push your “MyPackages” folder to your LOAD_PATH?

You might get bored of having to deal with abosolute or relative paths, and might want to solve this issue once and for all. So you might have come accross LOAD_PATH.

Yes, you can do:

myBase = "/home/martin/Desktop/MyPackages/"
push!(LOAD_PATH, myBase)

Congratulations, you can now directly do

using myLib

without any need for add or dev. And it will work if you need to call a function in myLib directly from the REPL.

However, this doesn’t solve any issues with dependencies.

As soon as you need to precompile myApp (which depends on myLib) this will fail, as you’ll get an unresolved reference. It turns out that, while Julia can find myLib, Pkg cannot… unless you do run Pkg.add.

However, things aren’t so easy, as Pkg.add won’t work because it searches for a registry…

Creating a local registry

So, let’s say that you want to go a step further, and be able to use your local packages in the same way that public registered packages. The way to do this is by setting up a local registry. This could be useful to store propietary code within an organization, or to set up an internal server in order to get access to packages without an internet connection.

In order to do this, the basic package you might want to look into is LocalRegistry.jl.

Conclusion

Organizing your work into local packages can seem intimidating at first. While you might be tempted to use the simplest possible workflow (just include everything) investing a little time in learning new tools that integrate well with git and let you collaborate with more people will eventually pay off.