Use Julia BinaryBuilder to automate installing the GAMS programs via Yggdrasil vs. downloading manually and adding to PATH?

[bdklahn]

Hello! Thanks for supporting Julia!

We purchased the license for some data files which come in gdx format. I wanted to convert to Apache Arrow tables, on the way to provide a more "normalized" form for ad-hoc use in a project.

My thinking was to convert via gdxdump or gdx2sqlite (to maybe preserve any helpful type information).
After manually downloading the linux version, I was able to do that. Then I was going to have a Julia function to set the PATH and use that. But I wondered if a more portable and easy way of getting the code via Julia might be to use Artifacts.toml, via BinaryBuider by contributing a build_tarballs.jl file to Yggdrasil.

The following is a crude minimal starter (GNU/Linux only, right now) I had been working on, when someone pointed out this project and that possibly the GAMS license might not permit this.
https://github.com/bdklahn/Yggdrasil/blob/master/G/GAMS/build_tarballs.jl

So I don't know if it's even possible, in terms of license, but . . .

Could this work by having users first download a license file to a standard location, then the Artifact could be "lazy" installed such that a prompt would show, if the license file was not found or usable?

Thanks for your time and consideration.

[odow]

If (and it's a big if), GAMS moves forward with some sort of automated install, you should not use Yggdrasil. The packages it builds need to be open-source.

A better precedent is to follow with I've done with Gurobi and Xpress:

• https://github.com/jump-dev/Gurobi_jll.jl
• https://github.com/jump-dev/Xpress_jll.jl

These are manually constructed JLL packages that use the first-party artifacts uploaded to conda. I also have written permission from each company to provide these third-party packages.

[bdklahn]

Thanks for the very useful/helpful input @odow !

I'll try and summarize, just to make sure I'm clear on my understanding (and maybe for others happening upon this).

• Yggdrasil (automated packaging, ultimately storing artifacts in a central repo) is meant for truly open source software.
• One might be able to get permission to redistribute artifacts already in an officially-sanctioned central repo (E.g. Anaconda dist) on a case-by-case basis; still clearly communicating the implicit and explicit license terms.

Anyway, BinaryBuilder is a bit heavy for use cases of just "pulling and re-packaging". Although there are many examples of doing that in Yggdrasil, Artifacts and JJL Wrappers is all needed for such a case. I did like the potential simplicity of the "PR one file to distribute".

In any case, maybe we should've looked a little more carefully at our data provider before purchasing. It seems they may be unnecessarily closely coupled to a particular modeling system and maybe not very open-source friendly. I.e., the programs they, themselves, create seem to only run on Windows.

Update:
I might have been confused my your switch from third to second person: "GAMS" vs. "you". Maybe you were referring to GAMS and not me.

In that case, GAM-dev might pretty easily officially sanction this in a an automated way.
The Julia JLL Artifacts pattern might simply pull from their CDN (including EULA files). This repo README.md could direct folks to agree and/or pay for the approprate license.

Artifacts makes it so easy for people to not have to go and know which which package they need to download and install where . . . and manually confirm the hash match . . . etc. I.e, users on multiple platforms could install in exactly the same simple way.
Idk. I'm imagine that could free up a little bit of tech support/feature request time:
"Do these three manual steps, and maybe still have to deal with library incompatibilitiy issues or whatever." or
use "add GAMS" (utilizing an official GAMS_jll package), and be done. :-)

Update 2:
I guess you probably meant either/or, in regards to whoever did this.
It looks like, for the "manual" part, you use (something like) the following script.
https://github.com/jump-dev/Gurobi_jll.jl/blob/master/scripts/update_artifacts.jl
(with an implicit host dependency on wget)

I guess that might have to be adjusted for the platform type packages available via the CDN. But maybe it could be easier here, if the employed self-extractor versions "just worked" for any supported platforms. I don't know if it's, otherwise, worth seeing if the following might be of use.
https://github.com/JuliaPackaging/JLLWrappers.jl/tree/master

Anyway, I'm not sure I have room on my full plate for this . . . and given I was only initially interested in a little easier way to read and convert gdx files in Julia, with no apparent native Julia library . . .

Thanks for the input.

[odow]

By "You" I meant GAMS. I have no plans to work on this, and I would discourse third-parties from coming up with their own solution.

The Artifacts must be a .tar.gz that requires no action other than extracting.

See Gurobi_jll: in addition to the Artifacts.toml file (which is created by scripts/update_artifacts.jl:
https://github.com/jump-dev/Gurobi_jll.jl/blob/master/Artifacts.toml

I manually wrote a JLLWrapper script for each supported platform:
https://github.com/jump-dev/Gurobi_jll.jl/blob/master/src/wrappers/aarch64-apple-darwin.jl

[bdklahn]

Yeah, it makes the most sense for GAMS to do this (assuming they want their software to be most widely attractive to folks).

I was aware that Artifacts only/best works with a more universal archive formats like tar.gz (& tar.bz, now?), where there is also easy support for for hashing of the extracted directory (git-tree-hash). But, yeah, it occurs to me that this is a little bit of a pain if/when dealing with (this) distribution via platform-specific self extractors.

Yes, I directly referenced your update_artifacts.jl script in my previous comment (which is not necessarily portable, with implicit wget and gzcat dependencies).
That JLLWrapper code is useful to see. Thanks!

All this said, it seems like maybe the easiest way for GAMS to auto-download is to just provide some prompts on install or load, if not finding an executable gams in the executable PATH.

E.g. prompt (to accept the url-referenced EULA and) download to package-version-specific Scratch space . . .
E.g.
https://github.com/bdklahn/TestMamba.jl/blob/f434b0516f23ecfd239a7c4972f1895bea50068a/deps/build.jl#L9-L14
https://github.com/bdklahn/TestMamba.jl/blob/f434b0516f23ecfd239a7c4972f1895bea50068a/src/TestMamba.jl#L10-L12

. . . or allow the user to override that default location with their own path (which they must not have added to their PATH).
The code could first test if the current platform is supported, choosing the appropriate (versioned) url, and MD5 hash check that before extraction.
I figure that would be no less safe and reproduceable than the current installation pattern.

In that case it might be best to close this issue and open a new, more subject-approprate, issue for that.

[renkekuhlmann]

Hi, thanks for the idea and the discussion. However, this is not on our agenda at the moment.