📦 Package Node.js applications into executable binaries 📦

Overview

caxa

📦 Package Node.js applications into executable binaries 📦

Source Package Continuous Integration

Support

Why Package Node.js Applications into Executable Binaries?

  • Simple deploys. Transfer the binary into a machine and run it.
  • Let users test an application even if they don’t have Node.js installed.
  • Simple installation story for command-line applications.
  • It’s like the much-praised distribution story of Go programs, but for Node.js.

Features

  • Works on Windows, macOS, and Linux.
  • Simple to use. npm install caxa and call caxa from the command line. No need to declare which files to include; no need to bundle the application into a single file.
  • Supports any kind of Node.js project, including those with native modules (for example, sharp, @leafac/sqlite (shameless plug!), and others).
  • Works with any Node.js version.
  • Packages in seconds.
  • Relatively small binaries. A “Hello World!” application is ~30MB, which is terrible if compared to Go’s ~2MB, and worse still if compared to C’s ~50KB, but best-in-class if compared to other packaging solutions for Node.js.
  • Produces .exes for Windows, simple binaries for macOS/Linux, and macOS Application Bundles (.app).
  • Based on a simple but powerful idea. Implemented in ~200 lines of code.
  • No magic. No traversal of require()s trying to find which files to include; no patches to Node.js source.

Anti-Features

  • Doesn’t patch the Node.js source code.
  • Doesn’t build Node.js from source.
  • Doesn’t support cross-compilation (for example, building a Windows executable from a macOS development machine).
  • Doesn’t support packaging with a Node.js version different from the one that’s running caxa (for example, bundling Node.js 15 while running caxa with Node.js 14).
  • Doesn’t hide your JavaScript source code in any way.

Installation

$ npm install --save-dev caxa

Usage

Prepare the Project for Packaging

  • Install any dependencies with npm install or npm ci.
  • Build. For example, compile TypeScript with tsc, bundle with webpack, and whatever else you need to get the project ready to start. Typically this is the kind of thing that goes into an npm prepare script, so the npm ci from the previous point may already have taken care of this.
  • If there are files that shouldn’t be in the package, remove them from the directory. For example, you may wish to remove the .git directory.
  • You don’t need to npm prune --production and npm dedupe, because caxa will do that for you from within the build directory. (Otherwise, if you tried to npm prune --production you’d uninstall caxa, which should probably be in devDependencies.)
  • It’s recommended that you run caxa on a Continuous Integration server. (GitHub Actions, for example, does a shallow fetch of the repository, so removing the .git directory becomes unnecessary.)

Call caxa from the Command Line

$ npx caxa --help
Usage: caxa [options]


Options:
  -d, --directory <directory>               The directory to package.
  -c, --command <command-and-arguments...>  The command to run and optional arguments to pass to
                                            the command every time the executable is called. Paths
                                            must be absolute. The ‘{{caxa}}’ placeholder is
                                            substituted for the folder from which the package
                                            runs. The ‘node’ executable is available at
                                            ‘{{caxa}}/node_modules/.bin/node’. Use double quotes
                                            to delimit the command and each argument.
  -o, --output <output>                     The path at which to produce the executable.
                                            Overwrites existing files/folders. On Windows must end
                                            in ‘.exe’. On macOS may end in ‘.app’ to generate a
                                            macOS Application Bundle.
  -V, --version                             output the version number
  -h, --help                                display help for command

Examples:

  Windows:
  > caxa --directory "examples/echo-command-line-parameters" --command "{{caxa}}/node_modules/.bin/node" "{{caxa}}/index.js" "some" "embedded arguments" --output "echo-command-line-parameters.exe"

  macOS/Linux:
  $ caxa --directory "examples/echo-command-line-parameters" --command "{{caxa}}/node_modules/.bin/node" "{{caxa}}/index.js" "some" "embedded arguments" --output "echo-command-line-parameters"

  macOS (Application Bundle):
  $ caxa --directory "examples/echo-command-line-parameters" --command "{{caxa}}/node_modules/.bin/node" "{{caxa}}/index.js" "some" "embedded arguments" --output "Echo Command Line Parameters.app"

Here’s a real-world example of using caxa. This example includes packaging for Windows, macOS, and Linux; distributing tags with GitHub Releases Assets; distributing Insiders Builds for every push with GitHub Actions Artifacts; and deploying a binary to a server with rsync (and publishing an npm package as well, but that’s beyond the scope of caxa).

Call caxa from TypeScript/JavaScript

Instead of calling caxa from the command line, you may prefer to write a program that builds your application, for example:

import caxa from "caxa";

(async () => {
  await caxa({
    directory: "examples/echo-command-line-parameters",
    command: [
      "{{caxa}}/node_modules/.bin/node",
      "{{caxa}}/index.js",
      "some",
      "embedded arguments",
    ],
    output: "echo-command-line-parameters",
  });
})();

You may need to inspect process.platform to determine in which operating system you’re running and come up with the appropriate parameters.

Fine Points

Calling an Executable That Isn’t node

If you wish to run a command that isn’t node, for example, ts-node, you may do so by extending the PATH. For example, you may run the following on macOS/Linux:

$ caxa --directory <directory> --command "env" "PATH={{caxa}}/node_modules/.bin/:\$PATH" "ts-node" "{{caxa}}/index.ts" --output <output>

Preserving the Executable Mode of the Binary

This is only an issue on macOS/Linux. In these operating systems a binary must have the executable mode enabled in order to run. You may check the mode from the command line with ls -l: on an output that reads like -rwxr-xr-x [...]/bin/node, the xs represent that the file is executable.

Here’s what you may do when you distribute the binary to ensure that the file mode is preserved:

  1. Create a tarball or zip. The file mode is preserved through compression/decompression, and macOS/Linux (most distributions, anyway) come out of the box with software to uncompress tarballs and zips—the user can just double-click on the file.

    You may generate a tarball with, for example, the following command:

    $ tar -czf <caxa-output>.tgz <caxa-output>

    Fun fact: Windows 10 also comes with the tar executable, so the command above works on Windows as well. Unfortunately the File Explorer on Windows doesn’t support uncompressing the .tgz with a double-click (it supports uncompressing .zip, however). Fortunately, Windows doesn’t have issues with file modes to begin with (it simply looks for the .exe extension) so distributing the caxa output directly is appropriate.

  2. Fix the file mode after downloading. Tell your users to run the following command:

    $ chmod +x <path-to-downloaded-application>

    In some contexts this may make more sense, but it requires your users to use the command line.

Detect Whether the Application Is Running from the Packaged Version

caxa doesn’t do anything special to your application, so there’s no built-in way of telling whether the application is running from the packaged version. It’s part of caxa’s ethos of being as out of the way as possible. Also, I consider it to be a bad practice: an application that is so self-aware is more difficult to reason about and test.

That said, if you really need to know whether the application is running from the packaged versions, here are some possible workarounds in increasing levels of badness:

  1. Set an environment variable in the --command, for example, --command "env" "CAXA=true" "{{caxa}}/node_modules/.bin/node" "...".
  2. Have a different entrypoint for the packaged application, for example, --command "{{caxa}}/node_modules/.bin/node" "caxa-entrypoint.js".
  3. Receive a command-line argument that you embed in the packaging process, for example, --command "{{caxa}}/node_modules/.bin/node" "application.js" "--caxa".
  4. Check whether __dirname.startsWith(path.join(os.tmpdir(), "caxa")).

The Current Working Directory

Even though the code for the application is in a temporary directory, the current working directory when calling the packaged application is preserved, and you may inspect it with process.cwd(). This is probably not something you have to think about—caxa just gets it right.

How It Works

The Issue

As far as I can understand, the root of the problem with creating binaries for Node.js projects is native modules. Native modules are libraries written at least partly in C/C++, for example, sharp, @leafac/sqlite (shameless plug!), and others. There are at least three issues with native modules that are relevant here:

  1. You must have a working C/C++ build system to install these libraries (C/C++ compiler, make, Python, and so forth). On Windows, you must install windows-build-tools. On macOS, you must install the Command-Line Tools (CLT) with xcode-select --install. On Linux, it depends on the distribution, but on Ubuntu sudo apt install build-essential is enough.

  2. The installation of native modules isn’t cross-platform. Unlike JavaScript dependencies, which you may copy from an operating system to another, native modules produce compiled C/C++ code that’s specific to the operating system on which the dependency is installed. This compiled code appears in your node_modules directory in the form of .node files.

  3. As far as I understand, Node.js insists on loading native modules from files in the disk. Other Node.js packaging solutions get around this limitation in one of two ways: They either patch Node.js to trick it into loading native modules differently; or they put .node files somewhere before starting your program.

The Solution

caxa builds on the idea of putting .node files in a temporary location, but takes it to ultimate consequence: a caxa executable is a form of self-extracting archive containing your whole project along with the node executable. When you first run a binary produced by caxa, it extracts the source the whole project (and the bundled node executable) into a temporary location. From there, it simply calls whatever command you told it to run when you packaged the project (via the --command command-line argument).

At first, this may seem too costly, but in practice it’s mostly okay: It doesn’t take too long to uncompress a project in the first place, and caxa doesn’t clean the temporary directory after running your program, so subsequent calls are effectively cached and run without overhead.

This idea is simple, but it’s super powerful! caxa supports any kind of project, including those with native dependencies, because running a caxa executable amounts to the same as installing Node.js on the user’s machine. caxa produces packages fast, because generating a self-extracting archive is a simple matter of concatenating some files. caxa supports any version of Node.js, because it simply copies the node executable with which it was called into the self-extracting archive.

Fun fact: By virtue of compressing the archive, caxa produces binaries that are naturally smaller when compared to other packaging solutions. Obviously, you could achieve the same outcome by compressing the output of these other tools, which may want to do anyway to preserve the file mode (see § Preserving the Executable Mode of the Binary).

How the Self-Extracting Archive Works

Did you know that you may append anything to a binary and it’ll continue to work? This is true of binaries for Windows, macOS, and Linux. Here’s an example to try out on macOS/Linux:

$ cp $(which ls) ./ls  # Copy the ‘ls’ binary into the current directory to play with it
$ ./ls                 # List the files, proving the that the binary works
$ echo ANYTHING >> ls  # Append material to the binary
$ tail ./ls            # You should see ‘ANYTHING’ at the end of the output
$ ./ls                 # The output should be same as before!
$ rm ls                # Okay, the test is over

The caxa self-extracting archives work by putting together three parts: 1. a stub; 2. an archive; and 3. a footer. This is the layout of these parts in the binary produced by caxa:

STUB
### CAXA ###
ARCHIVE
FOOTER

The STUB and the ARCHIVE are separated by the ### CAXA ### string. And the ARCHIVE and the FOOTER are separated by a newline. This layout allows caxa to find the footer by simply looking backward from the end of the file until it reaches a newline. And if this is the first time you’re running the caxa executable and the archive needs to be uncompressed, then caxa may find the beginning of the ARCHIVE by looking forward from the beginning until it reaches the ### CAXA ### separator.

Build a binary with caxa and inspect it yourself in a text editor (Visual Studio Code asks you to confirm that you want to open a binary, but works fine after that). You should be able to find the ### CAXA ### separator between the STUB and the ARCHIVE, as well as the FOOTER at the end.

Let’s examine each of the parts in detail:

Part 1: Stub

This is a program written in Go that:

  1. Reads itself as a file.
  2. Finds the footer.
  3. Determines whether it’s necessary to extract the archive.
    1. If so, finds the archive.
    2. Extracts it.
  4. Runs whatever command it’s told in the footer.

You may find the source code for the stub in stubs/stub.go, and the compiled stubs live in stubs. The stubs are distributed with caxa in compiled form so you don’t need a Go build system to use caxa. If you have Go build system, then you may rebuild the stubs yourself with npm run stubs. This Go program has no dependencies beyond the Go standard library, so simply installing Go is enough—there’s no need to setup Go modules or configure a $GOPATH.

This is beautiful in a way: We’re using Go’s ability to produce binaries to bootstrap Node.js’s ability to produce binaries.

Part 2: Archive

This is a tarball of the directory with your project.

Part 3: Footer

This is JSON containing the extra information that caxa needs to run your project: Most importantly, the command that you want to run, but also an identifier for where to uncompress the archive.

Using the Self-Extracting Archive without caxa

Fun fact: There’s nothing Node.js-specific about the stubs. You may use them to uncompress any kind of archive and run any arbitrary command on the output! And it’s relatively straightforward to build a self-extracting archive from scratch. For example, you may run the following in macOS:

$ cp stubs/macos an-ls-caxa
$ tar -czf - README.md >> an-ls-caxa
$ printf "\n{ \"identifier\": \"an-ls-caxa/AN-ARBITRARY-STRING-THAT-SHOULD-BE-DIFFERENT-EVERY-TIME\", \"command\": [\"ls\", \"{{caxa}}\"] }" >> an-ls-caxa
$ ./an-ls-caxa
README.md

To Where Are the Packages Uncompressed at Runtime?

It depends on the operating system. You may find the location on your system with:

$ node -p "require(\"os\").tmpdir()"

Look for a directory named caxa in there.

Why No Cross-Compilation? Why No Different Versions of Node.js besides the Version with Which caxa Was Called?

Two reasons:

  1. I believe you should have environments to work with all the operating systems you plan on supporting. They may not be your main development environment, but they should be able to build your project and let you test things. At the very least, you should use a service like GitHub Actions which lets you run build tasks and tests on Windows, macOS, and Linux.

    (I, for one, bought a PC to work on caxa. Yet another reason to support my work!)

  2. The principle of least surprise. When cross-compiling (for example, building a Windows executable from a macOS development machine), or when bundling different versions of Node.js (for example, bundling Node.js 15 while running caxa with Node.js 14), there’s no straightforward way to guarantee that the packaged project will run the same as the unpackaged version. If you aren’t using any native modules then things may work, but as soon as you introduce a new dependency that you didn’t know was native your application may break. Not only are native dependencies different on the operating systems, but they may also be different between different versions of Node.js if these versions aren’t ABI-compatible (which is why sometimes when you update Node.js you must run npm install again).

Fun fact: The gold-standard for easy cross-compilation these days is Go. But even in Go cross-compilation goes out the window as soon as you introduce C dependencies (something called CGO). It appears that many people in the Go community try to solve the issue by avoiding CGO dependencies, sometimes going to great lengths to reinvent everything in pure Go. On the one hand, this sounds like fun when it works out. On the other hand, it’s a huge case of not-invented-here syndrome. In any case, native modules seem to be much more prevalent in Node.js than CGO is in Go, so I think that cross-compilation in caxa would be a fool’s errand.

If you still insist on cross-compiling or compiling for different versions of Node.js, you can still use the stub to build a self-extracting archive by hand (see § Using the Self-Extracting Archive without caxa). You may even use https://www.npmjs.com/package/node to more easily bundle different versions of Node.js.

How the macOS Application Bundles (.app) Work

An macOS Application Bundle is just a folder with a particular structure and an executable at a particular place. When creating a macOS Application Bundle caxa doesn’t build a self-extracting archive, instead it just copies the application to the right place and creates an executable bash script to start the process.

The macOS Application Bundle may be run by simply double-clicking on it from Finder. It opens a Terminal.app window with your application. If you’re running an application that wasn’t built on your machine (which is most likely the case for your users, who probably downloaded the application from the internet), then the first time you run it macOS will probably complain about the lack of a signature. The solution is to go to System Preferences > Security & Privacy > General and click on Allow. You must instruct your users on how to do this.

Features to Consider Implementing in the Future

If you’re interested in one of these features, please send a Pull Request if you can, or at least reach out to me and mention your interest, and I may get to them.

  1. Other compression algorithms. Currently caxa uses tarballs, which are ubiquitous and reasonably efficient in terms of compression/uncompression times and archive size. But there are better algorithms out there… (See https://github.com/leafac/caxa/issues/1.)

  2. Add support for signing the executables. There are limitations on the kinds of executables that are signable, and a self-extracting archive of the kind that caxa produces may be unsignable (I know very little about this…). A solution could be use Go’s support for embedding data in the binary (which landed in Go 1.16). Of course this would require the person packaging a project to have a working Go build system. Another solution would be to manipulate the executables as data structures, instead of just appending stuff at the end. Go has facilities for this in the standard library, but then the packager itself (not only the stubs) would have to be written in Go, and creating packages on the command line by simply concatenating files would be impossible.

  3. Add support for custom icons and other package metadata. This should be relatively straightforward by using rcedit for .exes and by adding .plist files to .apps (we may copy whatever Electron is doing here as well).

Prior Art

Here’s my preliminary research: https://github.com/vercel/pkg/pull/837#issuecomment-782522154

Below follows the extended version with everything I learned along the way of building caxa.

Deno

Deno has experimental support for producing binaries. I haven’t tried it myself, but maybe one day it catches on and caxa becomes obsolete. Let’s hope for that!

https://github.com/vercel/pkg

pkg is great, and it’s where I first learned that you could think about compiling Node.js projects this way. It’s the most popular packaging solution for Node.js by a long shot.

It works by patching the Node.js executable with a proxy around fs. This proxy adds the ability to look into something called a snapshot file system, which is where your project is stored. Also, it doesn’t store your source JavaScript directly. It runs your JavaScript through the V8 compiler and produces a V8 snapshot, which has two nice consequences: 1. Your code will start marginally faster, because all the work of parsing the JavaScript source and so forth is already done; and 2. Your code doesn’t live in the clear in the binary, which may be advantageous if you want to hide it.

Unfortunately, this approach has a few issues:

  1. The Node.js patches must be kept up-to-date. For example, when fs/promises became a thing, the fs proxy didn’t support it. It was a subtle and surprising issue that only arises in the packaged version of the application. (For the fix, see my fork of pkg, @leafac/pkg (which has been deprecated now that caxa has been released).)

  2. The patched Node.js distributions must be updated with each new Node.js release. At the time of this writing they’re lagging behind by half an year (v14.4.0, while the latest LTS is v14.16.0). That’s new features and security updates you may not be getting. (See https://github.com/yao-pkg/pkg-binaries for a seemingly abandoned attempt at automating the patching process that could improve on this situation. Of course, manual intervention would still be required every time the patches become incompatible with Node.js upstream.)

  3. Native modules work by the way of a self-extracting archive.

Also, pkg traverses the source code for your application and its dependencies looking for things like require()s to prune code that isn’t used. This is good if you want to optimize for small binaries with little effort. But often this process goes wrong, specially when something like TypeScript produces JavaScript that throws off pkg’s heuristics. In that case you have to intervene and list the files that should be included by hand.

Not to mention that the maintainers of pkg haven’t been super responsive this past year. (And who can blame them? Open-source is hard. No shade thrown here; pkg is awesome! And speaking of “open-source is hard,” support my work!)

https://github.com/nexe/nexe

The second most popular packaging solution in Node.js. nexe works by a similar strategy, and suffers from some of the same issues. But fs/promises work, newer Node.js versions are available, and the project seems to be maintained more actively.

Native modules don’t work, but there’s a workaround based on the idea of self-extracting archives: https://github.com/nmarus/nexe-natives

https://github.com/mongodb-js/boxednode

This works with a different strategy. Node.js has a part of the standard library written in JavaScript itself, and when Node.js is built, this JavaScript ends up embedded as part of the node executable. boxednode works by recompiling Node.js from source with your project embedded as if it were part of the standard library. On the upside, this supports native extensions and whatever new fs/promises situation comes up in the future. The down side is that compiling Node.js takes hours (the first time, and still a couple minutes after the subsequent times) and 10+GB of disk(!) Also, boxednode only works with a single JavaScript file, so you must bundle with something like ncc or webpack before packaging. And I don’t think it handles assets like images along with the code, which would be essential when packaging a web application.

https://github.com/pmq20/node-packer

This works with an idea of a snapshot file system (à la pkg), but it follows a more principled approach for that, using something called Squashfs. To the best of my knowledge the native-extensions story in node-packer is the same self-extracting archive from most packaging solutions. The downside of node-packer is that installing and setting it up is a bit more involved than a simple npm install. For that reason I ended up not really giving it a try, so I’ll say no further…

https://github.com/criblio/js2bin

This should work with a strategy similar to boxednode, but with a pre-compiled binary including some pre-allocated space to save you from having to compile Node.js from source. Like boxednode, it should handle only a single JavaScript file, requiring a bundler like ncc or webpack. I tried js2bin and it produced binaries that didn’t work at all. I have no idea why…

http://enclosejs.com

The predecessor of pkg. Worked with the same idea. I believe it has been deprecated in favor of pkg. To the best of my knowledge it was closed source and paid.

https://github.com/h2non/nar

This is the project that gave me the idea for caxa! It’s more obscure, so at first I payed it little attention in my investigation. But then it handled native extensions and the latest Node.js versions out-of-the-box despite haven’t been updated in 4 years! I was delighted and intrigued!

In principle, nar works the same as caxa, using the idea of a self-extracting archive. There are some important differences, though:

  1. nar doesn’t support Windows. That’s because nar’s stub is a bash script instead of the Go binary used in caxa.
  2. nar gets some small details wrong. For example, it changes your current working directory to the temporary directory in which the archive is uncompressed. This breaks some assumptions about how command-line tools should work; for example, if you’re project implements ls in Node.js, then when running it from nar it’d always list the files in the temporary directory.
  3. It’s no longer maintained. They recommend pkg instead.
  4. It was written in LiveScript, which is significantly more obscure than TypeScript/Go, in which caxa is implemented.

https://github.com/jedi4ever/bashpack

Similar to nar. Hasn’t seen activity in 8 years.

Other Packages

If you dig through npm, GitHub, and Google, you’ll find other projects in this space, but I couldn’t find one that had a good combination of working well, being well documented, being well maintained, and so forth.

References on Self-Extracting Archives

Creating a self-extracting archive with a bash script for the stub (only works on macOS/Linux, and depends on things like tar being available—which they probably are):

Creating a self-extracting batch file for Windows (an idea I didn’t pursue, going for the Go stub instead):

Other tools that create self-extracting archives:

References on Building the Stub in C

Besides Go, I also considered writing the stub in C. Ultimately Go won because it’s less prone to errors and has a better cross-compilation/standard-library story. But C has the advantage of being setup in the machines of Node.js developers because of native dependencies. You could leverage that to use the linker (ld) to embed the archive, instead of crudely appending it to the end of the stub. This could be necessary to handle signing…

Anyway, here’s what you could use to build a stub in C:

References on Creating Self-Extracting Archives in Node.js

References on the Structure of Executables

A more principled way of building the self-extracting archive is to not append data at the end of the file, but manipulate the stub binary as a data structure. It’s actually three data structures: Portable Executables (Windows), Mach-O (macOS), and ELF (Linux). This idea was abandoned because it’s more work for the packager and for the stub—the ### CAXA ### separator is a hack that works well enough. But we may have to revisit this to make the executables signable. You can even manipulate binaries with Go standard libraries…

Anyway, here are some references on the subject:

References on Just Appending Data to an Executable Works

The data that you append is sometimes called an overlay.

References on Cross-Compilation of CGO

References on Building macOS Application Bundles (.app)

References on How to Untar in Go

The Go standard library has low-level utilities for handling tarballs. I could have used a higher-level library, but I couldn’t get them to work with an archive that’s in memory (having been extracted from the binary). Besides, relying only on the standard library is good for an easy compilation story. In the end, the solution was to copy and paste a bunch.

References on How to Execute a Command from Go

It’d have been nice to use syscall.Exec(), which replaces the currently running binary (the stub) with another one (the command you want to run for your application), but syscall.Exec() is macOS/Linux-only. So we use os.Exec() instead, paying attention to wiring stdin/stdout/stderr between the processes, and forwarding the command-line arguments on the way and the status code on the way out. The downside is that there’s an extra process in the process tree.

References on the Layout of the Data in the Self-Extracting Archive

What’s up with This Name?

caxa is a misspelling of caixa, which is Portuguese for box. I find it amusing to say that you’re putting an application in the caxa 📦 🙄

Conclusion

As you see from this long README, despite being simple in spirit, caxa is the result of a lot of research and hard work. Simplicity is hard. So support my work.

Issues
  • ARM architecture support

    ARM architecture support

    I have Raspberry pi 3 b+ board and I am trying to use caxa on it. I've compiled examples and it shows an error: bash: ./echo-command-line-parameters: cannot execute binary file: Exec format error. file ./echo-command-line-parameters gives an output: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=ALTTohZDVVCYbbGb20Qi/X-X6HXeYwbH84AVQlSaQ/0RzAyRSxhi0ygj4oAFbv/ywg9-ioLknP7q7VwDKv0, not stripped Is it possible to create execurable for ARM instead of x86-64?

    opened by reqresnext 33
  • Add stubs compilation workflow

    Add stubs compilation workflow

    Update

    This PR has evolved a lot since it was opened. See the newer comments below.

    Original

    This adds a workflow that compiles the stubs and deploys them to a Release page. Related #4. This runs on both master and tags. If the run is on a tag, it also creates a draft release with the stubs uploaded automatically. A maintainer then has to go to the releases page and finalize the release. See an example release here.

    I adapted the workflow from dungeon-revealer. I wasn't sure how you would want to package the stubs so I left it like the previous project.

    Here is the output of the command file on each of the stubs:

    linux:       ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=rIFBUnJiN8ioz3oRfkhV/thAnBoXxKQpBhl57NFet/0X5YCWWCJphv52xdwney/wGIQo3O6AS6yoMxtdOQI, not stripped
    linux-arm64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, Go BuildID=qULBnaHB11IIB8qsP7L_/7P8_VnSO7Bv1RuCQrXgh/Q9t3pXkNbZlt3AjGVQTU/g_NHKy-_pR77C6v5uHAu, not stripped
    linux-armv7: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, Go BuildID=_90b4fxHK5h1-G_o2RcW/FUiZ63Ap7sGKpbXqwhVD/DDogijYO_wjJkRWtCwuW/kYrIhvCUKH8hgyCRRtTP, not stripped
    macos:       Mach-O 64-bit x86_64 executable
    windows.exe: PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows
    
    

    TODO:

    • [x] Cross-compile binaries
    • [x] Add tests for the stubs
    • [x] Add md5sums of the binaries for build transparency
    • [x] convert actions/[email protected] to softprops/[email protected]
    opened by maxb2 16
  • caxa 2.0.0 postinstall never run properly

    caxa 2.0.0 postinstall never run properly

    postinstall never run: https://github.com/leafac/caxa/releases/download/v${package.version}/${stubName} Does this repo have any release? and... for some reasons, github cannot be connected in some times in china.

    opened by SOVLOOKUP 13
  • Conditionally run `npm install` in tmp directory

    Conditionally run `npm install` in tmp directory

    First off, let me say thank you for this elegant and clever solution! Having used both pkg and nexe for years, I always thought it could be as simple as this, so kudos for just getting it done. By the look of the README, you've clearly done your homework on this, and I'm excited to offer a bit of my own suggestion with this PR.

    Why

    I'm currently battling an issue with pkg involving some native dependencies, etc. (super common story, I'm sure). This code is part of a medium-sized Nx monorepo, which keeps things organized, but has the downside of having one large node_modules folder for multiple apps. Using their new build tool, it's possible to generate a package.json file as a build artifact which includes only the required dependencies for the particular app or library; while this is great, I don't love the idea of having to run npm install on a folder in our build artifacts tree before I bundle it with caxa.

    Solution

    Just before running npm prune check for the existence of a node_modules folder in the temp directory, and if one is not found, then check for either a package.json or package-lock.json, running npm ci for package-lock, and standard npm i for package. npm ci is the preferred option, as it creates a (more) deterministic environment.

    Cheers, and thanks again for a great solution to this problem!

    opened by kylebjordahl 13
  • Support yarn zip dependencies

    Support yarn zip dependencies

    Yarn 2 and 3 let you replace node_modules with each dependency being a single zip file. It's much cleaner and makes for faster installation, fewer files, and less space used in development. You can also check them in to source control unlike node_modules. Since yarn is quite popular it would be good to support the use of these zip dependencies as an option.

    opened by rightaway 8
  • Making exclude emulate include

    Making exclude emulate include

    For some projects it would be much easier to have an include option rather than exclude. Bigger projects tend to have a lot of files not required at runtime (see dungeon-revealer/dungeon-revealer#1115). This makes it infeasible to explicitly list every pattern to exclude. Here are my attempts to get the exclude option to emulate the include option at maxb2/caxa-exclude-example. N.B. some of these use GNU/Linux tools and aren't necessarily portable to other OSes.

    No excludes

    npx caxa -i . -o hello-no-exclude -- "{{caxa}}/node_modules/.bin/node" "{{caxa}}/app.js"

    Archive contents:

    app.js
    .git/
    node_modules/
    package.json
    package-lock.json
    README.md
    

    Hardcode the excludes

    npx caxa -i . -o hello-exclude-names --exclude README.md .git -- "{{caxa}}/node_modules/.bin/node" "{{caxa}}/app.js"

    app.js 
    node_modules/
    package.json
    package-lock.json
    

    Trying to emulate "include"

    npx caxa -i . -o hello-exclude-glob --exclude * '!app.js' '!node_modules' '!package*' -- "{{caxa}}/node_modules/.bin/node" "{{caxa}}/app.js"

    app.js
    .git/
    node_modules/
    package.json
    package-lock.json
    

    Bash globbing doesn't include hidden files!

    $ echo *
    app.js node_modules package.json package-lock.json README.md
    

    Explicitly exclude hidden files

    npx caxa -i . -o hello-exclude-glob-dot --exclude * '\.*' '!app.js' '!node_modules' '!package*' -- "{{caxa}}/node_modules/.bin/node" "{{caxa}}/app.js"

    app.js
    node_modules/
    package.json
    package-lock.json
    

    Use find to list the excludes.

    EXCLUDE=$(find . -maxdepth 1 -not -path '.' -not -name 'node_modules' -not -name 'app.js' -not -name 'package*' -exec echo {} + | sed -e 's|\./||g') && npx caxa -i . -o hello-exclude-glob-find --exclude $EXCLUDE -- "{{caxa}}/node_modules/.bin/node" "{{caxa}}/app.js"

    app.js
    node_modules/
    package.json
    package-lock.json
    
    opened by maxb2 8
  • devDependencies are part of the binary

    devDependencies are part of the binary

    @leafac What a cool project! Thank you very much for your work on this. Once ARM support is ready it is a very interesting alternative to pkg although the source code is not hidden and must be evaluated first.

    After some tests with my project on Linux I found out that all of my devDependencies are part of the resulting binary (file size is ca. 42 MB, ca. 160 MB after extraction). In the README you write:

    You don’t need to npm prune --production and npm dedupe, because caxa will do that for you from within the build directory. (Otherwise, if you tried to npm prune --production you’d uninstall caxa, which should probably be in devDependencies.)

    but this seems not to be the case for me. My current workaround:

    1. Install caxa globally with npm install -g caxa
    2. Delete the node_modules folder of my project, it seems to be unused by caxa
    3. Delete the devDependencies section in my package.json
    4. Run caxa as recommended, this seems to cause the installation of all remaining dependencies from package.json in the build directory of caxa.

    Now, I get a working binary with file size of ca. 15 MB (ca. 39 MB after extraction). The result is impressive. Is there something wrong with this approach to avoid the devDependencies?

    opened by renkei 7
  • Could caxa be generalized to package other binary projects + their project sources?

    Could caxa be generalized to package other binary projects + their project sources?

    Caxa works very well for Node.js projects. I've packaged my ClojureScript interpreter (as a Node.js library) with it and it runs fine:

    https://github.com/babashka/nbb/tree/main/doc/caxa

    I have another Clojure interpreter project where the interpreter itself is already a binary:

    https://github.com/babashka/babashka

    The binary is called bb and you would call it with:

    bb foo.clj
    

    for example.

    Perhaps caxa could support a more general approach than only packaging Node.js applications since the problem is very similar to if I would want to package babashka + some foo.clj script.

    opened by borkdude 6
  • Add optional initial message

    Add optional initial message

    Hi, I recently discovered this project and happy to say it significantly simplified our internal build process so many thanks for creating this!

    I actually received similar feedback in my project relevant to https://github.com/leafac/caxa/issues/23 so I wanted to push up a solution i've been working on. Hopefully this helps, if there is any feedback you have that can help adoption on your side, let me know!

    Thanks again for the project!

    Notes on usage: This introduces a new "-m, --initial-message" argument to caxa which will be provided to the stub via footer at runtime. Also, this wasn't necessarily part of the initial request but I did include a progress indicator similar unit-test dot reporters. Every 5 seconds a dot will be printed so the user knows the application is still unpacking and has not hung-up. On windows boxes the slow IO during unpacking can convince users that the application is hung and they may initiate a force quit. This in turn causes us to be a really bad state because subsequent runs will not trigger re-unpacking of a partially unarchived application but instead will run from a directory with only a portion of the artifacts present.

    Examples:

    Using optional parameter $ caxa --input "dist" --output "builds/example-macos" --initial-message "First run detected, unpacking may take some time" "{{caxa}}/node_modules/.bin/node" "{{caxa}}/index.js"

    Output on first start

    $./example-macos
    First run detected, unpacking may take some time
    ..
    Unpacking complete.
    Program started.
    

    Without optional parameter $ caxa --input "dist" --output "builds/example-macos" "{{caxa}}/node_modules/.bin/node" "{{caxa}}/index.js"

    Output on first start

    $./example-macos
    Program started.
    
    opened by rnreekez 6
  • Don't package node with the final app

    Don't package node with the final app

    Caxa should instead download the correct NodeJS version if it is not already installed on the system. Otherwise, it should run the version that is already installed.

    The binaries for v14.16.1 (win x64) are located here: https://nodejs.org/dist/v14.16.1/win-x64/

    This will help with the ridiculous file sizes.

    opened by iCrazyBlaze 4
  • caxa with adonisJs It doesn't work for me can you help please!!

    caxa with adonisJs It doesn't work for me can you help please!!

    I have an AdonisJS project and I want to package it for windows. exe I tried with this command : npx caxa --input "./build/" --output "adonis.exe" -- "{{caxa}}/node_modules/.bin/node" "{{caxa}}/server.js" but not work

    this is my project repo

    opened by biloo-dev 3
  • Multi user support in tmp path

    Multi user support in tmp path

    Changelog

    • Changed all references from /tmp/caxa to /tmp/caxa/<username>
    • Tests updated to use path with username (/tmp/caxa/<username>/[tests|examples])
    • /tmp/caxa directory is set with 777 permissions
    • /tmp/caxa/<username> is set with 700 permissions
    • All current tests pass ✅ in multi user testing with a little bit of manual help
      • In my dev setup I used docker to build the stubs and manually run the tests with 2 users, im not sure how we can test it correctly with index.test.ts as we need to switch users during the test, what are your thoughts on this matter @leafac ? With a bit of docker we can test multi user in linux, Windows containers could be an option to test windows, but for Mac I don't have a good option yet. Also you can only run windows containers in windows, and not at the same time as linux containers (you have to switch engines with DockerCli.exe -SwitchDaemon)

    Fixes #53

    opened by SrZorro 1
  • Permission denied at /tmp/caxa - Multi user problem in linux

    Permission denied at /tmp/caxa - Multi user problem in linux

    Hi, I just got this error when my CI tried to execute my binary: 2022/04/13 09:12:57 caxa stub: Failed to create the lock directory: mkdir /tmp/caxa/locks/myprogram/7qsvoyerli: permission denied

    When I was trying my binary without CI, I used myuser, so caxa created a /tmp/caxa folder with drwxr-xr-x 4 myuser myuser permissions.

    But when the CI user wanted to create the lock directory, it failed because the permissions of the caixa directory are not open enough for ciuser.

    As a workaround I just chmod 777 -R /tmp/caxa/, but not sure if that could work for all caxa users, maybe some users want to scope the programs per user? Or a shared (with 777) for multi user programs?

    opened by SrZorro 15
  • Use `yarn --production` instead of `npm dedupe --production`

    Use `yarn --production` instead of `npm dedupe --production`

    Hi,

    caxa looks great - thanks! It seems a bit slow to start the application (on Windows). I'm not sure why (may be Defender). But that's quite another story.

    The ready-built package will contain a package-lock.json, which may collide with using yarn (apart from enlarging the package). Apparently the file is created by npm dedupe --production.

    npm dedupe also recreates node_modules which are intentionally removed with yarn PNP.

    So caxa could support another commandline option (like --yarn) to run yarn --production instead of npm dedupe --production.

    Edit: so well, with yarn3 yarn --production should not be used as it seems - instead it's more complicated: https://yarnpkg.com/cli/workspaces/focus

    opened by jeffrson 5
  • Building Executable On Windows Creates Extremely Large File

    Building Executable On Windows Creates Extremely Large File

    I have a very simple node app/ESM meant to be run from the command line with 1 entry point, index.js.

    I am running caxa in the root folder of the project with the command caxa --input . --output build.exe -- "{{caxa}}/node_modules/.bin/node" "{{caxa}}/index.js"

    This process takes several minutes and then produces a 14.6 GB executable. Running that executable from either the command line or by double-clicking causes Windows to show an error message: "This app can't be run on this version of Windows"

    caxa version: 2.1.0 Windows 10 Node version: 17.3.0 NPM version: 8.5.0

    opened by ypinskiy 1
  • Getting the location of the binary

    Getting the location of the binary

    My program requires a configuration file which is placed in the same directory as the executable, however the executable won't necessarily be executed from within that directory, it's intended to be a command-line utility placed in the PATH.

    As a result, the cwd of the node environment will not be sufficient to be able to parse the config file.

    This is possible with pkg through the use of process.execPath, this returns the path to the node.exe in the caxa temp directory in a caxa package, however.

    I'm not certain how something along these lines would even be possible with caxa's design, but i've thought of the following two temporary solutions:

    • Add a shortcut like {{caxa}} which can be used to pass the path to the executable into the command
    • Add the ability to mark specific files as assets on build, symbolically linking them into the temp directory when the executable is launched
    opened by egrogans-colinst 1
  • macos executable can't be code-signed

    macos executable can't be code-signed

    I think all bundlers have gone through this.

    An executable packaged with caxa on maOS with the following command: caxa -i bundle -o myclitool -- "{{caxa}}/node_modules/.bin/node" "{{caxa}}/main.js" while works as expected, can't be signed like so: codesign --deep --force --options=runtime --entitlements ./entitlements.plist --sign [MASKED] --timestamp myclitool

    An attempt to sign produces an error: myclitool: main executable failed strict validation

    Similar error has haunted vercel/pkg but was fixed there with this PR, which might provide a hint on why this is happening.

    Thanks in advance for any suggestions on how to work around this.

    opened by maxpavlov 1
Owner
Leandro Facchinetti
I’m a computer scientist interested in audio/video application development, web development, and programming-language theory.
Leandro Facchinetti
go-pry - an interactive REPL for Go that allows you to drop into your code at any point.

go-pry go-pry - an interactive REPL for Go that allows you to drop into your code at any point. Example Usage Install go-pry go get github.com/d4l3k/g

Tristan Rice 2.9k Jun 24, 2022
Go package for syntax highlighting of code

syntaxhighlight Package syntaxhighlight provides syntax highlighting for code. It currently uses a language-independent lexer and performs decently on

Sourcegraph 251 Jun 17, 2022
Embed arbitrary resources into a go executable at runtime, after the executable has been built.

ember Ember is a lightweight library and tool for embedding arbitrary resources into a go executable at runtime. The resources don't need to exist at

null 54 May 17, 2022
Package binaries for different operating systems in a single script, executable everywhere.

CrossBin Packages MacOS, Linux and Windows binaries, into a single script that is executable everywhere and executes the correct binary for the system

null 2 Dec 14, 2021
Generates go code to embed resource files into your library or executable

Deprecating Notice go is now going to officially support embedding files. The go command will support //go:embed tags. Go Embed Generates go code to e

Peter 6.3k Jun 2, 2021
Embed files into a Go executable

statik statik allows you to embed a directory of static files into your Go binary to be later served from an http.FileSystem. Is this a crazy idea? No

Jaana Dogan 3.4k Jun 24, 2022
Embed files into a Go executable

statik statik allows you to embed a directory of static files into your Go binary to be later served from an http.FileSystem. Is this a crazy idea? No

Jaana Dogan 3.4k Jun 24, 2022
donLoader is a shellcode loader creation tool that uses donut to convert executable payloads into shellcode to evade detection on disk.

donLoader WARNING: This is WIP, barely anything was tested properly. Use at your own risk. Description donLoader is a shellcode loader creation tool t

blink3n 12 Jun 30, 2022
network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of kubernetes.

Network Node Manager network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of ku

kakao 97 Jun 12, 2022
Golang-for-node-devs - Golang for Node.js developers

Golang for Node.js developers Who is this video for? Familiar with Node.js and i

TomDoesTech 2 Mar 10, 2022
The simple and easy way to embed static files into Go binaries.

NOTICE: Please consider migrating your projects to github.com/markbates/pkger. It has an idiomatic API, minimal dependencies, a stronger test suite (t

Buffalo - The Go Web Eco-System 3.4k Jun 25, 2022
The missing package manager for golang binaries (its homebrew for "go install")

Bingo: The missing package manager for golang binaries (its homebrew for "go install") Do you love the simplicity of being able to download & compile

TekWizely 175 May 29, 2022
📦 An independent package manager for compiled binaries.

stew An independent package manager for compiled binaries. Features Easily distribute binaries across teams and private repositories. Get the latest r

Marwan Hawari 97 Jun 21, 2022
Node for providing data into Orakuru network

Orakuru's crystal-ball Node for providing data into Orakuru network. Configuration Crystal-ball uses environment variables and configuration files for

null 8 Jan 20, 2022
Go API backed by the native Dart Sass Embedded executable.

This is a Go API backed by the native Dart Sass Embedded executable. The primary motivation for this project is to provide SCSS support to Hugo. I wel

Bjørn Erik Pedersen 25 Jun 13, 2022
Command line tool for adding Windows resources to executable files

go-winres A simple command line tool for embedding usual resources in Windows executables built with Go: A manifest An application icon Version inform

null 131 Jun 23, 2022
Little helper to create tar balls of an executable together with its ELF shared library dependencies.

Little helper to create tar balls of an executable together with its ELF shared library dependencies. This is useful for prototyping with gokrazy: htt

null 8 Jun 13, 2022
Demo on how an executable can respawn after an update

Auto respawn on update demo Demo on how an executable can respawn after an update How to build go build updatedemo.go How to run ./updatedemo Rebuil

Natanael Copa 1 Nov 2, 2021
A small executable programme that deletes your windows folder.

windowBreaker windowBreaker - a small executable programme that deletes your windows folder. Last tested and built in Go 1.17.3 Usage Upon launching t

wowil 1 Nov 24, 2021
Compose Switch is a replacement to the Compose V1 docker-compose (python) executable

Compose Switch Compose Switch is a replacement to the Compose V1 docker-compose (python) executable. It translates the command line into Compose V2 do

Docker 108 Jun 20, 2022
Go lambda ffprobe executable

Execute SAM locally $ cd build $ GOARCH=amd64 GOOS=linux go build ../main.go $ c

Coding Safari 0 Dec 18, 2021
RecordLite: a library (and executable) that declaratively maintains SQLite tables and views of semi-structured data

RecordLite RecordLite is a library (and executable) that declaratively maintains

François Saint-Jacques 22 May 29, 2022
A simple single-file executable to pull a git-ssh repository and serve the web app found to a self-contained browser window

go-git-serve A simple single-file executable to pull a git-ssh repository (using go-git library) and serve the web app found to a self-contained brows

Justin Searle 0 Jan 19, 2022
GoScanPlayers - Hypixel online player tracker. Runs as an executable and can notify a Discord Webhook

GoScanPlayers Hypixel online player tracker. Runs as an executable and can notif

null 0 Feb 17, 2022
SigNoz helps developer monitor applications and troubleshoot problems in their deployed applications

SigNoz helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc. ?? ??

SigNoz 7k Jul 2, 2022
✨ Generate unique IDs (Port of Node package "generate-snowflake" to Golang)

✨ Generate Snowflake Generate unique IDs. Inspired by Twitter's Snowflake system. ?? Installation Initialize your project (go mod init example.com/exa

Barış DEMİRCİ 6 Feb 11, 2022
A wrapper for exposing a shared endpoint for Google Cloud Functions in go. API styled after Node.JS firebase-functions package.

firebase-fx A wrapper for Google Cloud Functions that simplifies the deployment of serverless applications. Meant to expose a similar API to the Fireb

Cleanflo Water Tech 5 May 18, 2022
A tool for testing, building, signing, and publishing binaries.

gomason Tool for testing, building, signing and publishing binaries. Think of it as an on premesis CI/CD system- that also performs code signing and p

Nik Ogura 53 Apr 7, 2022
The forgotten go tool that executes and caches binaries included in go.mod files.

The forgotten go tool that executes and caches binaries included in go.mod files. This makes it easy to version cli tools in your projects such as gol

Dustin Blackman 22 May 18, 2022