shards

shards copied to clipboard

Should shards be used as a build system manager in the future - much like Rust's cargo, node.js's npm, or Java's maven?

I can't find anything about building with NPM... I suppose you want shards to become aware of targets to build, downloading dependencies, building the targets, and maybe installing them?

It may be a little out of scope, at least for the time being. More often than not a simple Makefile is usually enough.

What would be your use cases? And what kind of problems should it be solving?

I don't know much about Cargo's building features nor Maven's, and I only skimmed rebar or the Haskell build tool (Cabal?).

I think it becomes interesting to define a build part for shards when you can actually define executables that can be build.

In that scenario, having different options for different targets (like cross-compilation, compiler or linker options) become necessary.

However I feel that Crystal still lacks a bit of maturity in relation to cross compilation (ie, it always uses CC so cross compilation requires you prepend CC=xxx to the build command).

I could see adding the ability to define arbitrary groups as being useful. Something akin to a Gemfile in ruby. Then a makefile etc.. could build the executable the appropriate groups enabled. I only gave the shards yml SPEC a cursory glance, but it appears to only allow dependencies and development_dependencies. I could see the possible need for test, production etc... groups as well, but there is no need to limit it to just those. It would be neat if any type of group could be named in the shards yml. precompile, assets, docker etc.. There are many possibilities.

The more I think about it, the more I like the idea. I'm not saying I will work on this anytime soon, but I believe this is a good development for Crystal.

An example use case (thinking out loud). Crystal's source could follow the Git scheme and build multiple binaries like crystal-build crystal-docs, and so on for each subcommand. A Shards SPEC could contain something like this:

targets:
  crystal-build:
    main: src/crystal/commands/build.cr
  crystal-docs:
    main: src/crystal/commands/docs.cr

Then:

$ shards build [all]
$ shards build docs [--force]

• The build command would resolve/install dependencies then build the specified targets (or everything). Actually it could even parse the files for require statements, look for missing dependencies, and verify if we actually need to build the binary, or not (with a flag to force a rebuild).
• Each target could have different helpers; for example cc: musl-gcc, release, debug, ...
• Maybe a global $HOME/.shard.yml (for example) for global settings.

I was just about to submit this same request. One language that I love is Typescript which has a file called tsconfig.json that allows you to specify various compiler settings which you can also modify using command line flags. This is a powerful feature because editors can also read the file to get a better idea of what your build actually looks like.

This would make a world of difference with the vscode plugin which, right now, throws all kinds of errors when attempting to work on crystal itself because of things being redefined.

We would no longer need Makefiles!

Here is what my brain came up with as an example shard.yml for crystal itself.

name: crystal
version: 0.19.4
description: The Crystal Programming Language http://crystal-lang.org 

authors:
  - Ary Borenszweig <[email protected]>
  - Juan Wajnerman <[email protected]>

license: Apache

targets:
  build:
    crystal: .build/crystal
    main: src/compiler/crystal.cr
    flags:
      - without_openssl
      - without_zlib

Now available in master.

I feel that Crystal still lacks a bit of maturity in relation to cross compilation

Actually, I think that Crystal is quite good at cross compilation, thanks to LLVM. At least it's easy to build an object file for whatever OS/arch combo. No need to deal with different compilers for different targets, just specify --target.

The problem is linking the generated object file. It's complex, and sometimes impossible (eg: link a macOS binary from Linux). It requires a toolchain and libraries for the target architecture, that must be specified with the CC and CXX environment variables and the --link-flags argument, or use --cross-compile then link manually. There ain't much we can do to improve that, though.

The build command was released in v0.7.0. I'm keeping this issue open, since it's still very basic (on purpose) and it should improve. I have some ideas from my work on Android targets, which involve cross compilation for different targets with specific toolchains.

The following features could be useful:

• specify a target path (defaulting to bin when missing);

• specify some arguments (eg: --cross-compile --target="arm-unknown-linux-androideabi");

• specify environment variables (eg: CC=toolchains/arm-androideabi/bin/clang);

• MAYBE: specify a manual link command (?) when we don't want to build an executable binary but a library (required for Android) or want/need to link in a slow VM (eg: AArch64 in qemu). Shards would set --cross-compile to the build, and add a few args to the link command, for example the object file to link, -o <target>, maybe the -l definitions (Crystal could print a JSON file instead of a linker command) or even -shared and -Wl,-soname,lib<name>.so for shared libraries.

For example:

targets:
  obj/local/armeabi/foo.so:
    main: src/foo.cr
    args: --target=arm-unknown-linux-androideabi
    link: toolchains/armeabi/bin/clang --sysroot=toolchains/armeabi/sysroot -Lobj/local/armeabi

  obj/local/armeabi-v7a/foo.so:
    main: src/foo.cr
    args: --target=arm-unknown-linux-androideabi --mattr=+armv7-a+vfp3
    link: toolchains/armeabi/bin/clang --sysroot=toolchains/armeabi/sysroot -Lobj/local/armeabi-v7a

  obj/local/arm64-v8a/foo.so:
    main: src/foo.cr
    args: --target=aarch64-unknown-linux-android
    link: toolchains/arm64-v8a/bin/clang --sysroot=toolchains/arm64-v8a/sysroot -Lobj/local/arm64-v8a

I am missing some stuff. In my first Crystal project I need to use makefiles to manage compilation.

1. npm have scripts in package.json that are arbitrary bash commands including locally installed executable. For example in webpack project:

  "scripts": {
    "test": "mocha --harmony --check-leaks",
    "travis:test": "npm run cover -- --report lcovonly",
    "travis:lint": "npm run lint-files && npm run nsp",
    "appveyor": "node --max_old_space_size=4096 node_modules\\mocha\\bin\\mocha --harmony",
    "build:examples": "cd examples && node buildAll.js",
    "pretest": "npm run lint-files",
    "lint-files": "npm run lint && npm run beautify-lint",
    "lint": "eslint lib bin hot",
    "beautify-lint": "beautify-lint 'lib/**/*.js' 'hot/**/*.js' 'bin/**/*.js' 'benchmark/*.js' 'test/*.js'",
    "nsp": "nsp check --output summary",
    "cover": "node --harmony ./node_modules/.bin/istanbul cover -x '**/*.runtime.js' node_modules/mocha/bin/_mocha",
    "publish-patch": "npm run lint && npm run beautify-lint && mocha && npm version patch && git push && git push --tags && npm publish"
  }

Then you can use npm run

2. in sbt you have repl where you can run issue commands. There is few interesting things on having interactive repl like that.

3. watch and compilation on changes should be included.

don't mind me, opinion is like ass everyone has one. :p

In my opinion, using makefiles instead of pushing the complexity to shards is the correct solution. Why reinvent the wheel when make works so well?

Yes, please use a regular Makefile to run arbitrary commands with dependencies. It's simpler and doesn't have to be reimplemented.

The problem with Makefiles is that they leaks project dependencies to environment. When moving project to new box/developer I need to have additional tooling and platform specific install script for Makefile dependencies. I would love if crystal projects are self-contained. This would make docker images easier to create and easier to have cross-platform projects and will not force people to learn make...

Ideally you should only clone project, run shards to install deps and start build with watch->compile. This is typical workflow in npm projects. For example: https://github.com/mxstbr/react-boilerplate

I am not necessary suggesting adding arbitrary shell scripting like npm, I feel it is a bit messy. I would like solution similar to sbt where continuous build+test is achieved using: ~compile http://www.scala-sbt.org/0.12.4/docs/Getting-Started/Running.html

Sorry, I want to avoid duplicating functionalities that are standardised, already available on all platforms and most likely already installed in developer boxes, or just a small package away.

what is standardised way to watch for file changes?

I am using watchman from facebook.

I find entr to be a really nice unixy file watcher tool.

@Chyzwar are you watching for file changes to run arbitrary tasks or just to restart your app. If its the later I created Sentry to solve this problem—influenced by Nodemon in Node.js.

In my view, combining build tool and dependency resolution has lead to lots of scope creep for say Maven in the Java world. Keeping them separate might avoid that.

Should shards be used as a build system manager in the future - much like Rust's cargo, node.js's npm, or Java's maven?

Please keep it simple, do one thing and do it well! Cargo is horribly designed, it’s everything-but-kitchen-sink bloatware. And the worst about it is that you need Cargo and Rust to build Cargo and you need Cargo to build Rust, so double chicken-or-egg problem with bootstrapping. Please, please, do not make the same mistake!

It’s better to keep it as separate tools and just call each other as external commands.

I’m author and maintainer of Rust (and Crystal) package in Alpine Linux. First I was really excited from Rust and wanted to bring it to Alpine and spent a lot of time on it. This experience from the distribution PoV eventually completely changed my relation to Rust. Now I’m very frustrated from Rust, even thinking about giving it up (maintaining rust/cargo package), mostly because of Cargo.

Maven and npm are even worse examples for inspiration.

@jirutka I totally agree about having one job and doing it well, but i'm not familiar with rust and cargo. What mistakes do rust/cargo make which hurt distro packagers, so that we don't accidentally make them?

Shards has already resisted becoming a system package manager (it only ever manages packages inside the current directory, no npm install -g to abuse) and a larger build tool. It currently provides only small shortcuts using shards build (see #136).

So... I've seen some people recommend using makefiles, but what is the recommended way to handle Crystal dependencies in makefiles? This is the best I've come up with:

.PHONY: all libs

PROGRAM = do_something
LIBS = lib/one_library lib/another_one

SOURCE_FILES := $(shell find source/ -type f)
LIB_FILES := $(foreach dir,$(LIBS),$(shell find $(dir) -type f 2> /dev/null))

all: $(PROGRAM)

$(PROGRAM): %: source/%.cr $(SOURCE_FILES) $(LIBS) $(LIB_FILES)
	crystal build -o $@ $<

# The phony libs target should only be run if the dependencies need
# to be installed - otherwise, the program will always be rebuilt.
ifneq ($(shell crystal deps check > /dev/null 2> /dev/null; echo $$?),0)
$(LIBS): libs
endif

libs:
	crystal deps install

It feels hacky, but I can't come up with a way to dynamically get the dependencies from shards.yml / shards.lock or another way to only install them when crystal deps check returns 1.

I'd do something like:

CRYSTAL_FILES := $(shell find src lib -type f -name '*.cr)

bin/my_program: $(CRYSTAL_FILES) lib
	crystal build -o $@ src/entrypoint.cr

lib: shards.yml shards.lock
	shards install
	touch lib

That's a pretty robust makefile because the dependency isn't on "crystal deps check returns 1" it's on whether shards.yml or shards.lock have been updated. And yes, you can depend on directories in make, as long as you ensure their mtime is updated. You might not even need the touch lib to update the directory mtime, but it's there just in case (you can try without it and see if you get any false shards install triggers). In fact, shards could probably update the lib directory mtime when it's sure that the lib directory is up to date. Then you could just add the above shards install recipe to every makefile and be sure it works without touch.

Oh, that's not too bad. Thanks.

Using that solution, it won't rebuild if you remove any of the libraries without removing the lib directory, but I guess I can't really come up with a reason you'd ever do that...?

@obskyr if you delete anything in the lib directory, lib's mtime will be updated. If you delete anything inside say lib/myshard/ then it won't trigger because only the mtime for lib/myshard directory will be updated (not lib). That's why I'm actually pretty sure it won't currently work without touch, unless shards always deletes and recreates the whole lib/shard directory every time it updates a version.

crystal-lang/crystal#11481 might eventually provide a way to discover true dependencies, including e.g. non-source files loaded from macros, and emit depfiles. Until then the $(shell find) approach is probably the most common way.

A drawback here is that $(shell find) is inherently platform-dependent; a Windows equivalent is $(shell dir src /B /S). Ideally we want to be able to do that on Windows with as few external program dependencies as possible, i.e. require only the build tools plus GNU Make, and not a full Unix-compatible development environment (GitHub's Windows CI runners for example expose the whole bin directory from their Git Bash in %PATH%, which is undesirable). So one may write:

# unix-like
GLOB = $(shell find $1 -type f -name $2)

# windows
GLOB = $(shell dir $1\\$2 /B /S)

SOURCES := $(call GLOB,src,*.cr)
LIB_SOURCES := $(call GLOB,lib,*.cr)

I use this Makefile template in Crystal projects: https://gist.github.com/straight-shoota/275685fcb8187062208c0871318c4a23

Some minimal amount of C-compiler step support would be helpful to make non-shared-object C-bindings more accessible as shards.

Some examples from tree-sitter/tree-sitter-javascript:

Rust has a build.rs with support for invoking the C compiler:

fn build() {
    // Paths are relative to `Cargo.toml` by default (i.e. `shard.yml` for Crystal)
    let mut c_config = cc::Build::new();
    c_config.std("c11").include("src");
    c_config.file("src/parser.c")
    c_config.file("src/scanner.c");
    c_config.compile("tree-sitter-javascript");
}

While Go offers an alternate Annotation-like approach, that wouldn't need any support from shards, but could instead be added to Crystal as a built-in annotation:

package tree_sitter_javascript

// #cgo CFLAGS: -std=c11 -fPIC
// #include "../../src/parser.c"
// #include "../../src/scanner.c"
import "C"

Hypothetical in crystal:

@[CC(std: "c11", include: "#{__DIR__}/../../src/parser.c, #{__DIR__}/../../src/scanner.c")]
lib LibTreeSitterJavascript
   # ... define extern types as usual ...
end