Add Plugins system support

[damon-kwok]

trafficstars

Summary

The plug-in system is similar to a set of callbacks that allow Corral to trigger and run custom tasks as it processes commands.

Motivation

• Provide expansion capability for Corral.
• Users can contribute Corral without changing corral code.

Detailed design

• The plug-in system is designed to support cross-platform
• All plug-ins must support cross-platform, which needs to be checked automatically before merging PR.

Your new plug-in would look like:

| corral
├── plugins
│   ├── example
│       ├── INSTALL     # plugin installer
│       ├── COMMAND     # (optional) run user command
│       ├── on_create   # (optional)
│       ├── on_fetch    # (optional)
│       ├── on_update   # (optional)
│       ├── on_build    # (optional)
│       ├── on_run      # (optional)
│       ├── on_clean    # (optional)
│       └── on_deploy   # (optional)

For windows:

MSYS2 needs to be built for windows

[damon-kwok]

Example 1: Write a statistic plugin to counts code lines at build time

| corral
├── plugins
│   └── vcpkg
│       ├── INSTALL
│       └── on_build

INSTALL:

curl -L "https://statistic.sh" | bash

on_build:

statistic *.pony ${project_root}

use statistic:

corral run -- ponyc

output:

pony        11567
markdown    115
sh          76

[damon-kwok]

Example 2: Write a vcpkg plugin to handle cross-platform native deps #142

| corral
├── plugins
│   └── vcpkg
│       ├── INSTALL
│       └── COMMAND

INSTALL:

  github_deps  microsoft/vcpkg $CORRAL_REPO/vcpkg
  cd $CORRAL_REPO/vcpkg
  chmod +x ./bootstrap-vcpkg.sh
  if [ "$OS_TYPE" = "cygwin" ] || [ "$OS_TYPE" = "msys2" ] || [ "$OS_TYPE" = "winnt" ]; then
    ./bootstrap-vcpkg.bat
  else
    ./bootstrap-vcpkg.sh
  fi
  vcpkg_setenv

COMMAND:

#!/usr/bin/env sh

vcpkg $@

Use vcpkg plugin

corral.json: (NOTE, Event: #144 )

{
  "plugins": ["vcpkg"],
  "deps": [
    {
      "locator": "github.com/ponylang/valbytes.git",
      "version": "0.5.0"
    }
  ],
  "info": {}, 
  "event": [      
    {
      "on_fetch": "vcpkg install openssl",
    }
  ]
}

use openssl in pony:

use "lib:crypto"
use "lib:ssl"

Test

corral fetch

output:

Computing installation plan...
The following packages will be built and installed:
    openssl[core]:x64-linux
  * openssl-unix[core]:x64-linux
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet x64-linux...
Starting package 1/2: openssl-unix:x64-linux
Building package openssl-unix[core]:x64-linux...
Could not locate cached archive: /home/damon/.cache/vcpkg/archives/a2/a2158e014fef4ae1cf65b8c0b9dff1aabb9e494f.zip
-- Downloading https://www.openssl.org/source/openssl-1.1.1g.tar.gz...
-- Extracting source /home/damon/.ome_local/repo/vcpkg/downloads/openssl-1.1.1g.tar.gz
-- Using source at /home/damon/.ome_local/repo/vcpkg/buildtrees/openssl-unix/src/1.1.1g-bb2a9a5801.clean
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/aes.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/asn1.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/asn1_mac.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/asn1err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/asn1t.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/async.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/asyncerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/bio.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/bioerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/blowfish.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/bn.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/bnerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/buffer.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/buffererr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/camellia.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cast.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cmac.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cms.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cmserr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/comp.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/comperr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/conf.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/conf_api.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/conferr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/crypto.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cryptoerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ct.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cterr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/des.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/dh.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/dherr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/dsa.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/dsaerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/dtls1.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/e_os2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ebcdic.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ec.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ecdh.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ecdsa.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ecerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/engine.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/engineerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/evp.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/evperr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/hmac.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/idea.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/kdf.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/kdferr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/lhash.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/md2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/md4.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/md5.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/mdc2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/modes.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/obj_mac.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/objects.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/objectserr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ocsp.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ocsperr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/opensslconf.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/opensslv.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ossl_typ.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pem.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pem2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pemerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pkcs12.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pkcs12err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pkcs7.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pkcs7err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rand.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rand_drbg.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/randerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rc2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rc4.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rc5.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ripemd.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rsa.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rsaerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/safestack.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/seed.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/sha.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/srp.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/srtp.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ssl.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ssl2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ssl3.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/sslerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/stack.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/store.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/storeerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/symhacks.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/tls1.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ts.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/tserr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/txt_db.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ui.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/uierr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/whrlpool.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/x509.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/x509_vfy.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/x509err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/x509v3.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/x509v3err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/share/openssl-unix/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /home/damon/.cache/vcpkg/archives/a2/a2158e014fef4ae1cf65b8c0b9dff1aabb9e494f.zip
Building package openssl-unix[core]:x64-linux... done
Installing package openssl-unix[core]:x64-linux...
Installing package openssl-unix[core]:x64-linux... done
Elapsed time for package openssl-unix:x64-linux: 3.329 min
Starting package 2/2: openssl:x64-linux
Building package openssl[core]:x64-linux...
Could not locate cached archive: /home/damon/.cache/vcpkg/archives/b7/b78320e8f13a52578c025cb1bc4b3173cd4c3054.zip
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl_x64-linux/share/openssl//usage
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /home/damon/.cache/vcpkg/archives/b7/b78320e8f13a52578c025cb1bc4b3173cd4c3054.zip
Building package openssl[core]:x64-linux... done
Installing package openssl[core]:x64-linux...
Installing package openssl[core]:x64-linux... done
Elapsed time for package openssl:x64-linux: 18.05 ms

Total elapsed time: 3.336 min

The package openssl is compatible with built-in CMake targets:

    find_package(OpenSSL REQUIRED)
    target_link_libraries(main PRIVATE OpenSSL::SSL OpenSSL::Crypto)

Package openssl:x64-linux is already installed.

[SeanTAllen]

I dont see from reading this how it provides any value other than "can write things in a language other than Pony" but that opens up a mess of questions about dependencies for the other code. So to use a plugin, I might have to have bash installed. How does that work on Windows? Does the plugin need to also have a powershell version and a batch version?

At the moment, I don't see how if plugins are part of the corral repo that this is desirable given all the questions that would be raised versus writing the command in Pony.

[damon-kwok]

Yes, bash is a required dependency. We can use the idea of OPAM( ocaml's package manager), which will automatically download msys2 on the windows platform. We can also use git-bash.

[SeanTAllen]

A plugin system that isn't in Pony also bypasses the capabilities based directory security that using only Pony provides. Commands written in Pony using the existing corral system to add commands can be limited to working on a subset of directories. By including non-pony code from ambient auth languages, that goes out the window.

[SeanTAllen]

Yes, bash is a required dependency. We can use the idea of OPAM( ocaml's package manager), which will automatically download msys2 on the windows platform. We can also use git-bash.

This is a huge requirement and one I'm not willing to take on lightly.

[damon-kwok]

The size of msys2 compressed package is about 50mb

[SeanTAllen]

The size of msys2 compressed package is about 50mb

The size doesn't matter. We are asking people to take on an additional currently unneeded dependency for what benefit?

[damon-kwok]

It allows users to use the common commands of unix to do anything.

[SeanTAllen]

It allows users to use the common commands of unix to do anything.

What problem that exists now that people need solved does this address?

[damon-kwok]

I want to add a plug-in system to corral. The plug-in system can help people achieve:

• Handle native dependencies.
• Custom automated tasks
• ...