incubator-teaclave-sgx-sdk

incubator-teaclave-sgx-sdk copied to clipboard

Hi all,

This project now belongs to apache, and has a new repo path at: https://github.com/apache/mesatee-sgx

Currently if one uses the forked crates, he/she still needs to use the old repo URL as git = "https://github.com/baidu/rust-sgx-sdk", rev = "v1.0.9", because all the forked crates are using the old repo URL.

Now, mixed usage of this SDK, like the following Cargo.toml would trigger collision between two sgx_tstd:

sgx_tstd = { rev = "v1.0.9", git = "https://github.com/apache/mesatee-sgx.git" }
wasmi = { git = "https://github.com/mesalock-linux/wasmi-sgx" }

You'll see

error: the `#[global_allocator]` in sgx_tstd conflicts with this global allocator in: sgx_tstd
error: duplicate lang item in crate `sgx_tstd`: `f32_runtime`.
  |
  = note: first defined in crate `sgx_tstd`.

error: duplicate lang item in crate `sgx_tstd`: `f64_runtime`.
  |
  = note: first defined in crate `sgx_tstd`.

error: duplicate lang item in crate `sgx_tstd`: `panic_impl`.
  |
  = note: first defined in crate `sgx_tstd`.

error: duplicate lang item in crate `sgx_tstd`: `begin_panic`.
  |
  = note: first defined in crate `sgx_tstd`.

error: duplicate lang item in crate `sgx_tstd`: `oom`.
  |
  = note: first defined in crate `sgx_tstd`.

error: aborting due to 6 previous errors

error: Could not compile `Helloworldsampleenclave`.

So I'm planning to have thorough update on all of the forked crates to use the new repo URL, and it would affect all of the projects which are depending on the old URL.

Any ideas?

Best, Yu

Congratulations for the Apache tribute. Any chance you can update mesalock crates along with the next version bump? However, this is a delicate issue for us. We'd be blocked if this transition doesn't work smoothly. However, we did lock down our mesalock dependencies to specific git commits, so this might be less of an issue

Thanks @brenzi !

I'm thinking of use a permanent (at least I think it's permanent) domain to host the redirect. I have a candidate rustsgx.com which triggers 301 redirect to https://github.com/apache/mesatee-sgx now. So the following git clone command works:

$ git clone https://rustsgx.com
Cloning into 'rustsgx.com'...
warning: redirecting to https://github.com/apache/mesatee-sgx/
remote: Enumerating objects: 363, done.
remote: Counting objects: 100% (363/363), done.
remote: Compressing objects: 100% (256/256), done.
remote: Total 15613 (delta 125), reused 212 (delta 71), pack-reused 15250
Receiving objects: 100% (15613/15613), 40.78 MiB | 4.87 MiB/s, done.
Resolving deltas: 100% (8118/8118), done.

Seems that git can work with HTTP 301 redirect. However, cargo does not allow me to do that. If I have sgx_types = { git = "http://rustsgx.com" } in Cargo.toml, error would be:

    Updating git repository `http://rustsgx.com/`
warning: spurious network error (2 tries remaining): cross host redirect not allowed; class=Net (12)
warning: spurious network error (1 tries remaining): cross host redirect not allowed; class=Net (12)
error: failed to load source for a dependency on `sgx_tstd`

Caused by:
  Unable to update http://rustsgx.com/

Caused by:
  failed to fetch into /home/ding/.cargo/git/db/_empty-f3f08613514edcc1

Caused by:
  cross host redirect not allowed; class=Net (12)

Any ideas?

Just find a flag in .cargo/config which enables command line git instead of the built-in libgit-sys:

[net]
git-fetch-with-cli = true

If we enable this, we can directly use git = "https://rustsgx.com"

cargo build --release
    Updating git repository `https://rustsgx.com/`
    Updating crates.io index
   Compiling cc v1.0.46
   Compiling sgx_types v1.0.9 (https://rustsgx.com/#1d8a9f87)
   Compiling sgx_build_helper v0.1.0 (https://rustsgx.com/#1d8a9f87)
   Compiling sgx_tstd v1.0.9 (https://rustsgx.com/#1d8a9f87)
   Compiling sgx_demangle v1.0.9 (https://rustsgx.com/#1d8a9f87)
   Compiling sgx_libc v1.0.9 (https://rustsgx.com/#1d8a9f87)
   Compiling sgx_unwind v0.0.1 (https://rustsgx.com/#1d8a9f87)
   Compiling sgx_trts v1.0.9 (https://rustsgx.com/#1d8a9f87)
   Compiling sgx_alloc v1.0.9 (https://rustsgx.com/#1d8a9f87)
   Compiling sgx_tprotected_fs v1.0.9 (https://rustsgx.com/#1d8a9f87)
   Compiling sgx_backtrace_sys v1.0.9 (https://rustsgx.com/#1d8a9f87)
   Compiling Helloworldsampleenclave v1.0.0 (/home/ding/rust-sgx-sdk/samplecode/hello-rust/enclave)
    Finished release [optimized] target(s) in 15.09s

Is this a good choice? Pros: no worry about any repo transfer in future. I'll pay for the domain registration and update the redirection if further repo transfer happens. Cons: Extra config line in .cargo/config required.

@bradyjoestar @elichai @akhilles @davidp94 @cbeck88 @jasl

@moriaab @avishaiw

I would prefer don't change global settings,

if I remember correctly, developers no need to do anything after you transferred the repo, GitHub will do the redirection automatically.

@jasl You're right about git. But cargo is stupid. "github.com/baidu/rust-sgx-sdk" and "github.com/apache/mesatee-sgx" are considered as two different crates. It'll lead to panic handler collisions and mess up the dependency tree.

This is a patch which you guys can apply to hello-rust sample. It shows some usage on "sdk.rustsgx.com" and "repo.rustsgx.com"

diff --git a/samplecode/hello-rust/enclave/.cargo/config b/samplecode/hello-rust/enclave/.cargo/config
new file mode 100644
index 0000000..c91c3f3
--- /dev/null
+++ b/samplecode/hello-rust/enclave/.cargo/config
@@ -0,0 +1,2 @@
+[net]
+git-fetch-with-cli = true
diff --git a/samplecode/hello-rust/enclave/Cargo.toml b/samplecode/hello-rust/enclave/Cargo.toml
index abd6c59..4c932cc 100644
--- a/samplecode/hello-rust/enclave/Cargo.toml
+++ b/samplecode/hello-rust/enclave/Cargo.toml
@@ -10,32 +10,36 @@ crate-type = ["staticlib"]
 [features]
 default = []

+[dependencies]
+log = { git = "https://repo.rustsgx.com/log-sgx" }
+
 [target.'cfg(not(target_env = "sgx"))'.dependencies]
 sgx_types = { git = "https://github.com/baidu/rust-sgx-sdk.git" }
-sgx_tstd = { git = "https://github.com/baidu/rust-sgx-sdk.git" }
+sgx_tstd  = { git = "https://github.com/baidu/rust-sgx-sdk.git" }
+
 [patch.'https://github.com/baidu/rust-sgx-sdk.git']
-sgx_alloc = { path = "../../../sgx_alloc" }
-sgx_build_helper = { path = "../../../sgx_build_helper" }
-sgx_cov = { path = "../../../sgx_cov" }
-sgx_crypto_helper = { path = "../../../sgx_crypto_helper" }
-sgx_libc = { path = "../../../sgx_libc" }
-sgx_rand = { path = "../../../sgx_rand" }
-sgx_rand_derive = { path = "../../../sgx_rand_derive" }
-sgx_serialize = { path = "../../../sgx_serialize" }
-sgx_serialize_derive = { path = "../../../sgx_serialize_derive" }
-sgx_serialize_derive_internals = { path = "../../../sgx_serialize_derive_internals" }
-sgx_tcrypto = { path = "../../../sgx_tcrypto" }
-sgx_tcrypto_helper = { path = "../../../sgx_tcrypto_helper" }
-sgx_tdh = { path = "../../../sgx_tdh" }
-sgx_tkey_exchange = { path = "../../../sgx_tkey_exchange" }
-sgx_tprotected_fs = { path = "../../../sgx_tprotected_fs" }
-sgx_trts = { path = "../../../sgx_trts" }
-sgx_tse = { path = "../../../sgx_tse" }
-sgx_tseal = { path = "../../../sgx_tseal" }
-sgx_tservice = { path = "../../../sgx_tservice" }
-sgx_tstd = { path = "../../../sgx_tstd" }
-sgx_tunittest = { path = "../../../sgx_tunittest" }
-sgx_types = { path = "../../../sgx_types" }
-sgx_ucrypto = { path = "../../../sgx_ucrypto" }
-sgx_unwind = { path = "../../../sgx_unwind" }
-sgx_urts = { path = "../../../sgx_urts" }
+sgx_alloc                       = { git = "https://sdk.rustsgx.com" }
+sgx_build_helper                = { git = "https://sdk.rustsgx.com" }
+sgx_cov                         = { git = "https://sdk.rustsgx.com" }
+sgx_crypto_helper               = { git = "https://sdk.rustsgx.com" }
+sgx_libc                        = { git = "https://sdk.rustsgx.com" }
+sgx_rand                        = { git = "https://sdk.rustsgx.com" }
+sgx_rand_derive                 = { git = "https://sdk.rustsgx.com" }
+sgx_serialize                   = { git = "https://sdk.rustsgx.com" }
+sgx_serialize_derive            = { git = "https://sdk.rustsgx.com" }
+sgx_serialize_derive_internals  = { git = "https://sdk.rustsgx.com" }
+sgx_tcrypto                     = { git = "https://sdk.rustsgx.com" }
+sgx_tcrypto_helper              = { git = "https://sdk.rustsgx.com" }
+sgx_tdh                         = { git = "https://sdk.rustsgx.com" }
+sgx_tkey_exchange               = { git = "https://sdk.rustsgx.com" }
+sgx_tprotected_fs               = { git = "https://sdk.rustsgx.com" }
+sgx_trts                        = { git = "https://sdk.rustsgx.com" }
+sgx_tse                         = { git = "https://sdk.rustsgx.com" }
+sgx_tseal                       = { git = "https://sdk.rustsgx.com" }
+sgx_tservice                    = { git = "https://sdk.rustsgx.com" }
+sgx_tstd                        = { git = "https://sdk.rustsgx.com" }
+sgx_tunittest                   = { git = "https://sdk.rustsgx.com" }
+sgx_types                       = { git = "https://sdk.rustsgx.com" }
+sgx_ucrypto                     = { git = "https://sdk.rustsgx.com" }
+sgx_unwind                      = { git = "https://sdk.rustsgx.com" }
+sgx_urts                        = { git = "https://sdk.rustsgx.com" }

the .cargo/config only affects the enclave itself, because it locates in the "enclave" folder, along with enclave's Cargo.toml. with the help of "sdk.rustsgx.com" and "repo.rustsgx.com", I can make all of the forked crates depends on a permanent "repo" address "sdk.rustsgx.com", and xref each other using "repo.rustsgx.com", no matter where the git repos are.

if this solution is acceptable, I can try if a shorter domain name is affordable. it'll be more ergonomic.

Yes, that cargo issue crosses me all the time. I'd be happy for some endorsing comments (or critique) on my suggestion to resolve this properly: https://github.com/rust-lang/cargo/issues/7497

I'm not sure a custom domain will resolve the underlying problem. It's just a workaround. I could live with changing the global setting but I don't think this is elegant because it might change other behaviour as well as a side effect

@brenzi Yeah I've seen your comments and I agree with you.

btw, the .cargo/config can be local and no need to be global. it can only affect the enclave itself by just putting it together with the enclave's Cargo.toml

the other choice is sgx.rs, sdk.sgx.rs and repo.sgx.rs

given that this project is now under the apache org, i don't see any reason why you'd want to transfer repos again. apache is pretty prestigious. So, why not just go with https://github.com/apache/mesatee-sgx in the forked crates?

@nabilschear The reason is that "mesatee" is a brand name owned by Baidu, and Baidu does not donate it to Apache. So we have to change it soon.

besides, we may have option to make this sdk an isolated project and get rid of the prefix in a couple of years. in this case, we have to change the repo name again.

understood. Whatever the solution to the path problem, when will it roll out to all the forked crates?

@nabilschear I will do this next week after I get back in town.

A new problem is that MesaTEE is being renamed... which means that the repo has to change its repo name again. So should we use the "sgx.rs" jumper, or keep up with every renaming? How do you think?

What about creating a registery? https://doc.rust-lang.org/cargo/reference/registries.html

@elichai Yes I'm thinking about it seriously!

I have done a lot of experiment on that. I think it should be do-able, but with less transparency of porting. Currently, the forked crates can be "compared" directly with the upstream, and the diff reveals the difference clearly just in one click.

Another though is that some crates are forked while some others are not. Does Cargo supports "partially overrided"?

Well you can still keep them on github. Just publish them afterwards. What do you mean partially ovrrided?

(btw I would first spend time reaserching the --sysroot new feature I linked before, I still have hopes for that)

In the previous crates.io design, a create published on crates.io can only depends on crates.io crates.

So can I publish a crate on sgx's crates.io, but make it depends on crates hosted on official crates.io?

regarding to the --sysroot feature, I'm trying it as well :-)

@dingelish that's a great question. Can we use 2 separate crates.io together? Idk.