support for 3rd party crates from crates.io mirrors?

[lilydjwg]

Hi, I heard racer supports completing for third party crates. That's great! However, it hard codes the github.com index. I'm using a crates.io mirror (index = "git://crates.mirrors.ustc.edu.cn/index" in ~/.cargo/config) because I'm in China and the network speed to GitHub is very slow.

Would racer read cargo's config file to find out which index to use?

[phildawes]

Hello, Racer's cargo support is a bit hacky, and it does hardcode the 'github' directory. The github specific code is in here, if you fancy making a patch: (search for github) https://github.com/phildawes/racer/blob/master/src/racer/cargo.rs

What does your .cargo directory structure look like? Thanks!

[lilydjwg]

This is what tree -L 4 ~/.cargo outputs. I've removed a lot of similar entries since it's too long.

/home/lilydjwg/.cargo
├── config
├── git
│   ├── checkouts
│   │   ├── bytes-b21aba8e191ae1b0
│   │   │   ├── 7edb577d0a
│   │   │   └── master
│   │   ├── cargo-0c67b1a5f6ce0f88
│   │   │   └── master
│   │   ├── cargo-2f7f63790bcc5cbc
│   │   │   └── master
│   │   ├── nix-rust-8d1a824a13ec5246
│   │   │   ├── ad87c3bdc9
│   │   │   ├── f37e45628e
│   │   │   └── master
│   └── db
│       ├── bytes-b21aba8e191ae1b0
│       │   ├── config
│       │   ├── description
│       │   ├── FETCH_HEAD
│       │   ├── HEAD
│       │   ├── hooks
│       │   ├── info
│       │   ├── objects
│       │   └── refs
│       ├── cargo-0c67b1a5f6ce0f88
│       │   ├── config
│       │   ├── description
│       │   ├── FETCH_HEAD
│       │   ├── HEAD
│       │   ├── hooks
│       │   ├── info
│       │   ├── objects
│       │   └── refs
└── registry
    ├── cache
    │   ├── crates.mirrors.ustc.edu.cn-4496af6807a54617
    │   │   ├── advapi32-sys-0.1.2.crate
    │   │   ├── aho-corasick-0.5.1.crate
    │   │   ├── ansi_term-0.7.2.crate
    │   │   ├── bitflags-0.1.1.crate
    ├── index
    │   ├── crates.mirrors.ustc.edu.cn-4496af6807a54617
    │   │   ├── cm
    │   │   ├── co
    │   │   ├── config.json
    │   │   ├── cp
    │   └── github.com-88ac128001ac3a9a
    │       ├── co
    │       ├── config.json
    │       ├── cp
    └── src
        ├── crates.mirrors.ustc.edu.cn-4496af6807a54617
        │   ├── advapi32-sys-0.1.2
        │   ├── aho-corasick-0.5.1
        │   ├── ansi_term-0.7.2
        │   ├── bitflags-0.1.1
        └── github.com-88ac128001ac3a9a
            ├── advapi32-sys-0.1.2
            ├── aho-corasick-0.3.4
            ├── aho-corasick-0.4.0

[bruno-medeiros]

BTW, this issue also seems to manifest when Cargo.lock references a specific Git revision, for example:

[[package]]
name = "clipboard"
version = "0.1.2"
source = "git+https://github.com/aweinstock314/rust-clipboard#4fe9892d7bf8bac075aa60493c6eee7a83e24b7f"

which also makes the crate source go to .cargo/git/checkouts.

Perhaps now cargo metadata should be used to fix this? The metadata command will provide the correct file system location of each crate root, so Racer doesn't need to replicate the logic to figure it out.

[bruno-medeiros]

(I might be up for working on this when I have the time, but it won't be anytime soon)

[jwilm]

@bruno-medeiros do you know if the cargo metadata functionality is available from a crate? execing a process each time a crate lookup is done would hurt performance..

[bruno-medeiros]

I don't know if it is, you'd have to ask the Cargo devs. But I don't think executing the metadata functionality inside Racer would make that much more of an improvement. (the exec overhead is not that bad?)

Note that you only request cargo metadata once per Racer operation. You request it for the crate where the Racer operation is invoked, and the metadata will give you the source info for all the dependent crates.

To improve on that (caching the metadata for example), would require Racer running on deamon mode.

[jwilm]

Cargo appears to be on crates.io, and the metadata operation seems to rely on resolve_dependencies. Maybe this feature could use that directly.

If the metadata resolution is fast enough, it might not be worth caching. Cache invalidation seems difficult here since it would require knowledge of cargo's metadata resolution.

[jwilm]

I looked into using the cargo metadata machinery to resolve dependencies. There is a dependency resolution step involving updating the registry information, and updating git repositories. From what I can tell, there is no way to use cargo internals to return the dependency paths on disk without this network activity, and it is slow even in the case of a no-op (think 1-2 seconds).

My current idea to resolve this would be writing a more formal dependency analysis crate that is knowledgeable about cargo/multirust/rustup.rs and can do the local path resolution quickly, but that's more work than simply dropping in cargo.

[bruno-medeiros]

You can cache the cargo metadata information based on the timestamps of Cargo.toml and Cargo.lock. Basically only run cargo metadata if the modification time of any of those two files changed.

[norru]

Has this feature being released? I've got latest RustDT, rustup and racer, Ctrl+Click on symbol and autocomplete only work on 1) current project 2) rustc src. External crates don't get resolved, for instance: ctrl+click on

use super::timing::Duration;

here's the Rust Tools Log:

>> Running: /home/nico/.cargo/bin/racer find-definition 3 19 /home/nico/Projects/3rdParty/amethyst/src/engine/state.rs
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
ERROR:racer::cargo: ERROR!: Error { repr: Os { code: 2, message: "No such file or directory" } } .cargo/registry/src/github.com-88ac128001ac3a9a/racer-1.2.9/src/racer/cargo.rs 435
END
  ^^^ Terminated, exit code: 0 ^^^

I can provide more detailed or specific test results. It's just that I need to know whether I'm chasing a wild goose, because feature is not implemented, before doing so.

Thanks!

[hawnzug]

I have the same problem with @lilydjwg when using crate.io mirror(git://mirrors.ustc.edu.cn/crates.io-index) in China. Here my tree output under .cargo/registry/index

tree -L 1
.
├── crates.mirrors.ustc.edu.cn-245570943fb24b9d
├── crates.mirrors.ustc.edu.cn-e2f007c923a8536a
├── github.com-1ecc6299db9ec823
├── github.com-88ac128001ac3a9a
└── mirrors.ustc.edu.cn-61ef6e0cd06fb9b8

5 directories, 0 files

Removing these hardcoded validation codes and simply pushing the path temporarily works for me, but it may cause some conflicts or misleading results if some crates were in both index(I'm not sure because now my crates is completely dependent on the mirror so this workaround seems to be working fine).

As @lilydjwg suggests, analyzing cargo's config file to find out which index to use may be a better approach.

[xzn]

A related issue is that 'patch' and 'replace' section of Cargo.toml are ignored. Using 'cargo metadata' would fix this as well, in addition to getting the right version of the crate requested. I've looked into src/racer/cargo.rs however the api seems limited though, not sure how to go about fixing this.

Another issue with 'cargo metadata' is that it makes network requests to update index and download missing crates, I've tried passing -Zoffline but this just causes error when crates are missing.

[kngwyu]

Fixed in my branch, tough this implementation depends on internet:fearful:

[ksqsf]

racer 2.0.14. Still affected. A simple workaround is as follows in case anybody reaches here.

cd ~/.cargo/registry/src
ln -s mirrors.ustc.edu.cn-* github.com-

[sidgwick]

@ksqsf thanks, it works for me.

[huangjj27]

For windows user, the mklink command also solve the problem

cd .cargo\registry\src
mklink github.com- mirrors.ustc.edu.cn-*

[sbwtw]

@huangjj27 Should be using /D option to create a dir link (default is file link)

mklink /D github.com- mirrors.ustc.edu.cn-*

[WangPengJu]

still can't work i try use

cd ~/.cargo/registry/src
ln -s mirrors.ustc.edu.cn-* github.com-

but still can't work