question, how about indexing big public repos?

[Slach]

High level details I tried index my fork repo https://github.com/Slach/ClickHouse, but something went wrong

How to debug it? Where bloop store their logs on Windows?

Proposal Add ability to show current logs

[ggordonhall]

Hi @Slach, as of v0.4.3 the indexing architecture has been heavily revised. Can you index this repo?

[Slach]

I clean %USERPROFILE%/AppData/Local/bloop/ and %USERPROFILE%/AppData/Roaming/bloop/ folders run bloop 0.4.3 under windows link with my github account select

and after 24 miniutes of "cloning" status

got "error" status

how to debug root reason for error? %APPDATA%\bloop\bleep\local_cache\github.com_Slach_ClickHouse size only 20 Kb instead of > 1Gb

[Slach]

still reproduce in 0.4.9 @ggordonhall look like you have a timeout for git clone i see "ssh" -o SendEnv=GIT_PROTOCOL [email protected] git-upload-pack 'Slach/ClickHouse.git' which downloaded 350Mb of traffic but stop and re-run after some time

Could you suggest how to configure git clone instead of ssh? or increase timeout for git cloning?

Why application still doesn't have any debug logs?

[rsdy]

@Slach I am trying to repro this on a Mac, but checkout works for your repo. Another strange thing is that you'd see an ssh process communicate, since after 0.3.5 all git-related operations in Bloop are implemented in Rust, and work over HTTPS.

[Slach]

@rsdy 0.4.14

still execute ssh.exe and failed

[Slach]

how about add debug logs?

[Slach]

found logs folder

2023-08-27T12:18:26.899802Z DEBUG sync{repo=github.com/Slach/ClickHouse}: bleep::background::sync: new status self.reporef=RepoRef { backend: Github, name: "Slach/ClickHouse" } 
new_status=Error { message: "git clone fetch: HeadUpdate(FileTransactionCommit(PackedTransactionCommit(Commit(Error { 
error: Os { code: 5, kind: PermissionDenied, message: \"Access is denied.\" }, 
instance: File { inner: Handle { 
id: 34, _marker: PhantomData<gix_tempfile::handle::Writable> }, 
lock_path: \"C:\\\\Users\\\\Slach\\\\AppData\\\\Roaming\\\\bloop\\\\bleep\\\\local_cache\\\\github.com_Slach_ClickHouse\\\\packed-refs.lock\" } }))))" }

2023-08-27T12:18:26.906129Z ERROR bleep::background::sync: failed to sync repository err=Sync(GitCloneFetch(HeadUpdate(FileTransactionCommit(PackedTransactionCommit(
Commit(Error { error: Os { code: 5, kind: PermissionDenied, 
message: "Access is denied." }, 
instance: File { inner: Handle { 
id: 34, _marker: PhantomData<gix_tempfile::handle::Writable> }, 
lock_path: "C:\\Users\\Slach\\AppData\\Roaming\\bloop\\bleep\\local_cache\\github.com_Slach_ClickHouse\\packed-refs.lock" } })))))) 
self.reporef=RepoRef { backend: Github, name: "Slach/ClickHouse" }

I tried to run bloop under Admin user after chmod 0777 /cygdrive/c/Users/Slach/AppData/Roaming/bloop/bleep/local_cache -Rv bloop successfully clone repo

but got another error

2023-08-28T04:00:42.214608Z DEBUG bleep::background::sync: new status self.reporef=RepoRef { backend: Github, name: "Slach/ClickHouse" } new_status=Indexing
2023-08-28T04:00:42.220905Z  INFO bleep::indexes::repo: finished indexing repo metadata repo.disk_path="C:\\Users\\Slach\\AppData\\Roaming\\bloop\\bleep\\local_cache\\github.com_Slach_ClickHouse"
2023-08-28T04:00:43.008778Z  WARN bleep::indexes::file: Path not found in language map entry_pathbuf="\\\\?\\C:\\Users\\Slach\\AppData\\Roaming\\bloop\\bleep\\local_cache\\github.com_Slach_ClickHouse\\config"
2023-08-28T04:00:43.008751Z  WARN bleep::indexes::file: Path not found in language map entry_pathbuf="\\\\?\\C:\\Users\\Slach\\AppData\\Roaming\\bloop\\bleep\\local_cache\\github.com_Slach_ClickHouse\\refs\\remotes\\origin\\fix-datetime-overflow-formatting"

and after it got

2023-08-28T04:00:53.550245Z DEBUG bleep::background::sync: new status self.reporef=RepoRef { backend: Github, name: "Slach/ClickHouse" } new_status=Done
2023-08-28T04:00:53.550307Z DEBUG bleep::background: sync finished result=Ok(Done)

but Natural language queries still not available

[ggordonhall]

Hey @Slach, do you still experience this?

[Slach]

@ggordonhall yes, pissue still present local copy of git repo indexed fine

but public repo with http://github.com/ClickHouse/ClickHouse url added but stuck in cloning state

and now, I don't see any ERROR in logs

2023-10-24T14:06:07.291844Z  INFO bleep::webserver: starting webserver bind=127.0.0.1:7878
2023-10-24T14:06:07.809923Z DEBUG bleep::webserver::github: github auth started interval=3s
2023-10-24T14:11:02.064284Z DEBUG bleep::webserver::github: acquired credentials
2023-10-24T14:11:02.491904Z DEBUG bleep::periodic::remotes: credentials exist
2023-10-24T14:11:03.441093Z  INFO bleep::analytics: sent analytics event...
2023-10-24T14:11:03.445534Z DEBUG bleep::webserver::github: github auth complete
2023-10-24T14:11:05.279812Z DEBUG bleep::periodic::remotes: repo list updated
2023-10-24T14:11:06.665517Z DEBUG bleep::periodic::remotes: updated user user="Slach"
2023-10-24T14:11:38.190833Z  INFO bleep::background: queueing for sync reporef=github.com/CLickHouse/ClickHouse
2023-10-24T14:11:38.191108Z  INFO bleep::background: indexing next.reporef=RepoRef { backend: Github, name: "CLickHouse/ClickHouse" }
2023-10-24T14:11:38.191167Z DEBUG bleep::background::sync: syncing repo self.reporef=RepoRef { backend: Github, name: "CLickHouse/ClickHouse" }
2023-10-24T14:11:38.191199Z DEBUG bleep::background::sync: new status self.reporef=RepoRef { backend: Github, name: "CLickHouse/ClickHouse" } new_status=Syncing
2023-10-24T14:11:38.355580Z  INFO bleep::analytics: sent analytics event...
2023-10-24T14:14:07.491093Z DEBUG bleep::periodic::remotes: timeout expired; refreshing repositories
2023-10-24T14:14:07.491198Z DEBUG bleep::periodic::remotes: credentials exist
2023-10-24T14:14:10.301717Z DEBUG bleep::periodic::remotes: repo list updated
2023-10-24T14:14:11.367351Z DEBUG bleep::periodic::remotes: updated user user="Slach"

looks like ssh.exe executed and closed after some time out

[Slach]

installed version of bloop 0.5.6