python-harborclient icon indicating copy to clipboard operation
python-harborclient copied to clipboard

Published 20 hours ago • verified publisher icon int32bit

ERROR (CommandError): You must provide project via either --os-project or env[HARBOR_PROJECT]

Open DavidRodHub opened this issue 7 years ago • 12 comments

Get the error when running command "harbor user"list.

Install harborclient per instructions using options 1

  1. Cloned latest source.
  2. docker biuld -t drodriguez/harborclient .
  3. Ran: docker run --rm
    -e HARBOR_USERNAME="admin"
    -e HARBOR_PASSWORD="Harbor12345"
    -e HARBOR_PROJECT=1
    -e HARBOR_URL="https://docker-reg.mydomain.com"
    drodriguez/harborclient harbor info

+------------------------------+-------------------------+ | Property | Value | +------------------------------+-------------------------+ | admiral_endpoint | NA | | auth_mode | ldap_auth | | disk_free | 108298657792 | | disk_total | 984372027392 | | harbor_version | v1.3.0 | | has_ca_root | False | | next_scan_all | 0 | | project_creation_restriction | everyone | | registry_url | docker-reg.mydomain.com | | self_registration | True | | with_admiral | False | | with_clair | False | | with_notary | False | +------------------------------+-------------------------+

4. Created an alias alias harbor='docker run
-e HARBOR_USERNAME="admin"
-e HARBOR_PASSWORD="Harbor12345"
-e HARBOR_URL="https://docker-reg.mydomain.com"
--rm drodriguez/harborclient harbor'

5. Ran command: harbor user-list

ERROR (CommandError): You must provide project via either --os-project or env[HARBOR_PROJECT]

For option ' --os-project' , what argurment does it required? I tried listing a project in our harbor registry, and got the same error.

DavidRodHub avatar May 09 '18 23:05 DavidRodHub

Thanks for your feedback, I have added missing HARBOR_PROJECT environment to docker alias in docs.

alias harbor='docker run 
-e HARBOR_USERNAME="admin" 
-e HARBOR_PASSWORD="Harbor12345" 
-e HARBOR_PROJECT=1
-e HARBOR_URL="https://docker-reg.mydomain.com" 
--rm drodriguez/harborclient harbor'

int32bit avatar May 10 '18 03:05 int32bit

That worked. Now I can run harbor project-list and see the projects in our harbor docker registry.

$ harbor project-list
+------------+----------------+----------+----------------------+------------+----------------------+--------+
| project_id |      name      | owner_id | current_user_role_id | repo_count |    creation_time     | public |
+------------+----------------+----------+----------------------+------------+----------------------+--------+
|     2      |      eaa       |    1     |          1           |    171     | 2018-01-31T19:49:30Z |   -    |
|     3      |    sandbox     |    1     |          1           |     9      | 2018-01-31T19:52:26Z |   -    |
|     5      | adaptors-tools |    1     |          1           |     2      | 2018-02-09T22:12:52Z |   -    |
|     6      |   thirdparty   |    1     |          1           |     1      | 2018-02-14T18:15:49Z |   -    |
+------------+----------------+----------+----------------------+------------+----------------------+--------+

Tried to list the repositories by running "harbor --debug list" but not successfuil.

DEBUG (connectionpool:824) Starting new HTTPS connection (1): docker-reg.mydomain.com
DEBUG (connectionpool:396) https://docker-reg.mydomain.com:443 "POST /login HTTP/1.1" 200 0
DEBUG (client:282) Successfully login, session id: 73f10df3ce3c792ad44616fa4acbfed7
REQ: curl -g -i 'https://docker-reg.mydomain.com/api/repositories?project_id=1' -X GET -H "Accept: application/json" -H "Harbor-API-Version: v2" -H "User-Agent: python-harborclient" -b "beegosessionID: 73f10df3ce3c792ad44616fa4acbfed7"
DEBUG (connectionpool:824) Starting new HTTPS connection (1): docker-reg.mydomain.com
DEBUG (connectionpool:396) https://docker-reg.mydomain.com:443 "GET /api/repositories?project_id=1 HTTP/1.1" 404 20
RESP: [404] {'Content-Length': '20', 'X-Content-Type-Options': 'nosniff', 'Keep-Alive': 'timeout=5', 'Server': 'nginx', 'Connection': 'keep-alive', 'Date': 'Thu, 10 May 2018 20:31:14 GMT', 'Content-Type': 'text/plain; charset=utf-8'}
RESP BODY: null

Traceback (most recent call last):
  File "/usr/local/bin/harbor", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/harborclient/shell.py", line 404, in main
    HarborShell().main(argv)
  File "/usr/local/lib/python2.7/site-packages/harborclient/shell.py", line 336, in main
    args.func(self.cs, args)
  File "/usr/local/lib/python2.7/site-packages/harborclient/v2/shell.py", line 322, in do_list
    repositories = cs.repositories.list(project_id)
  File "/usr/local/lib/python2.7/site-packages/harborclient/v2/repositories.py", line 11, in list
    repositories = self._list("/repositories?project_id=%s" % project)
  File "/usr/local/lib/python2.7/site-packages/harborclient/base.py", line 28, in _list
    data = self.api.client.get(url)
  File "/usr/local/lib/python2.7/site-packages/harborclient/client.py", line 239, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/local/lib/python2.7/site-packages/harborclient/client.py", line 224, in _cs_request
    **kwargs)
  File "/usr/local/lib/python2.7/site-packages/harborclient/client.py", line 210, in _time_request
    body = self.request(url, method, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/harborclient/client.py", line 201, in request
    raise exceptions.from_response(resp, resp.text, url, method)

DavidRodHub avatar May 10 '18 20:05 DavidRodHub

Could you please run

curl -g -i 'https://docker-reg.mydomain.com/api/repositories?project_id=1' -X GET -H "Accept: application/json" -H "Harbor-API-Version: v2" -H "User-Agent: python-harborclient" -b "beegosessionID: 73f10df3ce3c792ad44616fa4acbfed7"

in your bash and check if harbor API is OK?

int32bit avatar May 11 '18 02:05 int32bit

Here's the output of the command you requested:

HTTP/1.1 404 Not Found Server: nginx Date: Mon, 14 May 2018 04:55:35 GMT Content-Type: text/plain; charset=utf-8 Content-Length: 20 Connection: keep-alive Keep-Alive: timeout=5 Set-Cookie: beegosessionID=f92776e8efa944632bcc8718aff5361c; Path=/; secure; secure; HttpOnly X-Content-Type-Options: nosniff

Note that when I modified the alias command and changed project_id to '2' , it gave me the repository list as expected. but took about 5 minutes to generate the list.

alias harbor='docker run \

-e HARBOR_USERNAME="admin"
-e HARBOR_PASSWORD="password"
*> -e HARBOR_PROJECT=2 * -e HARBOR_URL="https://docker-reg.mydomain.com"
--rm drodriguez/harborclient harbor'

+----------------------------------------------------------+------------+------------+------------+------------+------------+----------------------+ | name | project_id | size | tags_count | star_count | pull_count | update_time | +----------------------------------------------------------+------------+------------+------------+------------+------------+----------------------+ | eaa/eaa-nx-clients-common | 2 | 538430066 | 3 | 0 | 325 | 2018-05-06T23:54:17Z | | eaa/eaa-nx-clients-common:1.5-eaa-12 | 2 | 537667334 | 3 | 0 | 325 | 2018-05-06T23:54:17Z | | eaa/eaa-nx-clients-common:12.0 | 2 | 538430066 | 3 | 0 | 325 | 2018-05-06T23:54:17Z | | eaa/eaa-nx-apm | 2 | 677258182 | 2 | 0 | 2 | 2018-05-09T20:46:53Z | | eaa/eaa-nx-apm:12.0 | 2 | 668649110 | 2 | 0 | 2 | 2018-05-09T20:46:53Z | | eaa/eaa-pa-problem-manager:2.1.11.0 | 2 | 731425810 | 10 | 0 | 1 | 2018

. .

DavidRodHub avatar May 14 '18 05:05 DavidRodHub

Let me know if you want me to try anything else regarding why the harbor list take over 5 minutes to list.

DavidRodHub avatar May 15 '18 15:05 DavidRodHub

I suggest you create a breakpoint(pdb for example) in https://github.com/int32bit/python-harborclient/blob/master/harborclient/v2/shell.py#L319 and run again step by step, then identify where the potential bottlenecks might be.

int32bit avatar May 15 '18 15:05 int32bit

Not familiar with python and the steps needed to create breakpoints. It's possible because of the distance between the harbor client (US) and harbor-docker-registery located in France is what's causing the delay in getting the repo list.

I made an attempt to install harborclient on the harbor registry located in France to see it that would make a difference getting the list report but was not successful installing. Reference case: --> https://github.com/int32bit/python-harborclient/issues/10

DavidRodHub avatar May 16 '18 02:05 DavidRodHub

Ran "harbor list" locally on the harbor-docker registry server in France and on the remote client in US. Local: 1m 10s Remote: 8m 23s

Big difference.

DavidRodHub avatar May 17 '18 00:05 DavidRodHub

I don't think the API request should take up so long time(At most 10 seconds in our environment). Could you please do a API request using pure HTTP like curl and observe which side is the key bottleneck(either server side or client side)? For example:

curl -g -i -X GET 'https://docker-reg.mydomain.com/api/repositories?project_id=1'  \
-H "Accept: application/json" \
-H "Harbor-API-Version: v2" \
-H "User-Agent: python-harborclient" \
-b "beegosessionID: ${SESSION_ID}"

You should replace your $SESSION_ID with your session token.

int32bit avatar May 17 '18 02:05 int32bit

How do I get the session token?

On Wed, May 16, 2018, 7:36 PM int32bit [email protected] wrote:

I don't think the API request should take up so long time(At most 10 seconds in our environment). Could you please do a API request using pure HTTP like curl and observe which side is the key bottleneck(either server side or client side)? For example:

curl -g -i -X GET 'https://docker-reg.mydomain.com/api/repositories?project_id=1'
-H "Accept: application/json"
-H "Harbor-API-Version: v2"
-H "User-Agent: python-harborclient"
-b "beegosessionID: ${SESSION_ID}"

You should replace your $SESSION_ID with your session token.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/int32bit/python-harborclient/issues/9#issuecomment-389725878, or mute the thread https://github.com/notifications/unsubscribe-auth/ADSLxPtIDxklhfsvwJDZ13YA9Mc_QBOYks5tzOI2gaJpZM4T5KoB .

DavidRodHub avatar May 17 '18 05:05 DavidRodHub

Depend on your deployment, but hope this help you:

curl -si -X POST  \
-d '{"pincipal": "username", "password": "your_password"}' \
https://docker-reg.mydomain.com/login

If this doesn't work, please ask your administrator for auth API.

int32bit avatar May 17 '18 06:05 int32bit

I was able to get the session token by running harbor-client command with --debug option

$ harbor --debug project-list

Output

DEBUG (connectionpool:824) Starting new HTTPS connection (1): docker-reg.mydomain.com DEBUG (connectionpool:396) https://docker-reg.mydomain.com:443 "POST /login HTTP/1.1" 200 0 DEBUG (client:282) Successfully login, session id: 5c6fe19751531ae250e0ecf60726e69f

REQ: curl -g -i 'https://docker-reg.mydomain.com/api/projects' -X GET -H "Accept: application/json" -H "Harbor-API-Version: v2" -H "User-Agent: python-harborclient" -b "beegosessionID: 5c6fe19751531ae250e0ecf60726e69f"

DEBUG (connectionpool:824) Starting new HTTPS connection (1): docker-fr.mycom-osi.com DEBUG (connectionpool:396) https://docker-reg.mydomain.com:443 "GET /api/projects HTTP/1.1" 200 378 RESP: [200] {'Content-Length': '378', 'Content-Encoding': 'gzip', 'Keep-Alive': 'timeout=5', 'Server': 'nginx', 'Connection': 'keep-alive', 'X-Total-Count': '4', 'Date': 'Thu, 17 May 2018 19:39:14 GMT', 'Content-Type': 'application/json; charset=utf-8'} RESP BODY: [{"update_time": "2018-02-09T22:12:52Z", "owner_name": "", "name": "adaptors-tools", "deleted": 0, "repo_count": 2, "creation_time": "2018-02-09T22:12:52Z", "togglable": true, "metadata": {"public": "false"}, "current_user_role_id": 1, "project_id": 5, "owner_id": 1}, {"update_time": "2018-01-31T19:49:30Z", "owner_name": "", "name": "eaa", "deleted": 0, "repo_count": 172, "creation_time": "2018-01-31T19:49:30Z", "togglable": true, "metadata": {"enable_content_trust": "false", "severity": "low", "auto_scan": "false", "prevent_vul": "false", "public": "false"}, "current_user_role_id": 1, "project_id": 2, "owner_id": 1}, {"update_time": "2018-01-31T19:52:26Z", "owner_name": "", "name": "sandbox", "deleted": 0, "repo_count": 12, "creation_time": "2018-01-31T19:52:26Z", "togglable": true, "metadata": {"public": "false"}, "current_user_role_id": 1, "project_id": 3, "owner_id": 1}, {"update_time": "2018-02-14T18:15:49Z", "owner_name": "", "name": "thirdparty", "deleted": 0, "repo_count": 1, "creation_time": "2018-02-14T18:15:49Z", "togglable": true, "metadata": {"public": "false"}, "current_user_role_id": 1, "project_id": 6, "owner_id": 1}]

+------------+----------------+----------+----------------------+------------+----------------------+--------+ | project_id | name | owner_id | current_user_role_id | repo_count | creation_time | public | +------------+----------------+----------+----------------------+------------+----------------------+--------+ | 2 | eaa | 1 | 1 | 172 | 2018-01-31T19:49:30Z | - | | 3 | sandbox | 1 | 1 | 12 | 2018-01-31T19:52:26Z | - | | 5 | adaptors-tools | 1 | 1 | 2 | 2018-02-09T22:12:52Z | - | | 6 | thirdparty | 1 | 1 | 1 | 2018-02-14T18:15:49Z | - | +------------+----------------+----------+----------------------+------------+----------------------+--------+

Ran curl command

$ curl -g -i 'https://docker-reg.mydomain.com/api/projects' -X GET -H "Accept: application/json" -H "Harbor-API-Version: v2" -H "User-Agent: python-harborclient" -b "beegosessionID: 5c6fe19751531ae250e0ecf60726e69f"

Output:

HTTP/1.1 200 OK Server: nginx Date: Thu, 17 May 2018 19:42:08 GMT Content-Type: application/json; charset=utf-8 Content-Length: 4 Connection: keep-alive Keep-Alive: timeout=5 Set-Cookie: beegosessionID=5a629dda777b9bd70bc3b819c0961c81; Path=/; secure; secure; HttpOnly X-Total-Count: 0

The response running on:

  • harbor registry server --> 0m0.028s
  • Remote server in US --> 0m0.928s

I believe the reason it takes over a minute to generate the list report is because there are over 400 tags in the harbor-docker registry.

DavidRodHub avatar May 17 '18 19:05 DavidRodHub