EvalAI
EvalAI copied to clipboard
Cloud-CV
EvalAI CLI doesn't work with local installation
I installed EvalAI locally following the instructions in the documentation. However, CLI commands are not working with local host
Steps to reproduce:
- Run EvalAI on localhost
$ cd evalai
$ docker-compose up --build
<TRUNCATED>
nodejs_1 | [16:19:03] Server started http://0.0.0.0:8888
- Ensure evalai-cli working on staging environment
$ evalai host -sh http://staging.eval.ai
http://staging.eval.ai is set as the host URL.
$ evalai set_token <STAGING_ACCOUNT_TOKEN>
Success: Authentication token is successfully set.
$ evalai challenges
+-----+---------------+----------------------------------------------------------------------------------------------------------+-----------------------+----------------------+----------------------+
| ID | Title | Short Description | Creator | Start Date | End Date |
+-----+---------------+----------------------------------------------------------------------------------------------------------+-----------------------+----------------------+----------------------+
| 460 | Habitat Chall | An autonomous navigation challenge that aims to benchmark and advance research in embodied AI. Unlike cl | test-host-team | 03/19/20 11:59:59 PM | 12/30/99 10:59:59 PM |
| | enge Auto tes | assical 'internet AI' image dataset-based challenges (e.g., ImageNet LSVRC, COCO, VQA), this is a challe | | | |
| | t 2020 | nge where participants upload code not predictions. The uploaded agents are evaluated in novel (unseen) | | | |
| | | environments to test for generalization. | | | |
+-----+---------------+----------------------------------------------------------------------------------------------------------+-----------------------+----------------------+----------------------+
| 472 | Habitat auto | Random number generation challenge for each submission | test-host-team | 12/31/18 07:00:00 PM | 05/31/99 07:59:59 PM |
| | create cluste | | | | |
| | r test challe | | | | |
| | nge | | | | |
+-----+---------------+----------------------------------------------------------------------------------------------------------+-----------------------+----------------------+----------------------+
| 597 | Random Number | Random number generation challenge for each submission | new team22 | 12/31/18 06:00:00 PM | 05/31/29 07:59:00 PM |
| | Generator Ch | | | | |
| | allenge | | | | |
+-----+---------------+----------------------------------------------------------------------------------------------------------+-----------------------+----------------------+----------------------+
| 649 | Fashion Outfi | The proposed challenge presents a fashion outfits generator problem in which the participants will be as | farfetch_outfits_test | 05/02/22 08:00:00 PM | 06/17/22 07:59:59 PM |
| | ts Challenge | ked to develop a model that is able to generate outfits (sets of products that can be worn together in a | | | |
| | | stylish and complementary way) | | | |
+-----+---------------+----------------------------------------------------------------------------------------------------------+-----------------------+----------------------+----------------------+
- Change evalai-cli host to localhost
$ evalai host -sh http://127.0.0.1:8888
http://127.0.0.1:8888 is set as the host URL.
$ evalai set_token <LOCAL_ACCOUNT_TOKEN>
Success: Authentication token is successfully set.
- check for challenges
Traceback (most recent call last):
File "/Users/vaibhav.adlakha/evalai-cli/venv/bin/evalai", line 33, in <module>
sys.exit(load_entry_point('evalai', 'console_scripts', 'evalai')())
File "/Users/vaibhav.adlakha/evalai-cli/venv/lib/python3.8/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/vaibhav.adlakha/evalai-cli/venv/lib/python3.8/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/vaibhav.adlakha/evalai-cli/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/vaibhav.adlakha/evalai-cli/venv/lib/python3.8/site-packages/click/core.py", line 1043, in invoke
return Command.invoke(self, ctx)
File "/Users/vaibhav.adlakha/evalai-cli/venv/lib/python3.8/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/vaibhav.adlakha/evalai-cli/venv/lib/python3.8/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/vaibhav.adlakha/evalai-cli/venv/lib/python3.8/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/vaibhav.adlakha/evalai-cli/evalai/challenges.py", line 79, in challenges
display_all_challenge_list()
File "/Users/vaibhav.adlakha/evalai-cli/evalai/utils/challenges.py", line 96, in display_all_challenge_list
display_challenges(url)
File "/Users/vaibhav.adlakha/evalai-cli/evalai/utils/challenges.py", line 82, in display_challenges
response = response.json()
File "/Users/vaibhav.adlakha/evalai-cli/venv/lib/python3.8/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
On inspecting the error, it looks like local evalai server returns a webpage in response, whereas the staging server sends back a JSON. Specifically, if I print out the response received here, in staging I get this
{"count":4,"next":null,"previous":null,"results": .... <TRUNCATED> .... }
whereas from local host I get this -
<!DOCTYPE html>
<html ng-app="evalai" ng-strict-di>
<head>
<title ng-bind="pageTitle + ' - EvalAI'"></title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="EvalAI is an open-source web platform for organizing and participating in challenges to push the state of the art on AI tasks.">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<!--Open Graph Related Stuff-->
<meta property="og:type" content="website" />
<meta property="og:title" content="EvalAI: Evaluating state of the art in AI" />
<meta property="og:url" content="https://eval.ai" />
<meta property="og:image" content="https://eval.ai/dist/images/evalai-cover.png" />
<meta property="og:image:type" content="image/png" />
<meta property="og:site_name" content="EvalAI" />
<meta property="og:description" content="EvalAI is an open-source web platform for organizing and participating in challenges to push the state of the art on AI tasks." />
<!--Twitter Card Stuff-->
<meta name="twitter:card" content="summary_large_image" />
<!-- Favicon -->
<link rel="shortcut icon" href="dist/images/favicon.ico" type="image/x-icon">
<link rel="icon" href="dist/images/favicon.ico" type="image/x-icon">
<!-- inject:css -->
<link rel="stylesheet" href="/dist/vendors/vendor.css">
<link rel="stylesheet" href="/dist/css/font-awesome.css">
<link rel="stylesheet" href="/dist/css/main.css">
<!-- endinject -->
<script type="text/javascript">
(function(i, s, o, g, r, a, m) {
i['GoogleAnalyticsObject'] = r;
i[r] = i[r] || function() {
(i[r].q = i[r].q || []).push(arguments)
}, i[r].l = 1 * new Date();
a = s.createElement(o),
m = s.getElementsByTagName(o)[0];
a.async = 1;
a.src = g;
m.parentNode.insertBefore(a, m)
})(window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga');
</script>
<base href="/"></base>
</head>
<body ng-cloak>
<toaster-container></toaster-container>
<!-- render all view here -->
<div ui-view></div>
<!-- html5shiv for IE8 -->
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
<!-- inject:js -->
<script src="/dist/vendors/vendor.js"></script>
<script src="/dist/js/app.js"></script>
<script src="/dist/js/config.js"></script>
<script src="/dist/js/controllers.js"></script>
<script src="/dist/js/directives.js"></script>
<script src="/dist/js/filters.js"></script>
<script src="/dist/js/route-config.js"></script>
<script src="/dist/js/services.js"></script>
<!-- endinject -->
</body>
</html>
OS: macOS Big Sur 11.6.4 Python version: 3.8.9 Docker version: 20.10.7
