Documentation

[eong93]

Hi,

Is there more documentation on how to run this? I'm having trouble with training from scratch. Also, is "kohei-solution-20170612_4.zip" available somewhere?

Thanks

[dlindenbaum]

Hello,

The software code associated with kohei-solution-20170612_4.zip is in this directory.
https://github.com/SpaceNetChallenge/BuildingDetectors_Round2/tree/master/1-XD_XD

The zip file isn't currently available for download but the directions should be able to be followed from there.

This will require retraining.

Thank you, Dave

[eong93]

Dave,

Yeah I've tried following the training step from the "Instruction Manual" document", but I get errors like FileNotFoundError: File b'/data/working/models/v9s/AOI_5_Khartoum_val_evalhist_t h.csv' does not exist as well as for the other files in /data/working dir. Isn't training supposed to create those files?

Also, am I right to assume that I'm supposed to mount a directory "data" like below to the docker? /data ├── test │ ├── AOI_2_Vegas_Test │ ├── AOI_3_Paris_Test │ ├── AOI_4_Shanghai_Test │ └── AOI_5_Khartoum_Test └── train ├── AOI_2_Vegas_Train ├── AOI_3_Paris_Train ├── AOI_4_Shanghai_Train └── AOI_5_Khartoum_Train

Thanks for the help

[dlindenbaum]

Hello,

What was the training command?

The working directory has large files that are created from the dataset. They are too large to host on Github. The training process is supposed to recreate them.

Are there any other errors above the FileNoteFound error. Usually this is caused because there was an error earlier in the data processing steps.

Thank you, Dave

[eong93]

Dave,

I followed these steps:

1. Run Container (container name: kohei-container) $ sudo nvidia-docker start -a kohei-container -i
2. Training root@xxxxxxxx:~$ ./train.sh
   /data/train/AOI_2_Vegas_Train
   /data/train/AOI_3_Paris_Train
   /data/train/AOI_4_Shanghai_Train
   /data/train/AOI_5_Khartoum_Train

Also, here is more of the output I got

`>>>>>>>>>> v17.py 2017-11-28 18:31:43,247 INFO Evaluate fscore on validation set: AOI_5_Khartoum 2017-11-28 18:31:43,247 INFO import modules Using Theano backend. WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be remove d in the next release (v0.10). Please switch to the gpuarray backend. You can g et more information about how to switch at this URL: https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpua rray%29

Using gpu device 0: Tesla K80 (CNMeM is disabled, cuDNN 5110) 2017-11-28 18:31:45,346 INFO Prediction phase Traceback (most recent call last): File "v17.py", line 623, in cli() File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 72 2, in call return self.main(*args, **kwargs) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 69 7, in main rv = self.invoke(ctx) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 10 66, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 89 5, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 53 5, in invoke return callback(*args, **kwargs) File "v17.py", line 582, in evalfscore area_id, enable_tqdm=True) File "/root/v9s.py", line 1440, in _internal_validate_predict_best_param param = _get_model_parameter(area_id) File "/root/v9s.py", line 298, in _get_model_parameter best_row = pd.read_csv(fn_hist).sort_values( File "/opt/conda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 646, in parser_f return _read(filepath_or_buffer, kwds) File "/opt/conda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 389, in _read parser = TextFileReader(filepath_or_buffer, **kwds) File "/opt/conda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 730, in init self._make_engine(self.engine) File "/opt/conda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 923, in _make_engine self._engine = CParserWrapper(self.f, **self.options) File "/opt/conda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 1390, in init self._reader = _parser.TextReader(src, **kwds) File "pandas/parser.pyx", line 373, in pandas.parser.TextReader.cinit (pan das/parser.c:4184) File "pandas/parser.pyx", line 667, in pandas.parser.TextReader.setup_parser source (pandas/parser.c:8449) FileNotFoundError: File b'/data/working/models/v9s/AOI_5_Khartoum_val_evalhist_t h.csv' does not exist`

[dlindenbaum]

Is that the entire output?

v17.py Is one of the last things to be printed. '/data/working/models/v9s/AOI_5_Khartoum_val_evalhist_t h.csv' should be produced in the pre_processing steps.

Usually there are errors higher up that indicate a problem

Thank you, Dave

[eong93]

Here's the first part:

root@*****:~# ./train.sh \

/data/train/AOI_2_Vegas_Train \
/data/train/AOI_3_Paris_Train \
/data/train/AOI_4_Shanghai_Train \
/data/train/AOI_5_Khartoum_Train

CLEAN UP rm -rf /data/working PREPROCESSING STEP python v5_im.py preproc_train /data/train/AOI_2_Vegas_Train 2017-11-28 18:29:49,663 INFO Preproc for training on AOI_2_Vegas 2017-11-28 18:29:49,663 INFO Generate IMAGELIST csv Traceback (most recent call last): File "v5_im.py", line 853, in cli() File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 722, in call return self.main(*args, **kwargs) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "v5_im.py", line 745, in preproc_train prep_valtrain_valtest_imagelist(area_id) File "v5_im.py", line 568, in prep_valtrain_valtest_imagelist df = _load_train_summary_data(area_id) File "v5_im.py", line 502, in _load_train_summary_data df = pd.read_csv(fn) File "/opt/conda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 646, in parser_f return _read(filepath_or_buffer, kwds) File "/opt/conda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 389, in _read parser = TextFileReader(filepath_or_buffer, **kwds) File "/opt/conda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 730, in init self._make_engine(self.engine) File "/opt/conda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 923, in _make_engine self._engine = CParserWrapper(self.f, **self.options) File "/opt/conda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py", line 1390, in init self._reader = _parser.TextReader(src, **kwds) File "pandas/parser.pyx", line 373, in pandas.parser.TextReader.cinit (pandas/parser.c:4184) File "pandas/parser.pyx", line 667, in pandas.parser.TextReader._setup_parser_source (pandas/parser.c:8449) FileNotFoundError: File b'/data/train/AOI_2_Vegas_Train/summaryData/AOI_2_Vegas_Train_Building_Solutions.csv' does not exist python v12_im.py preproc_train /data/train/AOI_2_Vegas_Train 2017-11-28 18:29:50,556 INFO Preproc for training on AOI_2_Vegas Traceback (most recent call last): File "v12_im.py", line 629, in cli() File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 722, in call return self.main(*args, **kwargs) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "v12_im.py", line 564, in preproc_train prefix=prefix)).exists() AssertionError python v16.py preproc_train /data/train/AOI_2_Vegas_Train Using Theano backend. WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10). Please switch to the gpuarray backend. You can get more information about how to switch at this URL: https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: Tesla K80 (CNMeM is disabled, cuDNN 5110) 2017-11-28 18:30:00,984 INFO Serialize OSM subset 2017-11-28 18:30:00,984 INFO Loading raster... Traceback (most recent call last): File "v16.py", line 1772, in cli() File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 722, in call return self.main(*args, **kwargs) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/conda/envs/py35/lib/python3.5/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "v16.py", line 1725, in preproc_train preproc_osm(area_id, datapath, is_train=True) File "v16.py", line 1411, in preproc_osm df = location_summary(area_id, datapath) File "v16.py", line 1137, in location_summary df_location.loc[:, 'xcenter'] = df_location.pos.apply(lambda x: x.xcenter) File "/opt/conda/envs/py35/lib/python3.5/site-packages/pandas/core/generic.py", line 2744, in getattr return object.getattribute(self, name) AttributeError: 'DataFrame' object has no attribute 'pos'

TRAINING v9s model

Thanks

[dlindenbaum]

Ok it looks like is is having a problem with reading the labels csv using pandas.

Do you have a csv file in summaryData. you can check to see if it's located correctly.

do you have a file at /data/train/AOI_2_Vegas_Train/summaryData

[eong93]

I don't any csv files nor do I have the summaryData subdirs in the AOI folders.

[dlindenbaum]

Hello,

The csv files that are stored in SummaryData were used as the training labels for the data. It looks like you may have to re-download the data. The solutions in this repository are designed to specifically use the SpaceNet data as packaged for the TopCoder competition.

See [SpaceNet Building Detector Round 2 on TopCoder] (https://community.topcoder.com/longcontest/?module=ViewProblemStatement&rd=16892&pm=14551) for more information about the contest and how to use the data.

There are instructions on how to download the data on that website.

For even more information on accessing the data from AWS please visit https://spacenetchallenge.github.io/

Thank you, Dave

[eong93]

Dave,

Thanks, I will try this

[dannyjeck]

The zip file isn't currently available for download but the directions should be able to be followed from there.

Rather than posting a large file to gihub (I assume file size is the issue), could you put it on s3? This would help avoid the relatively large costs of downloading the entire dataset and training from scratch. This might also fix #5 for many users who just want to replicate the results as well.

[cploveflower]

Hello,@eong93 how do you get the osmdata,is it convenient for you to tell me?

[oblute]

Still wondering here how one could get the osm data for training...the link to it is broken now.

[Maxine3218]

Still wondering here how one could get the osm data for training...the link to it is broken now.

Found this website that also hosts OSM data, don't know if that'll be helpful... https://www.interline.io/osm/extracts/