Archive Custom Data Sources

[cjmelick]

I am trying to add in a custom dataset where the archived model BUFKIT sounding doesn't have a server --- I am just adding a local data file. I keep feeling like I am missing some steps. Do I need to update Python scripts or other configuration files that aren't the two mentioned in the instructions?

This is the error I get when running the GUI: Well, this is embarrassing. SHARPpy broke. This is probably due to an issue with one of the data source servers, but if it keeps happening, send the detailed information to the developers.

Detailed Information: SHARPpy v1.3.0 Xenia Crash time: 2019-04-16 16:09:36.128783 Traceback (most recent call last): File "full_gui.py", line 54, in doCrasher ret = func(*args, **kwargs) File "full_gui.py", line 661, in createWindow return Main() File "full_gui.py", line 537, in init self.__initUI() File "full_gui.py", line 543, in __initUI self.picker = Picker(self.config, parent=self) File "full_gui.py", line 96, in init self.data_sources = data_source.loadDataSources() File "/p/home/melickc/projsharppy/anaconda2/lib/python2.7/site-packages/SHARPpy-1.3.0-py2.7.egg/datasources/data_source.py", line 39, in loadDataSources root = ET.parse(ds_file).getroot() File "/p/home/melickc/projsharppy/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1182, in parse tree.parse(source, parser) File "/p/home/melickc/projsharppy/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 657, in parse self._root = parser.close() File "/p/home/melickc/projsharppy/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1671, in close self._raiseerror(v) File "/p/home/melickc/projsharppy/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror raise err ParseError: no element found: line 9, column 0

Thanks, Chris

[wblumberg]

Hi Chris,

This looks to be an issue with the XML file that you've modified. Usually this is due to some formatting issue in the XML file, as this error indicates that the XML reader can't correctly parse your file. What does your file say on line 9, column 0? You also may want to check this website and see if you have the same error: https://stackoverflow.com/questions/23001757/python-elementtree-no-element-found-exception

If it's a local directory of BUFKIT files you will probably be okay just modifying the XML and making sure you have the correct CSV that tells SHARPpy the places where you have sounding files.

• Greg

[cjmelick]

Hi Greg,

I have an XML file called customdata.xml and the contents are only like 9 lines:

<datasource name="GALWEM" ensemble="false" observed="false">

    <outlet name="Local" url="{date}{cycle}_galwem_{srcid}.buf" format=

"bufkit" >

        <time range="240" delta="1" offset="12" delay="0" cycle="12"

archive="24"/>

        <points csv="galwem.csv" />

    </outlet>

</datasource>

~

My custom CSV file (galwem.csv) only has one station given (KOFF):

icao,iata,synop,name,state,country,lat,lon,elev,priority,srcid

KOFF,OFF,,,NE,US,41.12,-95.91,340.6,,koff

The sample BUFKIT data file is in the datasources directory.

Do I need to modify anything?

Thanks,

Chris

On Fri, May 3, 2019 at 3:52 PM Greg Blumberg [email protected] wrote:

Hi Chris,

This looks to be an issue with the XML file that you've modified. Usually this is due to some formatting issue in the XML file, as this error indicates that the XML reader can't correctly parse your file. What does your file say on line 9, column 0? You also may want to check this website and see if you have the same error: https://stackoverflow.com/questions/23001757/python-elementtree-no-element-found-exception

If it's a local directory of BUFKIT files you will probably be okay just modifying the XML and making sure you have the correct CSV that tells SHARPpy the places where you have sounding files.

• Greg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sharppy/SHARPpy/issues/164#issuecomment-489236051, or mute the thread https://github.com/notifications/unsubscribe-auth/AL3DNGTOYWNJ7UX5ZRBZACLPTSQX7ANCNFSM4HGLDRWA .

-- Christopher J. Melick, PhD

[wblumberg]

Hi Chris,

I got quite busy for the last month and my time is getting a little more free. Thank you for your patience!

The biggest thing I see is that I think you need to change your url tag to be an absolute path rather than a relative path and put your sample BUFKIT file outside the datasources directory. My instinct says that using a relative path for the url tag will lead to problems.

When setup.py install is run, it places only the .xml, .csv, .py files it needs into your ~/sharppy folder, so your .buf files are probably not being copied over into this folder from /datasources when this command is run.

Greg

[cjmelick]

Hi Greg,

No worries. Thanks for getting back to me.

I tried your suggestion but I am getting the same error message when I run the GUI.

Here is my updated customdata.xml file contents:

<datasource name="GALWEM" ensemble="false" observed="false">

    <outlet name="Local" url=

"/p/home/melickc/projsharppy/SHARPpy/19040512_galwem_koff.buf" format= "bufkit" >

        <time range="24" delta="1" offset="12" delay="0" cycle="12"

archive="24"/>

        <points csv=

"/p/home/melickc/projsharppy/SHARPpy/datasources/galwem.csv" />

    </outlet>

</datasource>

I have spelled out the full path URL and was explicit on the file name. I even tried to spell out the path to the csv file. I modified the time range value to be 24 because there were only 24 separate time elements in the BUFKIT file.

Have other ideas for modifications? Is there a way to run the python script in debug mode?

Thanks, Chris

On Tue, Jun 4, 2019 at 5:13 PM Greg Blumberg [email protected] wrote:

Hi Chris,

I got quite busy for the last month and my time is getting a little more free. Thank you for your patience!

The biggest thing I see is that I think you need to change your url tag to be an absolute path rather than a relative path and put your sample BUFKIT file outside the datasources directory. My instinct says that using a relative path for the url tag will lead to problems.

When setup.py install is run, it places only the .xml, .csv, .py files it needs into your ~/sharppy folder, so your .buf files are probably not being copied over into this folder from /datasources when this command is run.

Greg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sharppy/SHARPpy/issues/164?email_source=notifications&email_token=AL3DNGQGVGWLHYOJZ3XUIP3PY3SJPA5CNFSM4HGLDRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW6AXJY#issuecomment-498863015, or mute the thread https://github.com/notifications/unsubscribe-auth/AL3DNGS2QMIE4MUYBFSOQL3PY3SJPANCNFSM4HGLDRWA .

-- Christopher J. Melick, PhD

[wblumberg]

@tsupinie, any ideas?

[tsupinie]

My suggestion would be to double check your ~/.sharppy/datasources/customdata.xml file to make sure it has the </sourcelist> tag. Missing that in your first example was probably the cause of that error. Running setup.py should copy it over, but maybe manually copy it over, if that doesn't work for some reason?

[cjmelick]

Tim,

You are correct. Once I added the tag you suggested to customdata.xml in the one directory you mentioned, the error message I was receiving went away. However, the GUI was having processing the GALWEM.csv file I created maybe because there was only one station listed (KOFF). I only provided that one entry because the Bufkit file only had data for that station. However, after I copied over the GFS csv file to the GALWEM one without any changes, the GUI produced no errors and I was able to launch soundings. I am curious now, though, how is it possible I am able to get results for stations I have no data for? I noticed the archive time listed on the profile is correct but I am receiving current times in the drop-down menu. Also, I had to give a URL location instead of directory path to the data. I was able to accomplish this but it would be nice to not have to do that again on a regular basis.

Thanks, Chris

On Tue, Jun 11, 2019, 1:12 AM Tim Supinie [email protected] wrote:

My suggestion would be to double check your ~/.sharppy/datasources/customdata.xml file to make sure it has the tag. Missing that in your first example was probably the cause of that error. Running setup.py should copy it over, but maybe manually copy it over, if that doesn't work for some reason?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sharppy/SHARPpy/issues/164?email_source=notifications&email_token=AL3DNGRF5YJ2VGW7JOTEEZ3PZ464NA5CNFSM4HGLDRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXMBTBI#issuecomment-500701573, or mute the thread https://github.com/notifications/unsubscribe-auth/AL3DNGXIMO4B2BI6I56TJS3PZ464NANCNFSM4HGLDRWA .

[cjmelick]

I figured out the problem with the entry in the custom csv file for the one sounding location --- I needed to round the station elevation to the nearest integer and not use a decimal point. Still, I am kind of confused as to why I was getting soundings for other stations when I didn't have any data in the BUFKIT file for anything other than Offutt AFB.

On Tue, Jun 11, 2019 at 9:48 AM Chris Melick [email protected] wrote:

Tim,

You are correct. Once I added the tag you suggested to customdata.xml in the one directory you mentioned, the error message I was receiving went away. However, the GUI was having processing the GALWEM.csv file I created maybe because there was only one station listed (KOFF). I only provided that one entry because the Bufkit file only had data for that station. However, after I copied over the GFS csv file to the GALWEM one without any changes, the GUI produced no errors and I was able to launch soundings. I am curious now, though, how is it possible I am able to get results for stations I have no data for? I noticed the archive time listed on the profile is correct but I am receiving current times in the drop-down menu. Also, I had to give a URL location instead of directory path to the data. I was able to accomplish this but it would be nice to not have to do that again on a regular basis.

Thanks, Chris

On Tue, Jun 11, 2019, 1:12 AM Tim Supinie [email protected] wrote:

My suggestion would be to double check your ~/.sharppy/datasources/customdata.xml file to make sure it has the tag. Missing that in your first example was probably the cause of that error. Running setup.py should copy it over, but maybe manually copy it over, if that doesn't work for some reason?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sharppy/SHARPpy/issues/164?email_source=notifications&email_token=AL3DNGRF5YJ2VGW7JOTEEZ3PZ464NA5CNFSM4HGLDRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXMBTBI#issuecomment-500701573, or mute the thread https://github.com/notifications/unsubscribe-auth/AL3DNGXIMO4B2BI6I56TJS3PZ464NANCNFSM4HGLDRWA .

-- Christopher J. Melick, PhD