datavis-hackathon
datavis-hackathon copied to clipboard
NSF-Polar-Cyberinfrastructure
Antarctic Meteorology Research Center (AMRC) datasets
The AMRC at University of Wisconsin-Madison studies the weather in Antarctic in two ways; Automatic Weather Station (AWS) data and satellite composite imagery. For this workshop, a dataset has been prepared that contains five formats of AWS data and two formats of infrared satellite composites for one month May, 2014. This dataset can be found via our ftp site (ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/)
Here are the first few lines of ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/AWS_q3h_MAY_2014/ag4201405q3h.txt :
Year: 2014 Month: 05 ID: AG4 ARGOS: 8927 Name: AGO-4
Lat: 82.01S Lon: 96.76E Elev: 3597m
2014 121 5 1 0000 -50.9 608.6 9.1 223.0 444.0 444.0
2014 121 5 1 0300 -52.6 609.3 8.9 222.0 53.2 444.0
2014 121 5 1 0600 -53.9 610.2 7.3 226.0 52.0 444.0
Can you give a short description of what each column represents? Do all of the formats have the same data, just in a different format? Would it be interesting to see a heat map over time for one of these columns?
Thanks!
The data is all from the same time period, but different time scales. Also, the infrared satellite images are in area and netcdf format. I added more to the readme file (ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/readme.txt) to explain the different formats.
In terms of visualization, it would be great if we could display the satellite images in the database and create meteorograms real-time. For example, http://amrc.ssec.wisc.edu/data/view-data.php?action=view_image&product=surface/awsmeteorograms/8928.GIF
Thank you for expanding the readme!
display the satellite images in the database and create meteorograms real-time
So, this would take a directory of files in AREA or NetCDF format, extract the images within, then display them to the user? The meteorograms are a way of displaying the other temp, pressure, etc data over time?
looking at the meteorograms, I wonder if there is a way to use http://d3js.org/ to generate meteorograms interactively? very cool @ccostanza
I'm assuming we would use the netCDF format because that can be read by a large variety of software, and then display them to the user. Yes the meteorograms are a great way of displaying the atmospheric variables over time. We use McIDAS to make our meteorograms, but I'm sure other programs would work too.
I would just like to add that having our data in some type of polar database is the goal. Creating theses display would be a great addition for the visualization of our data.
d3 would be one option. Another is to try using Bokeh -- seems to be an easy way to display and interact with data. An initial idea of steps needed is to pull down any changes from the website, parse out the data and images, display the data with d3/Bokeh along side the images. Bokeh is not as easy to integrate into a site as d3, though. There is also Heatmap JS to display... heat maps.
As the data is in columned based text format, it can be easily queried by Spark SQL and visualized with Bokeh as Tyler suggested.
On Oct 7, 2014, at 9:35 PM, Chris Mattmann [email protected] wrote:
Bokeh is awesome, great idea too @tpalsulich
— Reply to this email directly or view it on GitHub.
I don't think I could bring any expertise to this issue, but as somebody who has used some Antarctic meteorology data, I think this sounds really interesting! Being able to visualize which data are available when for a given region (interactively) could be useful.
I have thoughts on visualization, because I think combining point and spatial data visually as an access point to a data portal would be useful. What I imagine is an interactive map- the base layer of which is the remote sensing image with a slider that allows you to select imagery back through time, like NASA worldview or google earth- and then one of the overlays is the point data set for stations- when you click on a point a pop out window has plotted parameters and station metadata. This could correspond the time frame around the image currently displayed/ have their own time sliders. And then, ideally, you could select the data you are interested in- point or spatial- and export it. I've been thinking about this in terms of Greenland- i.e. when MODIS shows a shift in snow reflectance, it would be useful to access station data between the two scenes to see what happening on the ground- and how much quicker would that analysis be if that data was all in one place and easily discoverable? A version of this might already exist somewhere and I am unaware of it!
Colleagues of mine at Kitware are in the process of developing an open-source web solution that could help with some of what you are looking for. geoJS, located at https://github.com/OpenGeoscience, is working toward an interactive, animated, multi-layer map web map. We can talk about this more at the hackathon. As you are probably familiar, wide area imagery is generally preprocessed into tile sets, so we might create “sets” of MODIS tiles binned by timeframe. The app would select which time window of imagery to display. Then vector or measurement data could be overlaid on top.
On Oct 27, 2014, at 2:36 AM, smskiles [email protected] wrote:
I have thoughts on visualization, because I think combining point and spatial data visually as an access point to a data portal would be useful. What I imagine is an interactive map- the base layer of which is the remote sensing image with a slider that allows you to select imagery back through time, like NASA worldview or google earth- and then one of the overlays is the point data set for stations- when you click on a point a pop out window has plotted parameters and station metadata. This could correspond the time frame around the image currently displayed/ have their own time sliders. And then, ideally, you could select the data you are interested in- point or spatial- and export it. I've been thinking about this in terms of Greenland- i.e. when MODIS shows a shift in snow reflectance, it would be useful to access station data between the two scenes to see what happening on the ground- and how much quicker would that analysis be if that data was all in one place and easily discoverable? A version of this might already exist somewhere and I am unaware of it!
— Reply to this email directly or view it on GitHub.
@smskiles that would be great way to incorporate both pieces of data! I'm looking forward to hearing all of the suggesting in a couple of days
Another idea I was thinking about - to be able to visualize what data are available when are something like these ray diagrams (in this case showing lifetimes, but could be dates of data availability, with color indicating what types of data are available, etc. Also, the 360-degree example could be overlayed on a map of Antarctica indicating longitude? This doesn't integrate with other data (as @smskiles suggested) but could be a way to know what data are available in one diagram...
Ex1: http://oliveruberti.files.wordpress.com/2013/04/lifelines.jpg Ex2: http://theinformationcapital.com/wp-content/uploads/2014/09/BluePlaques_spread1.jpg
@allenpope @ccostanza checked out the ray tracing examples. I think this could be something that Tangelo and D3 could easily support (cc @curtislisle and @MBoustani ) sounding like we have something emerging here..
I created a script to pull down the ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/AWS_q3h_MAY_2014 data into a JSON file. This should be an easier data format to work with, at least for Javascript web apps.
If there is interest, this could be adapted to the other data formats, too!
We were able to talk through making a visualization of the availability of automatic weather station data. This then lead to the ideas around making a better dataset location that could be searchable using Solr and then other software to do a visualization of the data. @darth-pr made many interesting suggestions! With the help of @allenpope we were able to make an interesting idea of a map to display the availability of the data and potentially exporting the data that is needed.
The data can be found here: ftp://amrc.ssec.wisc.edu/pub/aws/q1h/
This is Quality Controlled 1 hourly data. The format is as follows:
These files contain 1 hourly AWS data that has been quality controlled using our interactive IDL programs. The files are labelled with a three letter code to indicate the station, the year, the month, and the frequency of the data. (IIIYYYYMMq1h.txt for 1 hourly data)
Column 1 - year Column 2 - julian day Column 3 - month Column 4 - day Column 5 - ten minute observation time Column 6 - temperature (C) Column 7 - pressure (mb) Column 8 - wind speed (m/s) Column 9 - wind direction Column 10- relative humidity (%) Column 11- delta-T (C)
@tpalsulich is this JSON format already acceptable into Solr? Was talking with @ccostanza about this and thought what if we did an full ETL of this data into Solr.
Here is our little diagram. @ccostanza should be able to make sense of it, but the gist is being able to get out a spatial and temporal comparison of where AWS data are available.
ftp://amrc.ssec.wisc.edu/pub/aws/antrdr/ for .dat files
ftp://amrc.ssec.wisc.edu/pub/aws/q3h/ for 3 hourly files