pyspedas
pyspedas copied to clipboard
spedas
doesn't load in STATIC energy / phi / theta / etc data
I was trying to use pyspedas to pull in STATIC data but I noticed large discrepancies in the data pulled by PySPEDAS vs pulling data in IDL. Looking at the CDF file, it seems like pyspedas is ignoring the file's metadata and just pulling variables/support variables. Unfortunately, for STATIC, important parameters (like energy, phi, theta, etc) are given in metadata rather than variables/support variables.
Please enable reading in of the metadata parameters so STATIC data can be used to the full extent in python as well as in IDL.
Thank you!
I'm having the same issue as @NeeshaRS. I'm trying to pull STATIC's energy, mass and eflux data. I'm looking forward to this fix! Thank you!
I was looking over this with Neesa yesterday, and the problem is the Energy bins (labeled as "energy" inside the file) have an attribute VAR_TYPE of "metadata", rather than "data" or "support_data". I assume that's actually a mistake within the STATIC files, because energy should definitely be considered "support_data". But alas, there is no way they're going to change the file format at this point.
The cdf_to_tplot routine assumes that all things labeled as metadata are either options to input into tplot (ylog, spec, title, etc), or they are not needed. However in this case, we'd want to read in the metadata because its crucial to understanding the data in the file.
As a quick fix, on the mpl and master branches of pytplot, I added a "get_metadata" option to the cdf_to_tplot routine so that it at least reads in the data so they can access the energy bins. I'm not sure if there is a much better way of handling it, but that's what I've done for now.
Thanks Bryan! I'll go ahead and update the STATIC routine to use the new keyword automatically (and release a new version of the MPL pytplot so it all just works)
Oops - didn't mean to close this, though it should be fixed now as of v1.3.12
BUT the original problem we asked about does seem to have been fixed-- thank you!
and the array looks reasonable at first glance. will dig in a bit more to better compare with what IDL pulls. 
From the error, it looks like the eflux variable has too many dimensions. Is it a function of angle as well as energy?
One thing to note: in IDL SPEDAS, we have ~135k lines of code in the 'maven' folder alone, so there are many, many support routines that haven't been ported over yet just for MAVEN. It's possible that one of these routines does some calculations with these eflux variables prior to loading (e.g., averaging over angle) - but I haven't checked yet.
I looked into this some more; from the IDL crib sheet:
https://github.com/spedas/bleeding_edge/blob/master/projects/maven/sta/mvn_sta_l2_crib.pro
It looks like it loads the data from the CDF files with mvn_sta_l2_load, then produces tplot variables with mvn_sta_l2_tplot:
https://github.com/spedas/bleeding_edge/blob/master/projects/maven/sta/mvn_sta_l2_tplot.pro
So this file should have what you need to produce the tplot variables.
The eflux is indeed multi-dimensional, so you'll definitely have to do some post-processing to create the variables you're interested in, e.g., from mvn_sta_l2_tplot.pro:
store_data,'mvn_sta_c0_H_E',data={x:time,y:reform((eflux[*,*,1]-0.006*eflux[*,*,0]/(1.-(data[*,*,0]/1200.<.9))) >0.),v:energy}
Yeah, in IDL I use those functions and work it that way.
@juditbergfalk and I were just surprised because that function in python used to work and now it doesn't. I will dig into it a bit too (a lot of meetings today so I'll be in & out in my poking of things)... but in IDL when tplots are multidimensional, if they are 3D (like in your example) it plots a spectrogram. As far as I can tell, tplots are never more than 3D. Thanks for all your help with this!!!
Yeah, in IDL I use those functions and work it that way.
@juditbergfalk and I were just surprised because that function in python used to work and now it doesn't. I will dig into it a bit too (a lot of meetings today so I'll be in & out in my poking of things)... but in IDL when tplots are multidimensional, if they are 3D (like in your example) it plots a spectrogram. As far as I can tell, tplots are never more than 3D. Thanks for all your help with this!!!
Always happy to help!
In IDL tplot, if the variable is multi-dimensional, it'll just use the first element of the last dimension, e.g., if eflux were [time, energy, angle], the plot would be an energy vs. time spectrogram for the first angle only.
I agree with @NeeshaRS, the original problem is fixed. Thank you for that!
However, I keep getting a new error when running pytplot.tplot: "Problem with the number of line colors specified." and it doesn't open Tplot in a new window, rather in the notebook itself.
The error code is not showing if I don't have any pytplot.options specificed (which I'll need) but the plots are still showing up in the notebook. I hope it's an easy fix!
I agree with @NeeshaRS, the original problem is fixed. Thank you for that! However, I keep getting a new error when running pytplot.tplot: "Problem with the number of line colors specified." and it doesn't open Tplot in a new window, rather in the notebook itself.
Do you have a simple example that would allow me to reproduce the problem?
Here is an example:
import numpy as np
import pytplot # PyTplot
import pyspedas
import matplotlib.colors as colors # Create new colormap for matplotlib plot
# Color maps for spectrum plots
cmap = colors.LinearSegmentedColormap.from_list("", ["#edf8fb", "#bfd3e6", "#9ebcda", "#8c96c6", "#8856a7", "#810f7c"])
### Time
timeinsec = np.arange(1.53351360e9,1.53360000e9,1)
utc_inst = ['2018-08-06', '2018-08-06']
### SWIA download and create PyTplot variables
swi_vars = pyspedas.maven.swia(trange = [utc_inst[0], utc_inst[1]],
level = 'l2',
datatype = ['onboardsvymom'],
varformat = None,
get_support_data = False,
auto_yes = True,
downloadonly = False)
### SWIA Temperature
# Extract y-data
Temp_xyz_mvn = pytplot.get_data('temperature_mso_onboardsvymom').y
# Extract time
Temp_time_mvn = pytplot.get_data('temperature_mso_onboardsvymom').times
# Define |T|
Temp_total_mvn = []
for t in Temp_xyz_mvn:
Ttot = np.sqrt(t[0]**2 + t[1]**2 + t[2]**2) # Calculate total temperature
Temp_total_mvn.append(Ttot)
Temp_total_mvn = np.array(Temp_total_mvn)
# Define PyTplot variable
pytplot.store_data('MAVEN_SWIA_Tx', data={'x':Temp_time_mvn, 'y':Temp_xyz_mvn.T[0]})
pytplot.store_data('MAVEN_SWIA_Ty', data={'x':Temp_time_mvn, 'y':Temp_xyz_mvn.T[1]})
pytplot.store_data('MAVEN_SWIA_Tz', data={'x':Temp_time_mvn, 'y':Temp_xyz_mvn.T[2]})
pytplot.store_data('MAVEN_SWIA_T_total', data={'x':Temp_time_mvn, 'y':Temp_total_mvn})
pytplot.store_data('MAVEN_SWIA_T_MSO', data=['MAVEN_SWIA_Tx', 'MAVEN_SWIA_Ty', 'MAVEN_SWIA_Tz', 'MAVEN_SWIA_T_total'])
### SWEA download and create PyTplot variables
swe_vars = pyspedas.maven.swea(trange = [utc_inst[0], utc_inst[1]],
level = 'l2',
datatype = ['svyspec'], # Omni-directional energy spectra in units of differential energy flux (eV/cm2 sec ster eV) from SWEA survey data
varformat = None,
get_support_data = False, # get_support_data = True doesn't work! "AttributeError: 'dict' object has no attribute 'attrs'"
auto_yes = True,
downloadonly = False)
# Global options
pytplot.tplot_options('axis_font_size', 12) # Axis tick size
pytplot.tplot_options('axis_tick_num', [(0,1)]) # Reduces axis ticks
### PyTplot
# MAVEN: SWIA Ion temperature
pytplot.options('MAVEN_SWIA_T_MSO', 'ytitle', 'MAVEN - SWIA')
pytplot.options('MAVEN_SWIA_T_MSO', 'ysubtitle', 'Ion Temperature (km/s)')
pytplot.options('MAVEN_SWIA_T_MSO', 'legend_names', ['Tx','Ty','Tz','Ttotal'])
pytplot.options('MAVEN_SWIA_T_MSO', 'Color', [(69,117,180),(144,195,118),(215,48,39),(0,0,0)])
pytplot.tplot(['MAVEN_SWIA_T_MSO'])
# MAVEN: SWEA Spectrum
pytplot.options('diff_en_fluxes_svyspec', 'Colormap', cmap)
pytplot.options('diff_en_fluxes_svyspec', 'ylog', 1)
pytplot.options('diff_en_fluxes_svyspec', 'zlog', 1)
pytplot.options('diff_en_fluxes_svyspec', 'spec', 1)
pytplot.options('diff_en_fluxes_svyspec', 'ytitle', 'Energy (eV)')
pytplot.options('diff_en_fluxes_svyspec', 'ztitle', 'Diff En Flux (eV/cm^2/s/sr/eV)')
pytplot.tplot('diff_en_fluxes_svyspec', slice=True)
Thanks for the example! I just checked in a fix for the crash with get_support_data=True - I'll do another release of pytplot today to get this fix into PyPI.
As for the error with colors: I don't think matplotlib supports the color format you're using for the SWIA T pseudo-variable :
pytplot.options('MAVEN_SWIA_T_MSO', 'Color', [(69,117,180),(144,195,118),(215,48,39),(0,0,0)])
When I try one of these on a normal tplot variable, I get the error:
ValueError: (69, 117, 180) is not a valid value for color
Using the color formats specified in:
https://matplotlib.org/stable/tutorials/colors/colors.html
should fix the crash you're seeing; e.g., using:
pytplot.options('MAVEN_SWIA_T_MSO', 'Color', ['purple', 'green', 'red', 'black'])
worked for me.
One more note: axis_tick_num hasn't been implemented in the matplotlib version of pytplot yet, so setting that with tplot_options won't do anything.
Hope this helps!
Thanks for the example! I just checked in a fix for the crash with
get_support_data=True- I'll do another release of pytplot today to get this fix into PyPI.As for the error with colors: I don't think matplotlib supports the color format you're using for the SWIA T pseudo-variable :
pytplot.options('MAVEN_SWIA_T_MSO', 'Color', [(69,117,180),(144,195,118),(215,48,39),(0,0,0)])When I try one of these on a normal tplot variable, I get the error:
ValueError: (69, 117, 180) is not a valid value for colorUsing the color formats specified in:
https://matplotlib.org/stable/tutorials/colors/colors.html
should fix the crash you're seeing; e.g., using:
pytplot.options('MAVEN_SWIA_T_MSO', 'Color', ['purple', 'green', 'red', 'black'])worked for me.
One more note:
axis_tick_numhasn't been implemented in the matplotlib version of pytplot yet, so setting that withtplot_optionswon't do anything.Hope this helps!
Thank you for your help always! The main issue for me is that pytplot isn't opening in a new window and I'm not sure if that's related to these other issues with color and get_support_data. If you have a second today I can hop on zoom and show you what I mean. It's always harder to explain through messages. I'm curious if you run the code above, does it work for you?
What environment are you running in? If you're running it locally, so each call to pytplot.tplot will pause the script until you close the tplot window. Running inside a Jupyter notebook shouldn't have this issue.
If you're running locally and want to create multiple plots without having to close the tplot window, I suggest setting the 'display' keyword to False and saving the figures using save_png (or save_svg/save_pdf/save_eps).
I was able to run your script on Colab:
https://colab.research.google.com/drive/15_8696zPkkBBB4FKCuFvP8Y_cL35Oc92?usp=sharing
after changing the colors on the SWIA temp variable.
Note: the 'slice' option to tplot also isn't implemented yet for the matplotlib version of pytplot.
were you two able to sync up? @juditbergfalk had recommended doing a zoom with @supervised -- would that be possible? I feel like a 15min zoom would quickly get you both on the same page
Thank you @NeeshaRS . I agree, a Zoom meeting would probably a good idea if you're available @supervised . I'm just a little bit confused because both the slice and the colors worked before the update.
Thank you @NeeshaRS . I agree, a Zoom meeting would probably a good idea if you're available @supervised . I'm just a little bit confused because both the slice and the colors worked before the update.
The backend to PyTplot (the entire part that renders figures) has been completely re-written from the ground up over the last 8 months; the version that comes with PySPEDAS now uses a completely different library internally (matplotlib vs. Qt previously). The slice keyword isn't there because it hasn't been implemented in the new version yet, and the color keyword doesn't accept the previous values because the way of specifying colors in matplotlib is different than Qt. You can find more information here:
https://github.com/MAVENSDC/PyTplot/issues/159
I'll look into adding the slice functionality back.
I can setup a Zoom meeting if you need, would you be available Thursday morning at 11AM Mountain time?
Oh okay, that makes sense now. Sorry for not understanding. I guess I should have updated it. I was really pleased to see the slice option and was even telling @NeeshaRS how cool of a tool it is to have so I'm looking forward to see that again but I understand if it'll be a lot of work! Thank you for your help and patience! I believe the zoom meeting won't be needed. I'll just keep an eye out for the updates.
Oh okay, that makes sense now. Sorry for not understanding. I guess I should have updated it. I was really pleased to see the slice option and was even telling @NeeshaRS how cool of a tool it is to have so I'm looking forward to see that again but I understand if it'll be a lot of work! Thank you for your help and patience! I believe the zoom meeting won't be needed. I'll just keep an eye out for the updates.
The latest release of the matplotlib version of pytplot (v2.0.9 and later) now supports the 'slice' keyword; you can upgrade with:
pip install --upgrade pytplot-mpl-temp
Hope this helps!