NCEPLIBS-g2c ECMWF GRIB Decode Success with new AEC/CCSDS Library?

trafficstars

All,

With the release of the new AEC/CCSDS Decode Library, I have been trying to decode the public ECMWF GRIB data...with no success.

https://data.ecmwf.int/forecasts/

Function m_split in 'decode.c' keeps responding with "return M_ERROR - 5"

I'm calling g2c_aecunpackf from the fortran gf_unpack.f & I can see all my arguments are passed intact so....

If someone could test out the ECMWF GRIB from the pure C routines to confirm/deny whether it works correctly would be most helpful.

Thanks in advance, Jeff Krob NOAA/NESDIS

Dec 01 '23 11:12 JKrobNESDIS

Hi @JKrobNESDIS, can you pass along or point me to the ECMWF GRIB2 files you are working with? Even if just 1 message from the file?

Dec 01 '23 17:12 EricEngle-NOAA

@JKrobNESDIS I quickly read earlier and did not notice the link. I will download a file and take a look.

Dec 01 '23 20:12 EricEngle-NOAA

I have been looking at this over the last couple of days. I can confirm that this is a bug in the implementation of the AEC compression in the g2c library. I have identified the bug and am working a fix immediately.

@edhartnett would we be able to issue a v1.8.1 patch release?

Dec 05 '23 14:12 EricEngle-NOAA

We can do a new release if you need it.

Dec 15 '23 13:12 edwardhartnett

Edward, Eric,

I've been following this thread & am I correct to presume the issue has been fixed & the ECMWF GRIB data processes correctly? Because I put Eric's changes in 'aecunpack.c' into my code...and still get the same errors returned...just checking.

Thanks, Jeff

On Fri, Dec 15, 2023 at 8:29 AM Edward Hartnett @.***> wrote:

We can do a new release if you need it.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2c/issues/461#issuecomment-1857883497, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBCEF7EIY72GUAIIFEN655DYJRGB3AVCNFSM6AAAAABACU7DOGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJXHA4DGNBZG4 . You are receiving this because you were mentioned.Message ID: @.***>

Dec 15 '23 21:12 JKrobNESDIS

Hi @JKrobNESDIS. Would you be willing to share your code? Or least the portion that does the ECMWF GRIB2 reading/unpacking using g2c?

Dec 17 '23 16:12 EricEngle-NOAA

Hey Eric,

How would I go about doing that without posting on GitHub?

Jeff

On Sun, Dec 17, 2023 at 11:01 AM Eric Engle @.***> wrote:

Hi @JKrobNESDIS https://github.com/JKrobNESDIS. Would you be willing to share your code? Or least the portion that does the ECMWF GRIB2 reading/unpacking using g2c?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2c/issues/461#issuecomment-1859210142, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBCEF7HKZEVFAJ3PXU4BL2LYJ4JODAVCNFSM6AAAAABACU7DOGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJZGIYTAMJUGI . You are receiving this because you were mentioned.Message ID: @.***>

Dec 18 '23 19:12 JKrobNESDIS

Feel free to share with me via NOAA email or through the NOAA Google Drive.

Dec 18 '23 20:12 EricEngle-NOAA

Should this issue be closed?

Jun 17 '24 06:06 edwardhartnett

Hey Edward,

I have yet to get it to work for me with GRIB data from ECMWF. Every GRIB message processed consistently fails in the 'aec_decode' function. Since C is not my 'native language', I have no idea what the issue is. I was working with Eric Engle back in December with the issue...but it kinda fell by the wayside :-/ I don't know if anyone else has had success with this libaec library processing the ECMWF GRIB.

Thanks, Jeff

On Mon, Jun 17, 2024 at 2:58 AM Edward Hartnett @.***> wrote:

Should this issue be closed?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2c/issues/461#issuecomment-2172441404, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBCEF7B7RT4BYWABDGGLOI3ZH2CHZAVCNFSM6AAAAABJNMN7VOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZSGQ2DCNBQGQ . You are receiving this because you were mentioned.Message ID: @.***>

Jun 17 '24 09:06 JKrobNESDIS

Edward,

Details from above:

In function 'int aec_decode', the following 'do' loop is reporting an 'M_ERROR'

do { status = state->mode(strm); } while (status == M_CONTINUE);

Jeff

On Mon, Jun 17, 2024 at 5:35 AM Jeffrey Krob - NOAA Federal < @.***> wrote:

Hey Edward,

I have yet to get it to work for me with GRIB data from ECMWF. Every GRIB message processed consistently fails in the 'aec_decode' function. Since C is not my 'native language', I have no idea what the issue is. I was working with Eric Engle back in December with the issue...but it kinda fell by the wayside :-/ I don't know if anyone else has had success with this libaec library processing the ECMWF GRIB.

Thanks, Jeff

On Mon, Jun 17, 2024 at 2:58 AM Edward Hartnett @.***> wrote:

Should this issue be closed?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2c/issues/461#issuecomment-2172441404, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBCEF7B7RT4BYWABDGGLOI3ZH2CHZAVCNFSM6AAAAABJNMN7VOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZSGQ2DCNBQGQ . You are receiving this because you were mentioned.Message ID: @.***>

Jun 17 '24 09:06 JKrobNESDIS

I will try to devote some time to this in the next couple of weeks. We are spinning up development on NBM v5.0.

Jun 17 '24 11:06 EricEngle-NOAA

Sounds great, thanks for the update! I appreciate the help.

Jeff

On Mon, Jun 17, 2024 at 8:00 AM Eric Engle @.***> wrote:

I will try to devote some time to this in the next couple of weeks. We spinning up development on NBM v5.0.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2c/issues/461#issuecomment-2173196711, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBCEF7BQITIXGTMQU733N3TZH3FTXAVCNFSM6AAAAABJNMN7VOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZTGE4TMNZRGE . You are receiving this because you were mentioned.Message ID: @.***>

Jun 17 '24 15:06 JKrobNESDIS

Any progress on this issue? I'm getting ready to do the next release...

Jul 09 '24 11:07 edwardhartnett

No progress on this -- have other works tasks in the way right now.

Jul 09 '24 14:07 EricEngle-NOAA

@JKrobNESDIS - Can you provide code here where your application interfaces with g2c? If I remember correctly, your application is Fortran?

Jul 09 '24 14:07 EricEngle-NOAA

Hey Eric,

Sure, no problem. However, what is the process for uploading multiple files...or just post the text here? I'll get them sent out this evening.

Thanks again, Jeff

On Tue, Jul 9, 2024 at 10:49 AM Eric Engle @.***> wrote:

@JKrobNESDIS https://github.com/JKrobNESDIS - Can you provide code here where your application interfaces with g2c? If I remember correctly, your application is Fortran?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2c/issues/461#issuecomment-2217931409, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBCEF7HJF5DXLM5AC33ON3TZLPZ7FAVCNFSM6AAAAABJNMN7VOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJXHEZTCNBQHE . You are receiving this because you were mentioned.Message ID: @.***>

Jul 10 '24 17:07 JKrobNESDIS

Yes. Probably best to just put code text here.

Jul 10 '24 23:07 EricEngle-NOAA

Eric...

Just to be clear - you want me to dump aecunpack.c, decode.c, decenc_aec.c, gf_unpack7.f as well as my debug text file here? Kind seems like alot. :-/

Jeff

On Wed, Jul 10, 2024 at 7:28 PM Eric Engle @.***> wrote:

Yes. Probably best to just put code text here.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2c/issues/461#issuecomment-2221699533, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBCEF7C2ANTKAOFCIUVY56DZLW7RXAVCNFSM6AAAAABJNMN7VOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRRGY4TSNJTGM . You are receiving this because you were mentioned.Message ID: @.***>

Jul 10 '24 23:07 JKrobNESDIS

Eric - here is my version of gf_unpack7.f that works fine unpacking all other NCEP GRIB

  subroutine gf_unpack7(cgrib,lcgrib,iofst,igdsnum,igdstmpl,
 &		  	idrsnum,idrstmpl,ndpts,fld,ierr)

interface
    function g2c_aecunpackf(cgrib,lensec,idrstmpl,ndpts,fld)  
 &                     bind(c, name="g2c_aecunpackf")
      use iso_c_binding
      character(kind = c_char), intent(in) :: cgrib(*)
      integer(c_int), value, intent(in) :: lensec
  integer(c_int), intent(in) :: idrstmpl(*)
  integer(c_int), value, intent(in) :: ndpts
      real,intent(out) :: fld(*)
  integer(c_int) :: g2c_aecunpackf
    end function g2c_aecunpackf
end interface


  elseif (idrsnum.eq.42) then
	WRITE (99,*) '*****GF_UNPACK7 lensec-5, ndpts	= ',
 &                 lensec-5, ndpts
  WRITE(99,*)'******* CCSDS/AEC UNCOMPRESS ******'
	
ret = g2c_aecunpackf(cgrib(ipos), lensec-5, idrstmpl, ndpts, fld)

	OPEN (99,FILE='NGRB2PCG32.OUT',ACCESS='APPEND',
 &            STATUS='OLD',IOSTAT=JERR)
	WRITE(99,*)'*****AEC END*******',ret

Thanks, Jeff

Jul 11 '24 00:07 JKrobNESDIS

Hey Eric,

Just a follow-up, are we making any progress on this?

Thanks in advance, Jeff

On Wed, Jul 10, 2024 at 7:49 PM Jeffrey Krob - NOAA Federal < @.***> wrote:

Eric...

Just to be clear - you want me to dump aecunpack.c, decode.c, decenc_aec.c, gf_unpack7.f as well as my debug text file here? Kind seems like alot. :-/

Jeff

On Wed, Jul 10, 2024 at 7:28 PM Eric Engle @.***> wrote:

Yes. Probably best to just put code text here.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2c/issues/461#issuecomment-2221699533, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBCEF7C2ANTKAOFCIUVY56DZLW7RXAVCNFSM6AAAAABJNMN7VOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRRGY4TSNJTGM . You are receiving this because you were mentioned.Message ID: @.***>

Jul 24 '24 09:07 JKrobNESDIS

@JKrobNESDIS - I have been looking into this over the past few days. When I use grib2io (Python), which links to g2c through a Cython extension module, I do not have any decode/unpack issues when reading from AEC encoded ECMWF GRIB2 files. Here an screenshot example of this in a Jupyter notebook

Screenshot 2024-07-24 at 7 54 37 AM

I am thinking that the Fortran interface to g2_aecunpackf might not be configured correctly. Looking at the function definition from grib2.h

int g2c_aecunpackf(unsigned char *cpack, size_t len, int *idrstmpl, size_t ndpts, float *fld);

Try the following Fortran interface:

interface
  function g2c_aecunpackf(cpack, lensec, idrstmpl, ndpts, fld) bind(C, name="g2c_aecunpackf")
    use, intrinsic :: iso_c_binding
    implicit none
    ! Arguments
    character(kind=c_char), intent(in), dimension(*) :: cpack   ! Equivalent to unsigned char *
    integer(kind=c_size_t), value :: lensec
    integer(kind=c_int), dimension(*), intent(inout) :: idrstmpl
    integer(kind=c_size_t), value :: ndpts
    real(kind=c_float), dimension(*), intent(inout) :: fld
    ! Return type
    integer(kind=c_int) :: g2c_aecunpackf
  end function g2c_aecunpackf
end interface

which only differs from your version by making lecsec and ndpts type c_size_t and I think on most systems, the c size_t type is unsigned long.

Jul 24 '24 13:07 EricEngle-NOAA

Thanks, I'll check it out & let you know.

Jeff

https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free.www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Wed, Jul 24, 2024 at 9:25 AM Eric Engle @.***> wrote:

@JKrobNESDIS https://github.com/JKrobNESDIS - I have been looking into this over the past few days. When I use grib2io (Python), which links to g2c through a Cython extension module, I have do not have any decode/unpack issues when reading from AEC encoded ECMWF GRIB2 files. Here an screenshot example of this in a Jupyter notebook

Screenshot.2024-07-24.at.7.54.37.AM.png (view on web) https://github.com/user-attachments/assets/b4f373ab-6e6b-4c9b-ad2c-02e9f878693f

I am thinking that the Fortran interface to g2_aecunpackf might not be configured correctly. Looking at the function definition from grib2.h

int g2c_aecunpackf(unsigned char *cpack, size_t len, int *idrstmpl, size_t ndpts, float *fld);

Try the following Fortran interface:

interface function g2c_aecunpackf(cpack, lensec, idrstmpl, ndpts, fld) bind(C, name="g2c_aecunpackf") use, intrinsic :: iso_c_binding implicit none ! Arguments character(kind=c_char), intent(in), dimension() :: cpack ! Equivalent to unsigned char * integer(kind=c_size_t), value :: lensec integer(kind=c_int), dimension(), intent(inout) :: idrstmpl integer(kind=c_size_t), value :: ndpts real(kind=c_float), dimension(*), intent(inout) :: fld ! Return type integer(kind=c_int) :: g2c_aecunpackf end function g2c_aecunpackf end interface

which only differs from your version by making lecsec and ndpts type c_size_t and I think on most systems, the c size_t type is unsigned long.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2c/issues/461#issuecomment-2247929841, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBCEF7DFGHVSHBWIHYJTWSDZN6TLLAVCNFSM6AAAAABJNMN7VOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBXHEZDSOBUGE . You are receiving this because you were mentioned.Message ID: @.***>

Jul 24 '24 20:07 JKrobNESDIS

@JKrobNESDIS - Sorry that is has taken me a few days to reply here. Last week I was able to get an example Fortran program working related to this issue. Here is the example program.

program test
   use grib_mod
   implicit none

   type(gribfield) :: gfld

   character(len=:), allocatable :: filename

   integer :: j, k, ifile, iret
   integer :: jdisc, jpdtn, jgdtn
   integer :: firstval, lastval
   real :: fldmax, fldmin
   logical :: unpack = .true.

   integer, dimension(200) :: jids, jpdt, jgdt

   j = 0
   k = 0
   ifile = 10
   iret = 0
   filename = "./20231201000000-24h-oper-fc.grib2"

   ! Open GRIB2 file
   call baopenr(ifile, filename, iret)
   print *, "iret from baopenr = ", iret

   ! Set GRIB2 field identification values to search for
   jdisc = -1
   jids(:) = -9999
   jpdtn = -1
   jpdt(:) = -9999
   jgdtn = -1
   jgdt(:) = -9999

   do

      ! Get field from file
      call getgb2(ifile, 0, j, jdisc, jids, jpdtn, jpdt, jgdtn, jgdt, unpack, k, gfld, iret)
      print *, "iret from getgb2 = ", iret
      if (iret .ne. 0) exit
      print *, "j, k = ", j, k

      print *, "data representation template number = ", gfld%idrtnum

      ! Process field ...
      if (unpack) then
         firstval = gfld%fld(1)
         lastval = gfld%fld(gfld%ndpts)
         fldmax = maxval(gfld%fld)
         fldmin = minval(gfld%fld)
         print *, "min, max of data = ", fldmin, fldmax
      end if

      ! Free memory when done with field
      call gf_free(gfld)
      print *, "Free gfld..."

      ! Set j to k in order to advance to next message
      j = k

   end do

   call gf_finalize()
   print *, "Free g2 library memory..."

end program test

The program is using a modified local version of gf_unpack7 which contains an interface to g2c's aecunpackf() defined as:

interface
   function g2c_aecunpackf(cgrib,lensec,idrstmpl,ndpts,fld) bind(c, name="g2c_aecunpackf")
      use iso_c_binding
      character(kind=c_char), intent(in), dimension(*) :: cgrib   ! Equivalent to unsigned char *
      integer(c_int), value, intent(in) :: lensec
      integer(kind=c_int), dimension(*), intent(inout) :: idrstmpl
      integer(kind=c_size_t), value :: ndpts
      real(kind=c_float), dimension(*), intent(inout) :: fld
      integer(kind=c_int) :: g2c_aecunpackf
   end function g2c_aecunpackf
end interface

and is called in gf_unpack7 as like this

ret = g2c_aecunpackf(cgrib(ipos), lensec-5, idrstmpl, int(ndpts,kind=8), fld)

Note that I did not need any modified local copies of any of the AEC C source files from g2c. This program runs successfully on macOS and Linux (Fedora 40).

I am wondering about the following in your situation:

How was g2c and g2 built on your Windows system? MS compilers or Intel?
What Fortran compiler are you using for the program?

Aug 06 '24 13:08 EricEngle-NOAA

Hey Eric,

Thanks for the reply. I'll work with this over the next few days & let you know how it goes.

How was g2c and g2 built on your Windows system? MS compilers or Intel?

Microsoft Visual C++ 2022

What Fortran compiler are you using for the program?

Intel oneAPI Fortran Classic 2024.1

Thanks again,

Jeff

On Tue, Aug 6, 2024 at 9:12 AM Eric Engle @.***> wrote:

@JKrobNESDIS https://github.com/JKrobNESDIS - Sorry that is has taken me a few days to reply here. Last week I was able to get an example Fortran program working related to this issue. Here is the example program.

program test use grib_mod implicit none

type(gribfield) :: gfld

character(len=:), allocatable :: filename

integer :: j, k, ifile, iret integer :: jdisc, jpdtn, jgdtn integer :: firstval, lastval real :: fldmax, fldmin logical :: unpack = .true.

integer, dimension(200) :: jids, jpdt, jgdt

j = 0 k = 0 ifile = 10 iret = 0 filename = "./20231201000000-24h-oper-fc.grib2"

! Open GRIB2 file call baopenr(ifile, filename, iret) print *, "iret from baopenr = ", iret

! Set GRIB2 field identification values to search for jdisc = -1 jids(:) = -9999 jpdtn = -1 jpdt(:) = -9999 jgdtn = -1 jgdt(:) = -9999

do
  ! Get field from file
  call getgb2(ifile, 0, j, jdisc, jids, jpdtn, jpdt, jgdtn, jgdt, unpack, k, gfld, iret)
  print *, "iret from getgb2 = ", iret
  if (iret .ne. 0) exit
  print *, "j, k = ", j, k

  print *, "data representation template number = ", gfld%idrtnum

  ! Process field ...
  if (unpack) then
     firstval = gfld%fld(1)
     lastval = gfld%fld(gfld%ndpts)
     fldmax = maxval(gfld%fld)
     fldmin = minval(gfld%fld)
     print *, "min, max of data = ", fldmin, fldmax
  end if

  ! Free memory when done with field
  call gf_free(gfld)
  print *, "Free gfld..."

  ! Set j to k in order to advance to next message
  j = k
end do

call gf_finalize() print *, "Free g2 library memory..." end program test

The program is using a modified local version of gf_unpack7 which contains an interface to g2c's aecunpackf() defined as:

interface function g2c_aecunpackf(cgrib,lensec,idrstmpl,ndpts,fld) bind(c, name="g2c_aecunpackf") use iso_c_binding character(kind=c_char), intent(in), dimension() :: cgrib ! Equivalent to unsigned char * integer(c_int), value, intent(in) :: lensec integer(kind=c_int), dimension(), intent(inout) :: idrstmpl integer(kind=c_size_t), value :: ndpts real(kind=c_float), dimension(*), intent(inout) :: fld integer(kind=c_int) :: g2c_aecunpackf end function g2c_aecunpackf end interface

and is called in gf_unpack7 as like this

ret = g2c_aecunpackf(cgrib(ipos), lensec-5, idrstmpl, int(ndpts,kind=8), fld)

Note that I did not need any modified local copies of any of the AEC C source files from g2c. This program runs successfully on macOS and Linux (Fedora 40).

I am wondering about the following in your situation:

How was g2c and g2 built on your Windows system? MS compilers or Intel?

What Fortran compiler are you using for the program?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2c/issues/461#issuecomment-2271261104, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBCEF7DWYQUZOLHSLBHKNB3ZQDDS5AVCNFSM6AAAAABJNMN7VOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZRGI3DCMJQGQ . You are receiving this because you were mentioned.Message ID: @.***>

Aug 06 '24 23:08 JKrobNESDIS

No problem. There could be an incompatibility between g2c (built with MSVC++) and g2 (Intel Fortran) and your program (Intel Fortran) what does not show itself until program execution.

Aug 07 '24 12:08 EricEngle-NOAA

@EricEngle-NOAA are you submitting those g2c changes as a PR on g2c?

Aug 07 '24 14:08 edwardhartnett

@edwardhartnett - As of now there are no changes to g2c needed. The AEC compression is working correctly from within the g2c library. Changes are definitely needed in NCEPLIBS-g2, but we can perhaps discuss that in an issue for g2, likely in https://github.com/NOAA-EMC/NCEPLIBS-g2/issues/705 or https://github.com/NOAA-EMC/NCEPLIBS-g2/issues/458. IMO, this specific issue can be closed.

Aug 07 '24 14:08 EricEngle-NOAA

NCEPLIBS-g2c NCEPLIBS-g2c copied to clipboard

ECMWF GRIB Decode Success with new AEC/CCSDS Library?

NCEPLIBS-g2c
NCEPLIBS-g2c copied to clipboard