Ticket #30 (closed enhancement: fixed)

Opened 6 years ago

Last modified 6 years ago

gribapi python package

Reported by: spascoe Owned by: spascoe
Priority: major Component: JASMIN Analysis Platform
Keywords: Cc:

Description

IRIS requires the "gribapi" module to be installed to read grib files. The IRIS documentation links to this page: https://software.ecmwf.int/wiki/display/GRIB/Releases

We need to build an RPM for this.

Change History

comment:1 Changed 6 years ago by iwi

I did not build any RPMs. I had already previously downloaded gribapi binary RPMs from ECMWF, and have now signed these and added them to JASMIN-repo and installed on jasmin-sci1. There are some GRIB rpms already in the standard repo, apparently packaged a bit differently (by Fedora), but I have installed the ones packaged by ECMWF, that are a later version:

grib_api-fortran-1.9.18-1.x86_64
grib_api-1.9.18-1.x86_64
grib_api-devel-1.9.18-1.x86_64
grib_api-python-1.9.18-1.x86_64

With yum install, it was necessary to specify explicitly grib_api-devel.x86_64 for some reason, but for the other three above, the .x86_64 was not needed; I'm not sure why this is.

For information, here are the Fedora ones (not used):

grib_api.i686                   1.9.16-3.el6       epel-tier1                   
grib_api.x86_64                 1.9.16-3.el6       epel-tier1                   
grib_api-debuginfo.x86_64       1.9.16-3.el6       epel-tier1                   
grib_api-devel.i686             1.9.16-3.el6       epel-tier1                   
grib_api-devel.x86_64           1.9.16-3.el6       epel-tier1                   
grib_api-static.i686            1.9.16-3.el6       epel-tier1                   
grib_api-static.x86_64          1.9.16-3.el6       epel-tier1                   

By the way, the emos package, also in JASMIN-repo, contains the old GRIBEX library. You probably won't use this, but I installed it on the builder as a build-time dependency for umutil.

Handing this ticket back to you, so that you can add the above to your spreadsheet of what's installed, and then close the ticket. Actually, having said that, I now see as of a few days ago there is now version 1.10.0 available at https://software.ecmwf.int/wiki/display/GRIB/Red+Hat+Enterprise+Linux+6.0+x86_64 so if you want the very latest then give me back the ticket.

Last edited 6 years ago by iwi (previous) (diff)

comment:2 Changed 6 years ago by iwi

  • Status changed from new to assigned
  • Owner changed from iwi to spascoe

comment:3 Changed 6 years ago by spascoe

  • Owner changed from spascoe to iwi

There are two problems to resolve:

  1. The grib_api-python package doesn't create the file /usr/lib64/python2.6/site-packages/grib_api/init.py, therefore the "grib_api" python package is not importable
  2. We need this package build for python2.7.

therefore we will need to rebuild the grib_api package for python2.7 and fix the missing init.py file.

I have added an import test to the suite to flag this problem.

comment:4 Changed 6 years ago by spascoe

Iris expects the gribapi package to be importable as

import gribapi

Therefore the wrapping used in the current rpm won't work. It puts this module inside the package "grib_api". I.e.

from grib_api import gribapi

comment:5 Changed 6 years ago by spascoe

The RPMs on the ECMWF site are marked as experimental so maybe it's not surprising they don't work. I have emailed ECMWF to see if they can fix them or provide us with the SPEC/SRPMs. Alan is CC'd

comment:6 Changed 6 years ago by iwi

Will wait for their reply before trying to build any RPM for python2.7, as having a SRPM as a starting point may make the difference between needing a minor tweak for 2.7 versus a chunk of work.

comment:7 Changed 6 years ago by spascoe

ECMWF has created an issue at https://software.ecmwf.int/issues/browse/SUP-409

They then closed it as a duplicate but I can't see the new ticket. I've asked them whether they want to collaborate on fixing the RPM build or whether we should just do our own.

comment:8 Changed 6 years ago by spascoe

From SUP-409:

The spec files are included in the .tar.gz distribution so you can have a look there if you want. In the end I think it would be positive to use the same RPM files or at least specs, but I don't think we will be able to fix and release this for at least a couple of months as it is just medium priority and developers here have a huge backlog of high priority stuff.

For me the ideal way of fixing it would be in the .tar.gz distribution itself, without having to implement any "magic" in the spec file and keeping consistency between direct source compilation and RPMs. But if you need something quick it may just do to add a couple of commands to the spec file. Then you can run

$ make rpms

on the distribution directory and that should be all.

We should make minimal patches to the tarball to fix, build our own RPMs then feed the patch back to ECMWF.

comment:9 Changed 6 years ago by iwi

  • Owner changed from iwi to spascoe

I have generated minimal patches that:

  • provide an __init.py__, although the contents will probably still need tweaking (at the moment it just has the one line: "from gribapi import *")
  • add that __init.py__ to the distribution when the RPM is built
  • change the package name of the python bindings to automatically be based on the name of the RPM that provides the relevant python, so for our python 2.7 environment you get grib_api-python27 instead of grib_api-python. This would allow grib_api-python and grib_api-python27 to coexist if required.
  • per previous point, also add the relevant python base package as a dependency of the python bindings RPM.
  • change the site-packages directory used for the installation, so that instead of just assuming it's under what the build calls $(libdir) (which here is /usr/lib64), it tests sys.path in the relevant python. This is because our base python27 package has /usr/lib/python2.7/site-packages/ (as it's based closely on a vanilla installation, and use of /usr/lib64 would have required additional patching).

It works if I apply the patches to the vanilla grib_api-1.10.0.tar.gz tarball using:

patch -p1 < /path/to/gribapi-python-requires.diff 
patch -p1 < /path/to/gribapi-python-site-packages-dir.diff 
patch -p1 < /path/to/gribapi-python-init.diff 
automake

and then compile using:

export PYTHON=/usr/bin/python2.7
./configure --with-pic --enable-python --prefix=/usr
make rpms

However, this generates (as well as the binary RPMS) also a source RPM containing the spec file and a new tarball, also called grib_api-1.10.0.tar.gz, but in fact containing the version with my patches. This is potentially confusing, and breaks the philosophy that RPMs should contain pristine sources. So what I've done is to rebuild both source and binary RPMs from the following:

  • the original tarball
  • my patches as separate files alongside
  • the spec file extracted from the source RPM that "make rpms" produced, but edited so that at build time it applies the patches explicitly (and also runs automake and sets $PYTHON) as above

The rebuilt binaries are then I think essentially the same as those already generated by "make rpms", but the new corresponding source RPM has the patches kept separate, which will make maintenance easier until such time as they are incorporated into the distribution.

I am passing the ticket back to you for advice regarding the necessary contents of __init.py__ (see above), which I will provide as part of the patch gribapi-python-init.diff

After that, please pass the ticket to me and then I can rebuild the RPMs incorporating your __init.py__.

comment:10 Changed 6 years ago by spascoe

We don't need init.py at all.

Just make sure the binary RPM creates these:

/usr/lib/python2.7/site-packages/gribapi.py
/usr/lib/python2.7/site-packages/gribapi_swig.py
/usr/lib/python2.7/site-packages/_gribapi_swig.so

or /usr/lib64 or course. Notice there is no container package.

If I goto /home/builderdev/rpmbuild/BUILD/grib_api-1.10.0/python and type python setup.py build I get the correct structure in the build directory. There must be something in the SPEC which put's it inside a container package.

comment:11 follow-up: ↓ 12 Changed 6 years ago by spascoe

Try changing this in python/Makefile.in (could also change it in Makefile.am and run automake)

--- Makefile.in.orig	2013-03-06 11:55:38.000000000 +0000
+++ Makefile.in	2013-03-06 11:57:57.000000000 +0000
@@ -275,7 +275,7 @@
 
 # What I want installed
 @WITH_PYTHON_TRUE@site_packages_dir = $(shell $(PYTHON) -c "import sys, os; print filter(lambda d: os.path.basename(d) == 'site-packages', sys.path)[0]")
-@WITH_PYTHON_TRUE@pdir = $(site_packages_dir)/grib_api
+@WITH_PYTHON_TRUE@pdir = $(site_packages_dir)
 @WITH_PYTHON_TRUE@p_DATA = \
 @WITH_PYTHON_TRUE@	__init__.py \
 @WITH_PYTHON_TRUE@	_gribapi_swig.so \

comment:12 in reply to: ↑ 11 Changed 6 years ago by spascoe

The patch gribapi-python-site-packages-dir.diff already addresses these lines, so I'm creating gribapi-python-site-packages-dir-2.diff and trying it out.

Replying to spascoe:

Try changing this in python/Makefile.in (could also change it in Makefile.am and run automake)

--- Makefile.in.orig	2013-03-06 11:55:38.000000000 +0000
+++ Makefile.in	2013-03-06 11:57:57.000000000 +0000
@@ -275,7 +275,7 @@
 
 # What I want installed
 @WITH_PYTHON_TRUE@site_packages_dir = $(shell $(PYTHON) -c "import sys, os; print filter(lambda d: os.path.basename(d) == 'site-packages', sys.path)[0]")
-@WITH_PYTHON_TRUE@pdir = $(site_packages_dir)/grib_api
+@WITH_PYTHON_TRUE@pdir = $(site_packages_dir)
 @WITH_PYTHON_TRUE@p_DATA = \
 @WITH_PYTHON_TRUE@	__init__.py \
 @WITH_PYTHON_TRUE@	_gribapi_swig.so \

comment:13 Changed 6 years ago by spascoe

  • Status changed from assigned to closed
  • Resolution set to fixed

I have built grib-api-*-1.ceda.el16, put them on yumit and installed on sci1-dev and sci1. They install and the SciVM test suite runs with the new code (including Iris).

I am closing this and creating a new ticket to report to ECMWF.

comment:14 Changed 6 years ago by spascoe

See ticket:46 for reporting to ECMWF.

Note: See TracTickets for help on using tickets.