wiki:PythonCoding

Python Coding Meeting

This page presents notes made during a meeting held at the BADC on the 8th of August 2006 attended by Phil Kershaw, Stephen Pascoe, Ag Stephens, Dominic Lowe and Bryan Lawrence, aimed at identifying how best to integrate the ndg python based software.

The page has also been annotated after the meeting to clearly identify actions, and whether they have been properly hived off to tickets and/or other web pages.

Agenda

  • 11.00 Kick-Off, Agree aims etc
  • 11.15 Walkthroughs (Maximum 12 minutes each and we WILL stick to that)
  • 12.15 Utility Listing, Lunch and Discussion (I reckon 5 minutes each on listing utilities, 45 minutes discussion, 35 minutes lunch, coffee shop sandwiches together)
  • 14.00 14.15 are we on target, is this going to work?
  • 14.15 Dependencies and Issues
  • 15.15 Packaging and Testing
  • 16.15 Amalgamation
  • 16.45 Internal Code Standards and External Documentation.
  • 17.00 Close.

Things we talked about:

  • Ag and Bryan using WSGI ... at least for the front ends ... (now ticket:492)
    • further dependency issue?
  • Issues to do with web service framework packaging ...
  • Need a light interface to exist that respects ndg security ... returns a specific document ...
  • Version of ZSI ...
  • Security Interface Changes ...
  • License Issue
  • Document Strings
  • Building and Setup
  • Error Handling
  • Date Time Utilities
  • Protected/Private? methods & attributes issues
  • New and old style classes ...
  • queuing ... will be an issue for data extractor deployers, we might have a simple workaround (sqimpy), but it wont be part of ndg ...
  • USE DOCSTRINGS
  • elementtree and attributes and xpath
  • unicode issues with elementtree ... ag's deunicode ...

The stubs for the pages in the following box have been created: see CodingStandard and ndg.utils ticket:493 now exists for population.

We need a page each on the wiki for
 * NDG logging (ndg.utils)
 * NDG config files (ndg.utils)
 * NDG Error Handling, relationship with logging ... (ndg.utils)
 * NDG xml parsing and elementtree (ndg.utils)
   * inherit from elementtree rather than composition ...
 * NDG dates times and calenders (ndg.utils)
   * needs to extend the functionality of cdtime, but easier to install ...
     * start time, end time, interval, and get lists ...
 * and corresponding code in the source directory ...

Also a page each on 
 * NDG Look n Feel 
 * NDG unit tests ...
 * NDG documentation strings and CodingStandard

Dependencies

  • Browse
  • SecurityCore
    • zsi
    • elementtree, celementtree
    • pyxmlsec - message level encryption and signature, to be replaced with m2crypto, sha from python, and zsi has a canacolisation ... so these might go away ...
      • python libxml2
        • libxml2
    • m2crypto
      • for signing (currently done with pyxmlsec)
      • will do encryption
      • proxy certificates
  • SecurityApplicationClients?
    • SecurityCore
  • SecurityServers
    • SecurityCore
    • pyopenssl
      • openssl (need the proxy certificate version)
    • myproxy, can we avoid the globus distribution ...
    • sqlobject (optional)
      • database plugins
    • oracle interface needed ...
  • securityClients
    • SecurityCore
  • clients
    • dx and geo
      • zsi
    • bbftp client
    • securityClient
  • CSML
    • dependencies depend on data source
    • cdms-lite (scientific python for writing ...)
    • nappy
    • celementtree and elementtree
    • existWebService
    • if we have data providers who need to create grib control files then
      • wgrib
      • grads
      • grib2ctl.pl
  • DXServer
    • zsi
    • cdms-lite
  • DXClient
    • zsi
    • deployed as a cgi-script ...
  • GeosplatServer?
    • zsi
    • cdat
  • GeosplatClient?
    • zsi
    • deployed as a cgi script
  • zsi ... 2.0!
    • pyxml
    • in general we currently deploy as a socket/port ... and then (should) use proxypass to get it on port 80
  • cdat-lite
    • cdms-lite
      • cdunif
        • ppio
      • netcdf libraries
    • vcs-lite
  • existWebService
    • exist, wsdl
  • discoveryWebService
    • exist
    • postgres
  • ndg-bbftp
    • pybbftp
      • client
      • client-server

Installation Issues

This part of the discussion has now moved to Packages and ticket:494

Setup Issues

This is what our setups will do for

Targets:

  • Site Packages, Default Python Packages, and User Defined Directories.

Layout Directories (assumption is that this wrt /usr or /usr/local/ or userDefinedPath/) :

  • bin - binaries and scripts
  • lib - c-compiled libraries
  • lib/python2.4/site-packages/ndg - all our libraries
  • man
  • etc/ndg - config files
  • doc/ndg - documentation
  • examples

Can we make the application/scripter bundle eggable? Even if the rest is too complicated. (ticket:494)

Deployment issues

we need to think about how to deploy the webservices and cgi/wsgi instances in a consistent manner. ticket:495

SVN Layout

Should mimic the packages that we intend to deploy as bundles, plus branches into working grid in the bundle structure ...

A possibility is

  • wgrid/trunk/src/ndg/dx/client ...
  • wgrid/trunk/src/ndg/dx/install ...
  • wgrid/trunk/src/install/binds together the sub-installs ...

Too hard for now. Stephen will go away and think about this and make a pitch towards the end of September ... We should all move to the new naming structure, and use a common setup philosophy, that Stephen will document earlier than the end of September (ticket:496)

User Based Use Cases

  1. Wants secured large netcdf data, regularly, will script in python.
    • need bbftp client ...
      • so need to talk to session manager, implies need smClient ...
      • need to talk to dx, so need dx command line client
  2. User wants/gets a csml file and a python command line to manipulate content, potentially write a netcdf file, and use own python visualisation tools.
  3. All browser based
  4. User has a netcdf file and wants to do a remote visualisation using geosplat

System Tests

(See ticket:497 to handle these issues).

  1. vmware system in particular state so we can test installs ...
    • Agreed that we're using python setup at least to beta!
  2. User level testing:
    1. we agree that we can't do cross-system testing ... automatically, but we do need some specific human level tests ...
      • assume that hadcm3 control exists in both A and B metadata.
        1. Test user access to browse metadata, and link to data extractor, and login from remote site, and selection of atmospheric parameter T for three timesteps at the surface and get csml and netcdf file returned via bbftp.
          • the last piece of this could be scriptable ...
        2. use a special secure browse document, and test that we can login to browse and go to the data extractor without having to log in ...
    2. test IE, Safari, firefox, konqueror on the latter test.

Attachments