wiki:PythonCoding

Version 18 (modified by lawrence, 13 years ago) (diff)

--

Python Coding Meeting

  • 11.00 Kick-Off, Agree aims etc
  • 11.15 Walkthroughs (Maximum 12 minutes each and we WILL stick to that)
  • 12.15 Utility Listing, Lunch and Discussion (I reckon 5 minutes each on listing utilities, 45 minutes discussion, 35 minutes lunch, coffee shop sandwiches together)
  • 14.00 14.15 are we on target, is this going to work?
  • 14.15 Dependencies and Issues
  • 15.15 Packaging and Testing
  • 16.15 Amalgamation
  • 16.45 Internal Code Standards and External Documentation.
  • 17.00 Close.

Things we talked about:

  • Ag and Bryan using WSGI ... at least for the front ends ...
    • further dependency issue?
  • Issues to do with web service framework packaging ...
  • Need a light interface to exist that respects ndg security ... returns a specific document ...
  • Version of ZSI ...
  • Security Interface Changes ...
  • License Issue
  • Document Strings
  • Building and Setup
  • Error Handling
  • Date Time Utilities
  • Protected/Private? methods & attributes issues
  • New and old style classes ...
  • queuing ... will be an issue for data extractor deployers, we might have a simple workaround (sqimpy), but it wont be part of ndg ...
  • USE DOCSTRINGS
  • elementtree and attributes and xpath
  • unicode issues with elementtree ... ag's deunicode ...

We need a page each on the wiki for

  • NDG logging (ndg.utils)
  • NDG config files (ndg.utils)
  • NDG Error Handling, relationship with logging ... (ndg.utils)
  • NDG xml parsing and elementtree (ndg.utils)
    • inherit from elementtree rather than composition ...
  • NDG dates times and calenders (ndg.utils)
    • needs to extend the functionality of cdtime, but easier to install ...
      • start time, end time, interval, and get lists ...
  • and corresponding code in the source directory ...

Also a page each on

  • NDG Look n Feel
  • NDG unit tests ...
  • NDG documentation strings and CodingStandard

Dependencies

  • Browse
  • SecurityCore
    • zsi
    • elementtree, celementtree
    • pyxmlsec - message level encryption and signature, to be replaced with m2crypto, sha from python, and zsi has a canacolisation ... so these might go away ...
      • python libxml2
        • libxml2
    • m2crypto
      • for signing (currently done with pyxmlsec)
      • will do encryption
      • proxy certificates
  • SecurityApplicationClients?
    • SecurityCore
  • SecurityServers
    • SecurityCore
    • pyopenssl
      • openssl (need the proxy certificate version)
    • myproxy, can we avoid the globus distribution ...
    • sqlobject (optional)
      • database plugins
    • oracle interface needed ...
  • securityClients
    • SecurityCore
  • clients
    • dx and geo
      • zsi
    • bbftp client
    • securityClient
  • CSML
    • dependencies depend on data source
    • cdms-lite (scientific python for writing ...)
    • nappy
    • celementtree and elementtree
    • existWebService
    • if we have data providers who need to create grib control files then
      • wgrib
      • grads
      • grib2ctl.pl
  • DXServer
    • zsi
    • cdms-lite
  • DXClient
    • zsi
    • deployed as a cgi-script ...
  • GeosplatServer?
    • zsi
    • cdat
  • GeosplatClient?
    • zsi
    • deployed as a cgi script
  • zsi ... 2.0!
    • pyxml
    • in general we currently deploy as a socket/port ... and then (should) use proxypass to get it on port 80
  • cdat-lite
    • cdms-lite
      • cdunif
        • ppio
      • netcdf libraries
    • vcs-lite
  • existWebService
    • exist, wsdl
  • discoveryWebService
    • exist
    • postgres
  • ndg-bbftp
    • pybbftp
      • client
      • client-server

Installation Issues

Bundles

  • NDG-Discovery Bundle
    • discoveryWebService
    • packages: ndg.utils ndg.discovery
  • Full Data Provider Server Bundle
    • securityServer
    • securityApplicationClient
    • csml
    • bbftp server and client
    • browse
    • dxServer and Client
    • geosplat server and client
    • loggingWebService
    • util
    • existWebService
    • packages: ndg.utils ndg.csml ndg.delivery ndg.browse ndg.dx ndg.geosplat ndg.security ndg.cdat-lite
      • with client and server sub packages where appropriate, and vcs and cdms sub packages under cdat lite ...
  • NDG-Lite Bundle for application developers and scripters
    • securityApplicationClient
    • bbftp
    • commandLine dx
    • csml
    • packages: ndg.utils ndg.csml ndg.dx.client ndg.security.client ndg.delivery.client ndg.geosplat.client ndg.cdat-lite

Setup Issues

This is what our setups will do for

Targets:

  • Site Packages, Default Python Packages, and User Defined Directories.

Layout Directories (assumption is that this wrt /usr or /usr/local/ or userDefinedPath/) :

  • bin - binaries and scripts
  • lib - c-compiled libraries
  • lib/python2.4/site-packages/ndg - all our libraries
  • man
  • etc/ndg - config files
  • doc/ndg - documentation
  • examples

Can we make the application/scripter bundle eggable? Even if the rest is too complicated.

Deployment issues

we need to think about how to deploy the webservices and cgi/wsgi instances in a consistent manner.

SVN Layout

Should mimic the packages that we intend to deploy as bundles, plus branches into working grid in the bundle structure ...

A possibility is

  • wgrid/trunk/src/ndg/dx/client ...
  • wgrid/trunk/src/ndg/dx/install ...
  • wgrid/trunk/src/install/binds together the sub-installs ...

Too hard for now. Stephen will go away and think about this and make a pitch towards the end of September ... We should all move to the new naming structure, and use a common setup philosophy, that Stephen will document earlier than the end of September

User Based Use Cases

  1. Wants secured large netcdf data, regularly, will script in python.
    • need bbftp client ...
      • so need to talk to session manager, implies need smClient ...
      • need to talk to dx, so need dx command line client
  2. User wants/gets a csml file and a python command line to manipulate content, potentially write a netcdf file, and use own python visualisation tools.
  3. All browser based
  4. User has a netcdf file and wants to do a remote visualisation using geosplat

System Tests

  1. vmware system in particular state so we can test installs ...
    • Agreed that we're using python setup at least to beta!
  2. User level testing:
    1. we agree that we can't do cross-system testing ... automatically, but we do need some specific human level tests ...
      • assume that hadcm3 control exists in both A and B metadata.
        1. Test user access to browse metadata, and link to data extractor, and login from remote site, and selection of atmospheric parameter T for three timesteps at the surface and get csml and netcdf file returned via bbftp.
          • the last piece of this could be scriptable ...
        2. use a special secure browse document, and test that we can login to browse and go to the data extractor without having to log in ...
    2. test IE, Safari, firefox, konqueror on the latter test.

Attachments