wiki:PythonCoding

Version 13 (modified by lawrence, 13 years ago) (diff)

--

Python Coding Meeting

  • 11.00 Kick-Off, Agree aims etc
  • 11.15 Walkthroughs (Maximum 12 minutes each and we WILL stick to that)
  • 12.15 Utility Listing, Lunch and Discussion (I reckon 5 minutes each on listing utilities, 45 minutes discussion, 35 minutes lunch, coffee shop sandwiches together)
  • 14.00 14.15 are we on target, is this going to work?
  • 14.15 Dependencies and Issues
  • 15.15 Packaging and Testing
  • 16.15 Amalgamation
  • 16.45 Internal Code Standards and External Documentation.
  • 17.00 Close.

Things we talked about:

  • Ag and Bryan using WSGI ... at least for the front ends ...
  • Issues to do with web service framework packaging ...
  • Need a light interface to exist that respects ndg security ... returns a specific document ...
  • Version of ZSI ...
  • Security Interface Changes ...
  • License Issue
  • Document Strings
  • Building and Setup
  • Error Handling
  • Date Time Utilities
  • Protected/Private? methods & attributes issues
  • New and old style classes ...
  • queuing ...
  • USE DOCSTRINGS
  • elementtree and attributes and xpath
  • unicode issues with elementtree ... ag's deunicode ...

We need a page each on the wiki for

  • NDG logging (ndg.utils)
  • NDG config files (ndg.utils)
  • NDG Error Handling, relationship with logging ... (ndg.utils)
  • NDG xml parsing and elementtree (ndg.utils)
    • inherit from elementtree rather than composition ...
  • NDG dates times and calenders (ndg.utils)
    • needs to extend the functionality of cdtime, but easier to install ...
      • start time, end time, interval, and get lists ...
  • and corresponding code in the source directory ...

Also a page each on

  • NDG Look n Feel
  • NDG unit tests ...
  • NDG documentation strings and code style

Dependencies

  • Browse
  • SecurityCore
    • zsi
    • elementtree, celementtree
    • pyxmlsec - message level encryption and signature, to be replaced with m2crypto, sha from python, and zsi has a canacolisation ... so these might go away ...
      • python libxml2
        • libxml2
    • m2crypto
      • for signing (currently done with pyxmlsec)
      • will do encryption
      • proxy certificates
  • SecurityApplicationClients?
    • SecurityCore
  • SecurityServers
    • SecurityCore
    • pyopenssl
      • openssl (need the proxy certificate version)
    • myproxy, can we avoid the globus distribution ...
    • sqlobject (optional)
      • database plugins
    • oracle interface needed ...
  • securityClients
    • SecurityCore
  • clients
    • dx and geo
      • zsi
    • bbftp client
    • securityClient
  • CSML
    • dependencies depend on data source
    • cdms-lite (scientific python for writing ...)
    • nappy
    • celementtree and elementtree
    • existWebService
    • if we have data providers who need to create grib control files then
      • wgrib
      • grads
      • grib2ctl.pl
  • DXServer
    • zsi
    • cdms-lite
  • DXClient
    • zsi
    • deployed as a cgi-script ...
  • GeosplatServer?
    • zsi
    • cdat
  • GeosplatClient?
    • zsi
    • deployed as a cgi script
  • zsi ... 2.0!
    • pyxml
    • in general we currently deploy as a socket/port ... and then (should) use proxypass to get it on port 80
  • cdat-lite
    • cdms-lite
      • cdunif
        • ppio
      • netcdf libraries
    • vcs-lite
  • existWebService
    • exist, wsdl
  • discoveryWebService
    • exist
    • postgres
  • ndg-bbftp
    • pybbftp
      • client
      • client-server

Installation Issues

Bundles

  • NDG-Discovery Bundle
    • discoveryWebService
    • packages: ndg.utils ndg.discovery
  • Full Data Provider Server Bundle
    • securityServer
    • securityApplicationClient
    • csml
    • bbftp server and client
    • browse
    • dxServer and Client
    • geosplat server and client
    • loggingWebService
    • util
    • existWebService
    • packages: ndg.utils ndg.csml ndg.delivery ndg.browse ndg.dx ndg.geosplat ndg.security ndg.cdat-lite
      • with client and server sub packages where appropriate, and vcs and cdms sub packages under cdat lite ...
  • NDG-Lite Bundle for application developers and scripters
    • securityApplicationClient
    • bbftp
    • commandLine dx
    • csml
    • packages: ndg.utils ndg.csml ndg.dx.client ndg.security.client ndg.delivery.client ndg.geosplat.client

Setup Issues

This is what our setups will do for

Targets:

  • Site Packages, Default Python Packages, and User Defined Directories.

Layout Directories (assumption is that this wrt /usr or /usr/local/ or userDefinedPath/) :

  • bin - binaries and scripts
  • lib - c-compiled libraries
  • lib/python2.4/site-packages/ndg - all our libraries
  • man
  • etc/ndg - config files
  • doc/ndg - documentation
  • examples

Can we make the application/scripter bundle eggable?

Deployment issues

we need to think about how to deploy the webservices and cgi/wsgi instances in a consistent manner.

SVN Layout

Should mimic the packages that we intend to deploy as bundles, plus branches into working grid in the bundle structure ...

A possibility is

  • wgrid/trunk/src/ndg/dx/client ...
  • wgrid/trunk/src/ndg/dx/install ...
  • wgrid/trunk/src/install/binds together the sub-installs ...

Too hard for now. Stephen will go away and think about this and make a pitch towards the end of September ...

User Based Use Cases

  1. Wants secured large netcdf data, regularly, will script in python.
    • need bbftp client ...
      • so need to talk to session manager, implies need smClient ...
      • need to talk to dx, so need dx command line client
  2. User wants/gets a csml file and a python command line to manipulate content, potentially write a netcdf file, and use own python visualisation tools.
  3. All browser based
  4. User has a netcdf file and wants to do a remote visualisation using geosplat

System Tests

  1. vmware system in particular state so we can test installs ...

Attachments