wiki:TI12_Security/EggifyingNDGsecurity

Version 3 (modified by pjkersha, 13 years ago) (diff)

--

Eggifying NDG Security

NDG packages security included must be as easy to install as possible if they are to get take up by users and institutions.

The use of MyProxy, set-up of PKI and the number of 3rtd party python package dependencies makes the security installation more complicated. As of the Alpha version, security had a distutils based setup. This made installation of the python source more straightforward but does not address the issue of the installation of the required 3rd party packages. Also, there is a need to separate server and client side code. The client side is more light weight and so making this distinct makes the installation process much less onerous for those only interested in client side functionality e.g. application developers wishing to NDG-secure a resource.

Use of Python Eggs (See  http://peak.telecommunity.com/DevCenter/PythonEggs and  http://peak.telecommunity.com/DevCenter/setuptools) should address some of these issues:

  • setup scripts configured with the external dependencies they need and with the ability to automatically download and install these
  • separate client and server side eggs

NDG Security Installation Status as of Alpha Stage

  • flat package structure: a single NDG package containing all the code
  • distutils based setup to install it
  • Separate installation required for MyProxy, Globus !SimpleCA client package and OpenSSL and for 3rd party python dependencies.

How to make NDG Security Installation Easier

  • Separate eggs for client and server
  • eggs have dependencies set for the 3rd party python packages they need so that these are found and automatically installed on setup
  • Keep MyProxy, Globus packages and OpenSSL install processes separate otherwise too complex. - Egg setups could however be configured to detect whether these are in place and fail or give a warning if not there.

Proposed Package Structure

At the Python code review (Packages) we agreed a hierarchical package structure. Applying this to security instead of having a single NDG package we now have ndg.security.

Also separate client and server packages would be useful for security. For example,

  1. a data provider wants to deploy an Attribute Authority but doesn't need session management. They install the ndg.security.server.attAuthority egg.
  2. Bryan's situation: he's developing the Discovery CGI. He's only interested in client bindings to security services so he wants a ndg.security.client egg.

For client and server installations there are now separate client and server packages. However, there a common package is also needed because some modules are required by both. Finally, the Attribute Authority, Session Manager and Simple CA services are distinct and so can be divided into separate packages [Log and Gatekeeper may be added]. The resulting structure is,

ndg/
    security/
        client/
            AttAuthority/
            SessionMgr/
        common/
        server/
            AttAuthority/
            SessionMgr/
            ca/

Special Namespace Packages

Python egg setuptools enables a special kind of package known as a namespace package. This can be thought of simply as a placeholder to support a required package structure. It comes into play in the above since ndg and security don't actually need any modules but they are important in preserving the structure of the hierarchy and allowing distinct namespaces for NDG python software e.g.

  • ndg.security - NDG Security namespace
  • ndg.utils - NDG utilities namespace
  • ndg.dx - namespace for NDG Data Extractor

Sub dividing Security into separate Eggs

setuptools modifies and extends the distutils setup.py concept. A setup.py creates a .egg file. (.eggs use zip file format and can be accessed using unzip). This has consequences for the directory structure under the SVN trunk/. There needs to be separate arrangements of the ndg/security directory structure for each of the eggs we want to define. The structure below illustrates:

trunk/
    python/
        setup.py (contains settings to make egg for ''whole'' of NDG security)

        ndgSecurityServer/
            setup.py (contains settings to make server egg)
            ndg/
                __init__.py (with special namespace package declaration)
                security/
                    __init__.py (with namespace package declaration)
                    server/
                        __init__.py (no namespace package declaration)
                        ... server sub packages and modules *.py ...

        ndgSecurityClient/
            setup.py (contains settings to make client egg)
            ndg/
                __init__.py (with namespace package declaration)
                security/
                    __init__.py (with namespace package declaration)
                    client/
                        __init__.py (no namespace package declaration)
                        ... client sub packages and modules *.py ...

        ndgSecurityCommon/
            setup.py (contains settings to make client egg)
            ndg/
                __init__.py (with namespace package declaration)
                security/
                    __init__.py (with namespace package declaration)
                    common/
                        __init__.py (no namespace package declaration)
                        ... common sub packages and modules *.py ...
  • python/setup.py makes the overall NDG security egg - the kitchen sink :)
  • python/ndgSecurityServer/setup.py makes the NDG server egg. It is has setting to make it dependent on the common/ egg because this code is also needed. When setup.py is run, it will get and install the python/ndgSecurityServer\common egg.
  • python/ndgSecurityCommon/setup.py makes the common egg. It can be thought of as a virtual egg as no one would need to install it on it's own. It's only ever installed by virtue of it being a dependency for the client and server eggs.
  • python/ndgSecurityClient/setup.py makes the client egg. The same applies here as with the server egg: it is dependent on the common egg.

This makes for an unwieldy directory structure but is consistent with the idea that each egg should be independently downloadable.

Making setup.py Settings

Keywords to the setup function specify the packages to be included in the egg, namespace packages, and other dependencies

client/setup.py:

setup(
  name = 'ndg_security_client',
  ...
  install_requires = ['ndg_security_common'],
  namespace_packages = ['ndg', 'ndg.security'],
  packages = ['ndg.security.client', ...],
  ...
)

server/setup.py:

setup(
  name = 'ndg_security_server',
  ...
  Install_requires = ['ndg_security_common'],
  namespace_packages = ['ndg', 'ndg.security'],
  packages = ['ndg.security.server', ...],
  ...
)

common/setup.py:

setup(
  name = 'ndg_security_common',
  ...
  install_requires = [], # plus any other dependencies
  namespace_packages = ['ndg', 'ndg.security'],
  packages = ['ndg.security.common', ...],
  ...
)

When Dependencies Go Wrong

Third party python packages required by Python can be specified by giving their special name as used in the Python Cheese Shop. However, this may not work and it may be necessary to give setup.py the explicit URL where a particular package can be downloaded. Also, some packages may not install correctly or not follow the rules of distutils. To be completed... mention setup.cfg.

Setup for Data files, Configuration files and Scripts

To be completed...