source: TI01-discovery/trunk/ingestAutomation/OAIBatch/retired/README @ 3779

This directory  contains a python script (and some java ) to handle difs after harvesting.
 
Under this directory the following structure should be maintained:


./data
        - /DATACENTRE/
·               - discovery/:           Records with namespace, schema declaration deleted. After having run the script. Ready to ingest in the discovery service.
·               - oai/difYYYYMMDD/      Records as harvested from OAI.

Where  /DATACENTRE  varies to the different data providers

- cd OAIBatch/data/DATACENTRE/oai
- mkdir difYYYMMDD  (This directory will only maintain a copy of the difs in case the script  rewrites something wrong) 

HARVEST_HOME = The harvested_records directory 

cp  HARVEST_HOME/*.xml OAIBatch/data/datacentre/oai/difYYYYMMDD
cp  HARVEST_HOME/*.xml OAIBatch/data/datacentre/discovery

The file config.properties contains the name=value pair to parse the filename. Define under the property name oai_host  the string to eliminate from the filename. 

cat config.properties

        #### config.properties #######
# Define host_OAI as the string that OAI adds to the filenames after harvesting
# String added by OAI for BODC, SOC, NCAR 
# BODC = oai%3Agrid.bodc.nerc.ac.uk%3A
# SOC = oai%3Aoai.noc.soton.ac.uk%3A
# NCAR = oai%3Aucar.ncar.scd.cdp%3A

host_OAI=oai%3Agrid.bodc.nerc.ac.uk%3A

                ###########


Execute the script 
python oaiProc.py ./data/DATACENTRE/discovery/*

The script reads the files from OAIBatch/data/datacentre/discovery and outputs within the same directory the files.
The result will get rid of the "oai%3Aucar.ncar.scd.cdp%3A" that oai adds to
the filenames and it will leave <DIF> as the root element.

Once the script has finished: 
        
- cd /usr/local/WSClients/OAIBatch/data/datacentre/discovery
- rm oai*