source: TI01-discovery/branches/ingestAutomation-upgrade/OAIBatch/run_all_ingest.py @ 3839

Subversion URL: http://proj.badc.rl.ac.uk/svn/ndg/TI01-discovery/branches/ingestAutomation-upgrade/OAIBatch/run_all_ingest.py@3839
Revision 3839, 1.4 KB checked in by cbyrom, 11 years ago (diff)

Add script to run ingest for all avaiable config files.
Make oai_ingest_new2 a proper object.
Adjust db_funcs - now pass in details to set up database connection

  • although defaults available, if not done.

Simplify coord parsing in PostgresRecord? by using a reusable function.
+ various tidy ups and fixes.

  • Property svn:executable set to *
Line 
1#!/usr/bin/env python
2"""
3"""
4import os, sys, logging
5from oai_ingest_new2 import oai_ingest
6
7run_counter = 0
8
9lineSeparator = "-----------------------------"
10print lineSeparator
11print "RUNNING: run_all_ingest.py"
12print lineSeparator
13
14# configure logging
15logging.basicConfig(level=logging.DEBUG,
16                    format='%(asctime)s %(filename)s:%(lineno)d %(levelname)s %(message)s')
17
18
19current_dir = os.getcwd() + "/"# this is the base dir that the script is ran from
20logging.info("Running ingest script for all config files in the current run directory (%s)" %current_dir)
21
22
23# iterate over all files in the current directory and process any config ones
24config_suffix = '_config.properties'
25filenames = os.listdir(current_dir + 'datacentre_config/')
26for filename in filenames:
27        if filename.endswith(config_suffix):
28            logging.info("Found config file: %s" %filename)
29           
30            # get the datacentre from the filename
31            datacentre = filename.replace(config_suffix, '')
32            if datacentre.find('backup') > -1:
33                continue
34           
35            # now invoke the ingest script
36            logging.info("Running the ingest script for datacentre: %s" %datacentre)
37            oai_ingest(datacentre)
38            run_counter += 1
39               
40print "run_all_ingest.py complete - processed %s config files" %run_counter             
41
42
Note: See TracBrowser for help on using the repository browser.