source: TI01-discovery/branches/ingestAutomation-upgrade/OAIBatch @ 5418

Revision Log Mode:


Legend:

Added
Modified
Copied or renamed
Diff Rev Age Author Log Message
(edit) @5418   10 years sdonegan Class needed for individual deletion of records from postgres database via …
(edit) @5417   10 years sdonegan needed for deletion of recs - individually - check
(edit) @5416   10 years sdonegan Support extra abstractdocumentingester.py functionality - deletions and …
(edit) @5415   10 years sdonegan Support extra abstractdocumentingester.py functionality - deletions and …
(edit) @5414   10 years sdonegan Contains merged code to allow reporting of ingest, use of ndg redirect, …
(edit) @5413   10 years sdonegan Now includes classes to handle ingest identityTransforms by xquery and …
(edit) @5352   10 years sdonegan
(edit) @5351   10 years sdonegan testing config
(edit) @5326   10 years sdonegan Class to aid in creation of object to aid in extraction of ID plus other …
(edit) @5297   10 years cbyrom Add additional instructions for setting up ingester + improve the setup …
(edit) @5252   11 years cbyrom Simplify error handling, improve output logging + standardise use of upper …
(edit) @5248   11 years cbyrom Update ingest scripts for use with the OAIInfoEditor harvest functionality …
(edit) @5243   11 years cbyrom Adjust logging and output of error in ingest scripts.
(edit) @5218   11 years cbyrom Tidy up docs + fix bug with wrongly scoped variable + remove unecessary …
(edit) @5167   11 years sdonegan updates to allow oai_document_ingester to work
(edit) @5153   11 years sdonegan debugged numfilesproc error
(edit) @4980   11 years cbyrom Fix ingester to use new object structure correctly and more efficiently. …
(edit) @4962   11 years cbyrom Remove breakpoint.
(edit) @4948   11 years cbyrom Adjust ingest scripts to use the new ndg common client suite + add new …
(edit) @4854   11 years cbyrom Add new ingest script - to allow ingest of DIF docs from eXist hosted atom …
(edit) @4714   11 years sdonegan SJD cock up (4712)- put Calums latest (4664) back as head revision!
(edit) @4712   11 years sdonegan Updated to compile a report on number of files successfully (or not) …
(edit) @4711   11 years sdonegan Updated to collate information from each data centre ingest report and …
(edit) @4664   11 years cbyrom Update code to reflect refactoring of ndgUtils.
(edit) @4423   11 years cbyrom Move FileUtilities? module to ndgUtils to allow re-use elsewhere + un-oo …
(edit) @4258   11 years cbyrom Extend idget function to cope with MDIP records and implement usage of …
(edit) @4257   11 years cbyrom Fix handling of ndg hosted data - properly reading config settings from …
(edit) @4224   11 years sdonegan Added extra try-catch to deal with missing temporal coverage in original …
(edit) @4223   11 years sdonegan Updated comments regarding fix for missing temporalCoverage in DIF
(edit) @4222   11 years sdonegan Removed file/folder
(edit) @4221   11 years sdonegan Updated bgs DPPP record
(edit) @4208   11 years sdonegan updated configuration files for new tomcat
(edit) @4207   11 years sdonegan test update
(edit) @4206   11 years sdonegan test update on branch
(edit) @4027   11 years cbyrom Add setup script for oai ingester + remove dependencies in this project on …
(edit) @3998   11 years cbyrom Additional 'todo' improvements.
(edit) @3972   11 years cbyrom Use the short filename in the postgres DB for storing the original …
(edit) @3967   11 years cbyrom Update code to allow the extraction of authors, parameters and scope from …
(edit) @3913   11 years cbyrom Updates to readme doc.
(edit) @3912   11 years cbyrom Adjust logging in oai_document_ingester + adjust exception handling, …
(edit) @3910   11 years cbyrom Fix problems with molesReadWrite - including escaping of special …
(edit) @3909   11 years cbyrom Add default logger use to keywordAdder + tidy up code.
(edit) @3908   11 years cbyrom Add local moles schema - for use with molesReadWrite.
(edit) @3907   11 years cbyrom Add rollback to db funcs - to allow processing to continue correctly if an …
(edit) @3906   11 years cbyrom Add further documentation - including filling out 'to do' section with …
(edit) @3869   12 years cbyrom Add code to delete records if they weren't properly ingested - to allow …
(edit) @3868   12 years cbyrom Add documentation file with description of new ingestion process.
(edit) @3865   12 years cbyrom Fix keywordAdder so that processing is properly escaped if errors occur …
(edit) @3862   12 years cbyrom Fix workflow so that transforms are only ran when actually required - i.e. …
(edit) @3861   12 years cbyrom Clear out redundant code and rename main script, oai_document_ingester - …
(edit) @3860   12 years cbyrom Add new function, escapeSpecialCharacters() to PostgresRecord? to correct …
(edit) @3858   12 years cbyrom Properly update doc when modifications made + remove redundant code + …
(edit) @3857   12 years cbyrom Add 'debug' mode to ingest script - to allow more verbose logging.
(edit) @3855   12 years cbyrom Add new config file, ingest.config - to store details of the postgres DB …
(edit) @3854   12 years cbyrom Add helper methods to SpatioTemporalData? for easy retrieval of …
(edit) @3853   12 years cbyrom Update python scripts to call the new stored procedures, to properly …
(edit) @3852   12 years cbyrom Add port to db_connect and adjust set up of connection to allow this to be …
(edit) @3847   12 years cbyrom Create new object, SpatioTemporalData?, to wrapper temporal and spatial …
(edit) @3846   12 years cbyrom Ajudst DAO and Record classes to throw errors rather than catching them - …
(edit) @3845   12 years cbyrom Fix creation of backup dirs.
(edit) @3844   12 years cbyrom Improve checking of return vals from db calls - to nullify these if …
(edit) @3839   12 years cbyrom Add script to run ingest for all avaiable config files. Make …
(edit) @3838   12 years cbyrom Remove unecessary files and dirs - including all xquery stuff since this …
(edit) @3837   12 years cbyrom Add saxon library - required for the xquery transforms.
(edit) @3836   12 years cbyrom Re-organise location of config.properties files to make codebase neater.
(edit) @3835   12 years cbyrom Remove logger - replaced by inbuilt python logger.
(edit) @3821   12 years cbyrom Fix a few problems - including referencing the xquery libraries; these …
(edit) @3820   12 years cbyrom Remove unecessary file.
(edit) @3817   12 years cbyrom Add default logging support + create new version of ingest script, …
(edit) @3816   12 years cbyrom Add workflow to allow creation of moles doc from whatever starting point …
(edit) @3815   12 years cbyrom Improve the FileUtilities? - using 'find' to get the various required …
(edit) @3814   12 years cbyrom Fill out functionality for DAO - including check for uniqueness of …
(edit) @3810   12 years cbyrom Create Logger class to standardise output - and allow this to be switched …
(edit) @3809   12 years cbyrom Improve PostgresRecord? class to generalise the transforming of docs into …
(edit) @3808   12 years cbyrom Rename class - DAL -> DAO (data access object) - to make more correctly …
(edit) @3800   12 years cbyrom Upgraded version of ingest codebranch - including major refactoring of the …
(copy) @3797   12 years cbyrom Upgraded version of ingest codebranch - including major refactoring of the …
copied from TI01-discovery/trunk/ingestAutomation/OAIBatch:
(edit) @3779   12 years selatham retiring some old code
Note: See TracRevisionLog for help on using the revision log.