source: TI01-discovery/branches/ingestAutomation-upgrade @ 5351

Revision Log Mode:


Legend:

Added
Modified
Copied or renamed
Diff Rev Age Author Log Message
(edit) @5351   10 years sdonegan testing config
(edit) @5328   10 years sdonegan Updated sql for database merged from rev5026
(edit) @5327   10 years sdonegan Updated sql for database merged from rev5025
(edit) @5326   10 years sdonegan Class to aid in creation of object to aid in extraction of ID plus other …
(edit) @5297   10 years cbyrom Add additional instructions for setting up ingester + improve the setup …
(edit) @5252   10 years cbyrom Simplify error handling, improve output logging + standardise use of upper …
(edit) @5248   10 years cbyrom Update ingest scripts for use with the OAIInfoEditor harvest functionality …
(edit) @5243   10 years cbyrom Adjust logging and output of error in ingest scripts.
(edit) @5218   11 years cbyrom Tidy up docs + fix bug with wrongly scoped variable + remove unecessary …
(edit) @5167   11 years sdonegan updates to allow oai_document_ingester to work
(edit) @5153   11 years sdonegan debugged numfilesproc error
(edit) @4980   11 years cbyrom Fix ingester to use new object structure correctly and more efficiently. …
(edit) @4962   11 years cbyrom Remove breakpoint.
(edit) @4948   11 years cbyrom Adjust ingest scripts to use the new ndg common client suite + add new …
(edit) @4854   11 years cbyrom Add new ingest script - to allow ingest of DIF docs from eXist hosted atom …
(edit) @4714   11 years sdonegan SJD cock up (4712)- put Calums latest (4664) back as head revision!
(edit) @4712   11 years sdonegan Updated to compile a report on number of files successfully (or not) …
(edit) @4711   11 years sdonegan Updated to collate information from each data centre ingest report and …
(edit) @4664   11 years cbyrom Update code to reflect refactoring of ndgUtils.
(edit) @4423   11 years cbyrom Move FileUtilities? module to ndgUtils to allow re-use elsewhere + un-oo …
(edit) @4258   11 years cbyrom Extend idget function to cope with MDIP records and implement usage of …
(edit) @4257   11 years cbyrom Fix handling of ndg hosted data - properly reading config settings from …
(edit) @4224   11 years sdonegan Added extra try-catch to deal with missing temporal coverage in original …
(edit) @4223   11 years sdonegan Updated comments regarding fix for missing temporalCoverage in DIF
(edit) @4222   11 years sdonegan Removed file/folder
(edit) @4221   11 years sdonegan Updated bgs DPPP record
(edit) @4208   11 years sdonegan updated configuration files for new tomcat
(edit) @4207   11 years sdonegan test update
(edit) @4206   11 years sdonegan test update on branch
(edit) @4178   11 years cbyrom Add windows batch file equivalent of create_database.sh
(edit) @4027   11 years cbyrom Add setup script for oai ingester + remove dependencies in this project on …
(edit) @3998   11 years cbyrom Additional 'todo' improvements.
(edit) @3973   11 years cbyrom Add eclipse properties files to allow easy set up of codebase from …
(edit) @3972   11 years cbyrom Use the short filename in the postgres DB for storing the original …
(edit) @3968   11 years cbyrom Add test data to create_database.sh script.
(edit) @3967   11 years cbyrom Update code to allow the extraction of authors, parameters and scope from …
(edit) @3913   11 years cbyrom Updates to readme doc.
(edit) @3912   11 years cbyrom Adjust logging in oai_document_ingester + adjust exception handling, …
(edit) @3910   11 years cbyrom Fix problems with molesReadWrite - including escaping of special …
(edit) @3909   11 years cbyrom Add default logger use to keywordAdder + tidy up code.
(edit) @3908   11 years cbyrom Add local moles schema - for use with molesReadWrite.
(edit) @3907   11 years cbyrom Add rollback to db funcs - to allow processing to continue correctly if an …
(edit) @3906   11 years cbyrom Add further documentation - including filling out 'to do' section with …
(edit) @3870   11 years cbyrom Remove unecessary code relating to the eXist DB.
(edit) @3869   11 years cbyrom Add code to delete records if they weren't properly ingested - to allow …
(edit) @3868   11 years cbyrom Add documentation file with description of new ingestion process.
(edit) @3865   11 years cbyrom Fix keywordAdder so that processing is properly escaped if errors occur …
(edit) @3863   11 years cbyrom Add new stored proc to cleanly delete an ingested document and all its …
(edit) @3862   11 years cbyrom Fix workflow so that transforms are only ran when actually required - i.e. …
(edit) @3861   11 years cbyrom Clear out redundant code and rename main script, oai_document_ingester - …
(edit) @3860   11 years cbyrom Add new function, escapeSpecialCharacters() to PostgresRecord? to correct …
(edit) @3859   11 years cbyrom Improve exception logging + correct return type for ingest stored procs.
(edit) @3858   11 years cbyrom Properly update doc when modifications made + remove redundant code + …
(edit) @3857   11 years cbyrom Add 'debug' mode to ingest script - to allow more verbose logging.
(edit) @3855   11 years cbyrom Add new config file, ingest.config - to store details of the postgres DB …
(edit) @3854   11 years cbyrom Add helper methods to SpatioTemporalData? for easy retrieval of …
(edit) @3853   11 years cbyrom Update python scripts to call the new stored procedures, to properly …
(edit) @3852   11 years cbyrom Add port to db_connect and adjust set up of connection to allow this to be …
(edit) @3851   11 years cbyrom Add various stored procs to create/update records - including proc, …
(edit) @3850   11 years cbyrom Add some test data for the DB to allow quick testing of data model.
(edit) @3849   11 years cbyrom Adjust tables to use timestamps instead of dates - and add create dates to …
(edit) @3848   11 years cbyrom Adjust create database script to allow specification of port - along with …
(edit) @3847   11 years cbyrom Create new object, SpatioTemporalData?, to wrapper temporal and spatial …
(edit) @3846   11 years cbyrom Ajudst DAO and Record classes to throw errors rather than catching them - …
(edit) @3845   11 years cbyrom Fix creation of backup dirs.
(edit) @3844   11 years cbyrom Improve checking of return vals from db calls - to nullify these if …
(edit) @3843   11 years cbyrom Add filename column back into original_document + adjust trigger procedure …
(edit) @3839   11 years cbyrom Add script to run ingest for all avaiable config files. Make …
(edit) @3838   11 years cbyrom Remove unecessary files and dirs - including all xquery stuff since this …
(edit) @3837   11 years cbyrom Add saxon library - required for the xquery transforms.
(edit) @3836   11 years cbyrom Re-organise location of config.properties files to make codebase neater.
(edit) @3835   11 years cbyrom Remove logger - replaced by inbuilt python logger.
(edit) @3834   11 years cbyrom Changes to data model: i) Simplify creation script - to take input …
(edit) @3831   11 years cbyrom Extend the new data model to include spatio-temporal tables - including …
(edit) @3824   11 years cbyrom Adjust data model to use discovery id as a fieldname - and make this …
(edit) @3821   12 years cbyrom Fix a few problems - including referencing the xquery libraries; these …
(edit) @3820   12 years cbyrom Remove unecessary file.
(edit) @3817   12 years cbyrom Add default logging support + create new version of ingest script, …
(edit) @3816   12 years cbyrom Add workflow to allow creation of moles doc from whatever starting point …
(edit) @3815   12 years cbyrom Improve the FileUtilities? - using 'find' to get the various required …
(edit) @3814   12 years cbyrom Fill out functionality for DAO - including check for uniqueness of …
(edit) @3810   12 years cbyrom Create Logger class to standardise output - and allow this to be switched …
(edit) @3809   12 years cbyrom Improve PostgresRecord? class to generalise the transforming of docs into …
(edit) @3808   12 years cbyrom Rename class - DAL -> DAO (data access object) - to make more correctly …
(edit) @3807   12 years cbyrom Create new DB scripts to cope with the document records being stored in …
(edit) @3800   12 years cbyrom Upgraded version of ingest codebranch - including major refactoring of the …
(copy) @3797   12 years cbyrom Upgraded version of ingest codebranch - including major refactoring of the …
copied from TI01-discovery/trunk/ingestAutomation:
(edit) @3779   12 years selatham retiring some old code
Note: See TracRevisionLog for help on using the revision log.