source: TI01-discovery/branches/ingestAutomation-upgrade @ 4258

Revision Log Mode:


Legend:

Added
Modified
Copied or renamed
Diff Rev Age Author Log Message
(edit) @4258   12 years cbyrom Extend idget function to cope with MDIP records and implement usage of …
(edit) @4257   12 years cbyrom Fix handling of ndg hosted data - properly reading config settings from …
(edit) @4224   12 years sdonegan Added extra try-catch to deal with missing temporal coverage in original …
(edit) @4223   12 years sdonegan Updated comments regarding fix for missing temporalCoverage in DIF
(edit) @4222   12 years sdonegan Removed file/folder
(edit) @4221   12 years sdonegan Updated bgs DPPP record
(edit) @4208   12 years sdonegan updated configuration files for new tomcat
(edit) @4207   12 years sdonegan test update
(edit) @4206   12 years sdonegan test update on branch
(edit) @4178   12 years cbyrom Add windows batch file equivalent of create_database.sh
(edit) @4027   12 years cbyrom Add setup script for oai ingester + remove dependencies in this project on …
(edit) @3998   13 years cbyrom Additional 'todo' improvements.
(edit) @3973   13 years cbyrom Add eclipse properties files to allow easy set up of codebase from …
(edit) @3972   13 years cbyrom Use the short filename in the postgres DB for storing the original …
(edit) @3968   13 years cbyrom Add test data to create_database.sh script.
(edit) @3967   13 years cbyrom Update code to allow the extraction of authors, parameters and scope from …
(edit) @3913   13 years cbyrom Updates to readme doc.
(edit) @3912   13 years cbyrom Adjust logging in oai_document_ingester + adjust exception handling, …
(edit) @3910   13 years cbyrom Fix problems with molesReadWrite - including escaping of special …
(edit) @3909   13 years cbyrom Add default logger use to keywordAdder + tidy up code.
(edit) @3908   13 years cbyrom Add local moles schema - for use with molesReadWrite.
(edit) @3907   13 years cbyrom Add rollback to db funcs - to allow processing to continue correctly if an …
(edit) @3906   13 years cbyrom Add further documentation - including filling out 'to do' section with …
(edit) @3870   13 years cbyrom Remove unecessary code relating to the eXist DB.
(edit) @3869   13 years cbyrom Add code to delete records if they weren't properly ingested - to allow …
(edit) @3868   13 years cbyrom Add documentation file with description of new ingestion process.
(edit) @3865   13 years cbyrom Fix keywordAdder so that processing is properly escaped if errors occur …
(edit) @3863   13 years cbyrom Add new stored proc to cleanly delete an ingested document and all its …
(edit) @3862   13 years cbyrom Fix workflow so that transforms are only ran when actually required - i.e. …
(edit) @3861   13 years cbyrom Clear out redundant code and rename main script, oai_document_ingester - …
(edit) @3860   13 years cbyrom Add new function, escapeSpecialCharacters() to PostgresRecord? to correct …
(edit) @3859   13 years cbyrom Improve exception logging + correct return type for ingest stored procs.
(edit) @3858   13 years cbyrom Properly update doc when modifications made + remove redundant code + …
(edit) @3857   13 years cbyrom Add 'debug' mode to ingest script - to allow more verbose logging.
(edit) @3855   13 years cbyrom Add new config file, ingest.config - to store details of the postgres DB …
(edit) @3854   13 years cbyrom Add helper methods to SpatioTemporalData? for easy retrieval of …
(edit) @3853   13 years cbyrom Update python scripts to call the new stored procedures, to properly …
(edit) @3852   13 years cbyrom Add port to db_connect and adjust set up of connection to allow this to be …
(edit) @3851   13 years cbyrom Add various stored procs to create/update records - including proc, …
(edit) @3850   13 years cbyrom Add some test data for the DB to allow quick testing of data model.
(edit) @3849   13 years cbyrom Adjust tables to use timestamps instead of dates - and add create dates to …
(edit) @3848   13 years cbyrom Adjust create database script to allow specification of port - along with …
(edit) @3847   13 years cbyrom Create new object, SpatioTemporalData?, to wrapper temporal and spatial …
(edit) @3846   13 years cbyrom Ajudst DAO and Record classes to throw errors rather than catching them - …
(edit) @3845   13 years cbyrom Fix creation of backup dirs.
(edit) @3844   13 years cbyrom Improve checking of return vals from db calls - to nullify these if …
(edit) @3843   13 years cbyrom Add filename column back into original_document + adjust trigger procedure …
(edit) @3839   13 years cbyrom Add script to run ingest for all avaiable config files. Make …
(edit) @3838   13 years cbyrom Remove unecessary files and dirs - including all xquery stuff since this …
(edit) @3837   13 years cbyrom Add saxon library - required for the xquery transforms.
(edit) @3836   13 years cbyrom Re-organise location of config.properties files to make codebase neater.
(edit) @3835   13 years cbyrom Remove logger - replaced by inbuilt python logger.
(edit) @3834   13 years cbyrom Changes to data model: i) Simplify creation script - to take input …
(edit) @3831   13 years cbyrom Extend the new data model to include spatio-temporal tables - including …
(edit) @3824   13 years cbyrom Adjust data model to use discovery id as a fieldname - and make this …
(edit) @3821   13 years cbyrom Fix a few problems - including referencing the xquery libraries; these …
(edit) @3820   13 years cbyrom Remove unecessary file.
(edit) @3817   13 years cbyrom Add default logging support + create new version of ingest script, …
(edit) @3816   13 years cbyrom Add workflow to allow creation of moles doc from whatever starting point …
(edit) @3815   13 years cbyrom Improve the FileUtilities? - using 'find' to get the various required …
(edit) @3814   13 years cbyrom Fill out functionality for DAO - including check for uniqueness of …
(edit) @3810   13 years cbyrom Create Logger class to standardise output - and allow this to be switched …
(edit) @3809   13 years cbyrom Improve PostgresRecord? class to generalise the transforming of docs into …
(edit) @3808   13 years cbyrom Rename class - DAL -> DAO (data access object) - to make more correctly …
(edit) @3807   13 years cbyrom Create new DB scripts to cope with the document records being stored in …
(edit) @3800   13 years cbyrom Upgraded version of ingest codebranch - including major refactoring of the …
(copy) @3797   13 years cbyrom Upgraded version of ingest codebranch - including major refactoring of the …
copied from TI01-discovery/trunk/ingestAutomation:
(edit) @3779   13 years selatham retiring some old code
Note: See TracRevisionLog for help on using the revision log.