source: TI01-discovery/branches/ingestAutomation-upgrade/OAIBatch @ 6130

Revision Log Mode:


Legend:

Added
Modified
Copied or renamed
Diff Rev Age Author Log Message
(edit) @6130   10 years sdonegan Updated handling of directories - put in absolute production paths in badc …
(edit) @6128   10 years sdonegan Took out local directory references and put absolute paths where necessary
(edit) @6100   10 years sdonegan updated reporting
(edit) @6099   10 years sdonegan Updated reporting
(edit) @6093   10 years sdonegan added back in the individual dc reporting - not sure why it was dropped!!
(edit) @6092   10 years sdonegan fixed bug with lineseparator
(edit) @6091   10 years sdonegan Updated error log handling - configured for use in production environment …
(edit) @6031   10 years sdonegan Updated NDG redirection service to production on …
(edit) @6017   10 years sdonegan Updated redirectUrlchanger class so deals with different input formats …
(edit) @5976   10 years sdonegan updated keyword provision when calling method
(edit) @5847   10 years sdonegan fixed bug in getting latest revision date & distinguishing from creation …
(edit) @5846   10 years sdonegan Fixed bug so handles special chars in dataset title and handling of …
(edit) @5737   10 years sdonegan Adjusted filename handling and id's to fix problem with presenting …
(edit) @5736   10 years sdonegan Sorted bug in handling errors where no records available to delete
(edit) @5716   10 years sdonegan New method to allow deletion of all records in Discovery database …
(edit) @5714   10 years sdonegan Fixing big with mdip handling via oai_info_editor (lower case mdip format)
(edit) @5713   10 years sdonegan Udating location
(edit) @5687   10 years sdonegan Updated to handle lower case dif format spec from info editor..
(edit) @5686   10 years sdonegan withdrawing explict DeleteRecord? activation - handled by postgresql …
(edit) @5603   10 years sdonegan small update to reporting
(edit) @5602   10 years sdonegan Debugged ATOM feed ingestion to deal with revised eXist atom architecture …
(edit) @5600   10 years sdonegan Debugged ATOM feed ingestion to deal with revised eXist atom architecture
(edit) @5587   10 years sdonegan Debugging integreattion
(edit) @5586   10 years sdonegan Debugging integreattion
(edit) @5585   10 years sdonegan Updated feed handling to use new reporting methods as developed for …
(edit) @5584   10 years sdonegan Updated feed handling to use new reporting methods as developed for …
(edit) @5583   10 years sdonegan Updated reporting to cmd line on oai_document_ingester.py
(edit) @5582   10 years sdonegan Updated reporting to give correct numbers on ingests
(edit) @5580   10 years sdonegan
(edit) @5553   10 years sdonegan Removed file/folder
(edit) @5548   10 years sdonegan Renamed and sorted config files to match names on oai provider info page …
(edit) @5547   10 years sdonegan replace these original config files with normalised ones to oai_provider …
(edit) @5542   10 years sdonegan Adjust messaging so FULL report is made available to info editor for email
(edit) @5537   10 years sdonegan Updated to allow inclusion of end date ordering
(edit) @5524   10 years sdonegan Updated to allow extraction of end date into special column to allow …
(edit) @5523   10 years sdonegan synchronising dev with buildouts for info editor operations
(edit) @5509   10 years sdonegan dodgy update wiped out recent amendments... restored this version
(edit) @5508   10 years sdonegan Removed file/folder
(edit) @5506   10 years sdonegan corrected bug in indentation…
(edit) @5498   10 years sdonegan debugged problem with previous version - good ol indentation error missed
(edit) @5496   10 years sdonegan update to latest working merged ingest
(edit) @5495   10 years sdonegan Removed file/folder
(edit) @5464   10 years sdonegan Allow start date to be inserted into new column in original doc to allow …
(edit) @5418   10 years sdonegan Class needed for individual deletion of records from postgres database via …
(edit) @5417   10 years sdonegan needed for deletion of recs - individually - check
(edit) @5416   10 years sdonegan Support extra abstractdocumentingester.py functionality - deletions and …
(edit) @5415   10 years sdonegan Support extra abstractdocumentingester.py functionality - deletions and …
(edit) @5414   10 years sdonegan Contains merged code to allow reporting of ingest, use of ndg redirect, …
(edit) @5413   10 years sdonegan Now includes classes to handle ingest identityTransforms by xquery and …
(edit) @5352   10 years sdonegan
(edit) @5351   10 years sdonegan testing config
(edit) @5326   10 years sdonegan Class to aid in creation of object to aid in extraction of ID plus other …
(edit) @5297   10 years cbyrom Add additional instructions for setting up ingester + improve the setup …
(edit) @5252   10 years cbyrom Simplify error handling, improve output logging + standardise use of upper …
(edit) @5248   10 years cbyrom Update ingest scripts for use with the OAIInfoEditor harvest functionality …
(edit) @5243   10 years cbyrom Adjust logging and output of error in ingest scripts.
(edit) @5218   10 years cbyrom Tidy up docs + fix bug with wrongly scoped variable + remove unecessary …
(edit) @5167   10 years sdonegan updates to allow oai_document_ingester to work
(edit) @5153   10 years sdonegan debugged numfilesproc error
(edit) @4980   11 years cbyrom Fix ingester to use new object structure correctly and more efficiently. …
(edit) @4962   11 years cbyrom Remove breakpoint.
(edit) @4948   11 years cbyrom Adjust ingest scripts to use the new ndg common client suite + add new …
(edit) @4854   11 years cbyrom Add new ingest script - to allow ingest of DIF docs from eXist hosted atom …
(edit) @4714   11 years sdonegan SJD cock up (4712)- put Calums latest (4664) back as head revision!
(edit) @4712   11 years sdonegan Updated to compile a report on number of files successfully (or not) …
(edit) @4711   11 years sdonegan Updated to collate information from each data centre ingest report and …
(edit) @4664   11 years cbyrom Update code to reflect refactoring of ndgUtils.
(edit) @4423   11 years cbyrom Move FileUtilities? module to ndgUtils to allow re-use elsewhere + un-oo …
(edit) @4258   11 years cbyrom Extend idget function to cope with MDIP records and implement usage of …
(edit) @4257   11 years cbyrom Fix handling of ndg hosted data - properly reading config settings from …
(edit) @4224   11 years sdonegan Added extra try-catch to deal with missing temporal coverage in original …
(edit) @4223   11 years sdonegan Updated comments regarding fix for missing temporalCoverage in DIF
(edit) @4222   11 years sdonegan Removed file/folder
(edit) @4221   11 years sdonegan Updated bgs DPPP record
(edit) @4208   11 years sdonegan updated configuration files for new tomcat
(edit) @4207   11 years sdonegan test update
(edit) @4206   11 years sdonegan test update on branch
(edit) @4027   11 years cbyrom Add setup script for oai ingester + remove dependencies in this project on …
(edit) @3998   11 years cbyrom Additional 'todo' improvements.
(edit) @3972   11 years cbyrom Use the short filename in the postgres DB for storing the original …
(edit) @3967   11 years cbyrom Update code to allow the extraction of authors, parameters and scope from …
(edit) @3913   11 years cbyrom Updates to readme doc.
(edit) @3912   11 years cbyrom Adjust logging in oai_document_ingester + adjust exception handling, …
(edit) @3910   11 years cbyrom Fix problems with molesReadWrite - including escaping of special …
(edit) @3909   11 years cbyrom Add default logger use to keywordAdder + tidy up code.
(edit) @3908   11 years cbyrom Add local moles schema - for use with molesReadWrite.
(edit) @3907   11 years cbyrom Add rollback to db funcs - to allow processing to continue correctly if an …
(edit) @3906   11 years cbyrom Add further documentation - including filling out 'to do' section with …
(edit) @3869   11 years cbyrom Add code to delete records if they weren't properly ingested - to allow …
(edit) @3868   11 years cbyrom Add documentation file with description of new ingestion process.
(edit) @3865   11 years cbyrom Fix keywordAdder so that processing is properly escaped if errors occur …
(edit) @3862   11 years cbyrom Fix workflow so that transforms are only ran when actually required - i.e. …
(edit) @3861   11 years cbyrom Clear out redundant code and rename main script, oai_document_ingester - …
(edit) @3860   11 years cbyrom Add new function, escapeSpecialCharacters() to PostgresRecord? to correct …
(edit) @3858   11 years cbyrom Properly update doc when modifications made + remove redundant code + …
(edit) @3857   11 years cbyrom Add 'debug' mode to ingest script - to allow more verbose logging.
(edit) @3855   11 years cbyrom Add new config file, ingest.config - to store details of the postgres DB …
(edit) @3854   11 years cbyrom Add helper methods to SpatioTemporalData? for easy retrieval of …
(edit) @3853   11 years cbyrom Update python scripts to call the new stored procedures, to properly …
(edit) @3852   11 years cbyrom Add port to db_connect and adjust set up of connection to allow this to be …
Note: See TracRevisionLog for help on using the revision log.