Ticket #1036 (assigned task)

Opened 11 years ago

Last modified 10 years ago

(DI-3-3) Update ingest scripts to allow increased functionality

Reported by: sdonegan Owned by: sdonegan
Priority: required Milestone: NDG3
Component: discovery Version:
Keywords: discovery ingest Cc:

Description

Need the following: -Implement better handling of deletions from postgres db -Update ingest code to be able to synchronise harvests with DB contents -Better data handling within DB (?) -Develop backup system for postgres (does it need it as can hust re-ingest!)

Change History

comment:1 Changed 11 years ago by sdonegan

  • Status changed from new to assigned

comment:2 Changed 10 years ago by sdonegan

Have added extra functionality for handling deletions from database (use DeleteRecord?.py), ingesting single records (extra options to oai_document_ingester.py "-i") and further improved logging. Can now delete records without having to clear out entire database and start again. These classes will now allow a front end interface for users to synchronise their oai repository with the discovery db without having to reingest everything for a single record! This upgraded ingest code now operational on PROGLUE (not glue..!)

Postgres DB on proglue now backed up daily.

TODO: need to merge code currently as tagged release (TI01-discovery/tags/stable-TI01-ingestAutomation_Proglue_upgradesAndReporting) with updated Atom compatible ingest scripts.

Note: See TracTickets for help on using tickets.