Ticket #937 (new defect)

Opened 12 years ago

Last modified 11 years ago

Caching in discovery/browse is too permanent

Reported by: selatham Owned by: lawrence
Priority: required Milestone: NDG3
Component: MILK Version:
Keywords: DiscoveryService Cc: spascoe

Description (last modified by spascoe) (diff)

This is what I think is happening:-

  • Ingest records to back-end.
  • NDG front-end displays new info on search results listing, But when you go to 'more' it displays old info.

Is this caching? How often is cache cleared? Needs to be hourly. We need ability to clear cache administratively (without restarting the service).

The discovery portal caches documents retrieved from exist to aid performance but this causes problems when exist is updated. The new discovery back end might have fixed this, but I doubt it. We need to test whether web pages remain consistent when updated records are ingested.

The code for caching should probably be reviewed and cleaned up.

Change History

comment:1 Changed 12 years ago by lawrence

  • Status changed from new to assigned
  • Priority changed from blocker to critical
  • Cc spascoe, domloe added
  • Milestone changed from Updated NERC Data Discovey Service to portal1

The current cache is near permanent and uses the cache class that was put in to handle csml documents etc. It doesn't use the pylons caching which can be set to use timing. The only way things get flushed is when the cache is full, which at ten documents would be quite quick under load ... so it's basically a broken implementation for production.

A better way to do it will be to follow the  pylons documentation, and use both the decorator on the relevant function calls and the etag caching.

We should probably think about the timing of this, as it's a fairly fundamental change (but it should improve performance and be less difficult for Sue et al).

comment:2 Changed 11 years ago by lawrence

  • Keywords DiscoveryService added
  • Component changed from DiscoveryService to discovery

Moved from DiscoveryService? component to discovery as part of NDG2 cleanup

comment:3 Changed 11 years ago by lawrence

  • Component changed from discovery to MILK
  • Milestone changed from NDG2 Cleanup to NDG3

comment:4 Changed 11 years ago by spascoe

  • Status changed from assigned to new
  • Description modified (diff)
  • Cc domloe removed
  • Component changed from MILK to discovery
  • Priority changed from critical to required
  • Owner changed from lawrence to sdonegan

This is a pure discovery portal issue, not related to data services, so I'm reassigning to Steve and changing the component.

Added a bit more explanation to the description.

I don't think this is critical for NDG3 so made it "required".

comment:5 Changed 11 years ago by spascoe

  • Owner changed from sdonegan to lawrence

Whoops, didn't notice this was assigned to Bryan. reverting the assignment.

comment:6 Changed 11 years ago by lawrence

  • Component changed from discovery to MILK

Stephen: we need to talk about what we think MILK is all about. MILK is the client to the data AND information services, so caching is NOT just a discovery issue, it's a browse and data client issue too ... let's not push this to discovery on it's own!

comment:7 Changed 11 years ago by spascoe

Quick comments pre face to face meeting:

If MILK is the whole NDG portal application there is a heap of tickets that should exist in it. E.g. #959, #797, #899, #903, etc.

My understanding that this ticket related to caching of DIF records. Portals Project decoupled visualisation from directly reading DIF records (by introducing WebMapContext links) and I expect to do similar things for WCS in NDG3.

For NDG3 I think we are trying to produce an WMS/WCS/KML web-application that can be deployed independent of the Discovery/MOLES pipeline. If so we need to build that and then integrate it with the NDG Portal code. If we work on the fully integrated codebase development will be slower and we won't end up with a system that allows stand-alone apps (without deep knowledge of the code).

Note: See TracTickets for help on using tickets.