wiki:Inputfiles

Files used by the ORAC processor

Input

The files required to run the ORAC processor are:

# Default name Descriptive name Type Purpose
1 User_defined.txt Driver file ASCII Specifies data files, relevant folders, and channels to consider
2 INST_ChX.sad Channel specification file ASCII Defines the principal characteristics of a particular instrument channel
3 INST_CLASS_LUT_ChX.sav LUT file ASCII Look-up table LUT of cloud radiative properties for particle model CLASS
4 User_defined.alb.nc Albedo file NCDF Surface albedo data by pixel
5 User_defined.clf.nc Cloud flag file NCDF Cloud flagging bit map by pixel
6 User_defined.geo.nc Geometetry file NCDF Satellite/solar/relative viewing angles by pixel
7 User_defined.loc.nc Geolocation file NCDF Lat/lon by pixel
8 User_defined.lsf.nc Land-sea file NCDF Land-sea flag by pixel
9 User_defined.lwrtm.nc LW-RTM file NCDF Atmospheric RT results in the longwave
10 User_defined.msi.nc Imagery file NCDF Multi-spectral imagery data by pixel
11 User_defined.prtm.nc RTM profile file NCDF Profile data (T,p) used in RTM calculations
12 User_defined.swrtm.nc SW-RTM file NCDF Atmospheric RT results in the shortwave
13 User_defined.uv.nc Wind vector file NCDF ECMWF surface wind vectors by pixel
14 User-defined.config.nc Configuration file NCDF Summarises the channel configuration

1. Driver file

This is the driver file for the ORAC system. Examples of these files can be found at http://proj.badc.rl.ac.uk/orac/trunk/src/modis_driver.txt and http://proj.badc.rl.ac.uk/orac/trunk/src/aatsr_driver.txt. These comprise 9 lines:

  1. The folder in which the output of the pre-processor can be found.
  2. The root filename of the 11 files to be processed.
  3. The folder into which the output of ORAC should be stored. This and the previous two entries should be delimited with '.
  4. The folder in which the instrument channel look-up tables can be found. These are stored in the folder sad_dir on the repository.
  5. The name of the instrument that will be processed. For MODIS, this should also specify the platform (e.g. MODIS-TERRA or MODIS-AQUA).
  6. The number of channels available in the input files.
  7. A list of space-delimited flags indicating for each channel available in the input files if it should be processed (1=yes; 0=no).
  8. The cloud class to be used. Possibilities are WAT, for water cloud, and ICE, for ice cloud.

An example is,

'/home/midi/eodg2/data/testoutput'
'DAYAATSR_UoOx_AATSR_1-59.3--59.3_ORACV1929_ENV_20140127105248_200806200023_V1.0'
'/home/midi/eodg2/data/testoutput'
'/home/blt/orac_sad/sad_dir'
AATSR
6
1 1 1 1 1 1
WAT

Additionally, most fields within the Ctrl structure can be optionally set through the driver file (for a full list, consult the select case statement around line 690 of ReadDriver?.F90). These lines have the syntax NAME = VALUE. For example, if you wish to only process the first three lines of an input file you would add the following beneath the above,

Ctrl%Ind%Y1 = 3

as the Ctrl%Ind%Y1 field sets the last row to be processed in a file. Comments are denoted by the # symbol. Structure element selection can be denoted by the % or . symbol. 1-D arrays can be delimited by whitespace or commas while 2-d arrays can be delimited with either commas for the first dimension and semicolons for the second or whitespace and commas.

1b. How to select with channel to use for effective radius

Using MODIS as an example the default value for Ctrl%ReChans? is

Ctrl%ReChans=20,6,7,5

Ctrl%ReChans must have 4 elements based on the length of
Ctrl%r_e_chans=5,6,7,20 (the set of candidate Re channels).

If 5,6,7,20 are given to ORAC (with the bits for all Re channel set in the channel bit mask in the driver file) then it will choose one Re channel based on availability and decreasing priority in the Ctrl%ReChans? list. So if 20 is available then 20 is used or if 20 and 6 are not available then 7 is used. If for example you don't want 6 to ever be used then use

Ctrl%ReChans = 20,0,7,5

For Aqua you might want to use

Ctrl%ReChans = 20,7,6,5

If you want more control and don't want ORAC to chose just one effective radius channel then use

Ctrl%ReChans = 0,0,0,0

which basically turns this feature off. In which case it will use whatever is set in the channel bit mask even if more than one Re channel is set.

2. Channel specification file

Each channel for each instrument has an associated file giving its principal characteristics. The file name is constructed as in the section title using the instrument code and the channel identifier. The identifier convention is that ascending number corresponds to increasing wavelength, however, this need not be strictly adhered to, the only requirement is that all channels with solar contributions are numbered below those with only thermal contributions. (Thus for example, the order [Ch1=0.67, Ch2=0.87, Ch3=11] is conventional, [Ch1=0.87, Ch2=0.67, Ch3=11] is acceptable, [Ch1=0.67, Ch2=11, Ch3=0.87] is not acceptable. Modifications to these files should be limited to the system administrator. A sample specification file for the ATSR-2 3.7um channel is given below.

ATSR-2_Ch5.sad     % Channel characterisation file
3.7 um             % Descriptor               SAD_CHAN()%Desc
Ch5                % File ID                  SAD_CHAN()%Fileid
2715.30            % Central wavenmuber       SAD_CHAN()%Wvn
1                  % Thermal cource flag      SAD_CHAN()%Thermal%Flag
238452.0           % Planck coefficient 1     SAD_CHAN()%Thermal%B1
3885.87            % Planck coefficient 2     SAD_CHAN()%Thermal%B2
-2.041717          % Planck coefficient 3     SAD_CHAN()%Thermal%T1 

1.005042           % Planck coefficient 4     SAD_CHAN()%Thermal%T2
.5,.5,1.,.5,.5     % Thermal noise Eqv Homog  SAD_CHAN()%Thermal%NeHomog
.15,.15,1.,.25,.25 % Thermal noise Eqv Coreg  SAD_CHAN()%Thermal%NeCoreg
0.28               % Thermal NeBT             SAD_CHAN()%Thermal%NeBT
1                  % Solar source flag        SAD_CHAN()%Solar%Flag
5.39,0.177         % Solar constant           SAD_CHAN()%Solar%F0,F1
.5,.5,1.,.5,.5     % Solar noise Eqv Homog    SAD_CHAN()%Solar%NeHomog
.15,.15,1.,.25,.25 % Solar noise Eqv Coreg    SAD_CHAN()%Solar%NeCoreg
0.0                % Solar NeFR               SAD_CHAN()%Solar%NeFR
0.01 1.0           % 'typical' sea/land refl  SAD_CHAN()%Solar%Rs(2)

This channel is chosen as an example because it has both solar and thermal components and hence the characterisation file is more complete than most other channels. If a source (thermal or solar) is not present, then the detail following the flag for that source is omitted; the file below is for the ATSR-2 0.67um channel which has only solar sources.

ATSR-2_Ch2.sad     % Channel characterisation file
0.67 um            % Descriptor               SAD_CHAN()%Desc
Ch2                % File ID                  SAD_CHAN()%Fileid
14925.4            % Central wavenmuber       SAD_CHAN()%Wvn
0                  % Thermal source flag      SAD_CHAN()%Thermal%Flag
1                  % Solar source flag        SAD_CHAN()%Solar%Flag
21.46, 0.703       % Solar constant           SAD_CHAN()%Solar%F0,F1
.5, .75,2.,1.,.75  % Solar noise Eqv Homog    SAD_CHAN()%Solar%NeHomog
1.5,2., 3.,1.,1.   % Solar noise Eqv Coreg    SAD_CHAN()%Solar%NeCoreg
0.0058             % Solar NedR               SAD_CHAN()%Solar%NedR
0.1 5.0            % 'typical' sea/land refl  SAD_CHAN()%Solar%Rs(2)

3. LUT file

LUT file names are constructed from the instrument, channel, cloud class and radiative properties codes. Thus the water cloud bidirectional reflectance file for the SEVIRI 0.8um channel is named SEVIRI-1_WAT_RBD_Ch2.sad. Files are ASCII formatted for portability. Each file has header information describing its contents and allowing for some consistency checks between it and the requested parameters. The header contents are described in the following table; see following sections for full details.

Needed for ? Header information present ?
grid descriptors (nX, dX, X)
LUT Description Solar Thermal Wavelength Tau Sat zen Sun zen Rel azi Re
Rbd Bidirection reflectance * * * * * * *
Tb Beam direct transmission * * * * *
Tbd + Tfbd Beam diffuse transmission + flux * * *
Td + Tfd Diffuse transmission + flux * * * * * *
Rd + Rfd Diffuse reflection + flux * * * * * *
Em Emission * * * * *

The exact contents of the LUT files depend on which grid descriptors apply to the LUT in question (see table above). However, the general structure is the same in each case: Line 1: Wavelength (f8.2). Store in SAD_LUT()%wavelength Line 2: Grid descriptor 1: number of values (integer), grid step size for this descriptor (float) Line 3: Grid descriptor 1: grid descriptor values Line 4: Grid descriptor 2: as above Line 5 etc as above for all relevant grid descriptors Line n to end of file: LUT values (f8.3); stored in SAD_LUT( )%LUT(x, y, z, ...)

For example, the RBD LUT files contain the following information:

Wavelength (f8.2). Store in SAD_LUT()%wavelength
no. of tau (i2), delta_Tau (f8.2). Check against SAD_CloudClass()%nTau),
Tau values (nTau * f8.3). First (1) stored in SAD_LUT()%Grid%MinTau, last (nTau) stored in %MaxTau
No. of Satzen values, dSatzen i4, f8.2 check nsat against SAD_CloudClass()%nSat, dsat stored in SAD_LUT()%Grid%dSatzen, nsat stored in SAD_LUT()%Grid%nSatzen
Satzen values (nSat * f8.2) satz(1) stored in SAD_LUT()%Grid%MinSatzen, satz(nsat) stored in MaxSatzen
No. of Sunzen values, dSunzen (i4, f8.2) check nsun against SAD_CloudCLass()%nSun, dsun stored in SAD_LUT()%dSunzen, nsun stored in SAD_LUT()%nSunzen
Sunzen values (nsun* f8.2) sunz (1) stored in SAD_LUT()%Grid%MinSunzen, sunz(nsun) in MaxSunzen
No. of Relazi values, dRelazi (i4, f8.2) check nazi against SAD_CloudCLass()%nAzi, dRelazi stored in SAD_LUT()%Grid%dRelazi,
Relazi values (nazi * f8.2) azi(1) stored in SAD_LUT()%Grid%MinRelazi, azi(nazi) stored in MaxRelazi
No. of Re values, dRe (i4, f8.2) check nr against SAD_CloudCLass()%nRe, dr stored in SAD_LUT()%Grid%nRe,
Re values (nre * f8.2), Re(1) stored in SAD_LUT()%Grid%MinRe, re(nre) stored in MaxRelazi
RBD (f8.3) stored in SAD_LUT()%Solar%Rbd (chan_index, 1:ntau, 1:nsat, 1:nsun,1:nazi,1:nr) (written left indices first)

4. Albedo file

Provides the surface albedo and emissivity information. The variables in the file are: alb_abs_ch_numbers, emis_abs_ch_numbers, alb_data, emis_data.

5. Cloud flag file

Cloud flags are assumed to be provided in a file separately to the MSI data. For ATSR-2 data, pre-processor code will generate a cloud flag file. See the section on image segmentation above. The variables in the file are: clfag.

6. Geometry file

ATSR MSI data includes geometric information, but in order to retain the functionality required for other data sources, it is assumed that a separate pixel based geometry file will be available. For ATSR images, this will be generated 'offline' by the pre-processor code. See the section on image segmentation above. The variables in the file are: solzen, satzen, solaz, relazi.

7. Geolocation file

Latitude and longitude for each pixel, taken from the geolocation considerations of the original satellite data. The variables in the file are: lat, lon.

8. Land-sea file

ORAC will not obtain the underlying surface character for a particular pixel but will assume a pixel based map is available via this file. For ATSR-2 processing, this will obtained by the pre-processor code. See the section on image segmentation below. The variables in the file are: lsflag.

9. LW-RTM file

The longwave radiative transfer file supplies the atmospheric channel transmittances and radiance terms (see the ATBD) for the channels with thermal emission sources. The source of this file for ORAC prototyping and testing will be the Eumetsat RTM model. See the !Read_Lw_RTM module description for details of the Lw RTM file format and contents. The variables in the file are: lw_channel_abs_ids, lw_channel_instr_ids, lw_channel_wvl, counter_lw, solza_lw, satza_lw, relazi_lw, emiss_lw, tac_lw, tbc_lw, rbs_up_lw, rac_up_lw, rac_down_lw.

10. Imagery file

To avoid the difficulties of ingesting multiple data formats from different imagers within the retrieval itself, this is managed within the preprocessing to produce this file summarising only the desired spectral information in a format approriate to ORAC. The variables in the file are: mis_instr_ch_numbers, msi_abs_ch_numbers, msi_abs_ch_wl, msi_ch_swflag, msi_ch_lwflag, msi_ch_procflag, time_data, msi_data.

Image segmentation: The ORAC preprocessing allows for the image data to be read in "chunks" of a specified number of image rows, in order to reduce the memory required. It is unnecessary to hold the entire image arrays in memory when the processing of each pixel is independent of the rest of the image. The image data files (i.e. the multi-spectral image file and all associated files: geometry, location, land/sea flags and cloud flags) are written in image rows. An image row consists of all the channel values at each x location for a given y location within the image (or all the geometry/location/flags at each x for a given y).

11. RTM profile file

This file contains the level pressure and temperature information that was used in the RTM models. See the Read_Lw_RTM module description for details of the RTM profile file format and contents. The variables in the file are: i_pw, j_pw, counter_pw, lon_pw, lat_pw, skint_pw, explnsp_pw, lsf_pw, satzen_pw, solzen_pw, pprofile_lev, tprofile_lev, gphprofile_lev.

12. SW-RTM file

The shortwave radiative transfer file supplies the atmospheric channel transmittances and radiance terms (see the ATBD) for the channels without thermal emission sources. The source of this file for ORAC prototyping and testing will be a modified LOWTRAN7 RTM model. See the Read_Sw_RTM module description for details of the Sw RTM file format and contents. The variables in the file are: sw_channel_abs_ids, sw_channel_instr_ids, sw_channel_wvl, solza_sw, satza_sw, relazi_sw, counter_sw, tac_sw, tbc_sw.

13. Wind vector file

Provides wind vectors from derived from ECMWF data. The variables in the file are: uscan, vscan.