|
|
AUSTRALIAN
NATIONAL UNIVERSITY System Design Note 11.03 Created: 11 September 2001 Last modified: 11 September 2001 |
NIFS DETECTOR CONTROLLER SOFTWARE DESIGN
Peter Young
Research School of Astronomy
and Astrophysics
Institute of Advanced
Studies
Australian National
University
Revision History
|
Revision No. |
Author & Date |
Approval & Date |
Description |
|
Revision 1 |
Peter Young 10 March 2000 |
Peter J. McGregor 05 September 2001 |
Original document. |
|
Revision 2 |
Peter Young 10September 2001 |
Peter J. McGregor 10 September 2001 |
Revised after CDR. |
|
|
|
|
|
Contents
4 Spectrograph
Detector Controller Software Design
4.2 Gemini
CAD/CAR/APPLY/SIR Interface – EPICS Databases
4.2.2 Status and
Alarm Database
4.5 SDSU-2 IR
Controller Interface
4.6 RSAA’s SDSU-2 Camera C++ Class
4.6.1 The SDSU2_IR Class Specific
Readout Methods
4.6.2 Detector
Readout Linearity
4.6.4 NIFS
Specific Image Processing
4.6.5 Internal
Data Storage Requirements
4.7 Unraveling
Pixel Data During Readout
4.8 Moving Data to
the Gemini DHS and Quick-Look Displays.
4.8.2 Viewing the
Image as it is Being Built
4.9 Writing Local
FITS Files – the Fits Class
4.11 Gemini Status
and Alarms Database
5 Spectrograph
Detector Controller Software Implementation Detail
5.1 Detector
Controller IOC Task Diagram and Data Flows
5.1.1 The Control
Tasks (dc_ss, SDSU_ss, and obs_ss)
5.1.3 The Detector
Readout Task (Readout)
5.1.4 The Data
Transfer Task (Data)
5.1.5 The
Interface Between the NIFS C++ Code and EPICS.
5.2.2 The OBSERVE Command – Processing Timeline
5.2.3 Data
Transfer Throughput
1 Purpose
This document describes the Detector Controller software design for the Gemini Near-infrared Integral Field Spectrograph (NIFS). As described in the CICS Detailed Design Document, this subsystem is responsible for controlling the instrument’s detector array and obtaining and processing data from it.
2 Applicable Documents
|
Document
ID |
Source |
Title |
|
cics_smb_041 |
IGPO |
CICS Detailed Design Document |
|
NOAO_CCD |
NOAO |
Gemini CCD Controller Software Manual |
|
Swg_hty_require |
IfA |
Suggestions on Relaxing Gemini Requirements for Software Developers |
|
|
|
|
3 Introduction
This document describes the detector controller software design for the Gemini NIFS instrument. The design has been driven by the need to fast-track the construction of the NIFS instrument and therefore reuses as much software as possible from multiple sources. To meet Gemini interface requirements and to make things as simple as possible, it was decided that the “thin EPICS layer” approach, adopted by the GMOS detector controller group, be copied. See the GMOS CCD Controller Software Manual, Wolff & Ruckle (1999), for a description of this approach. This will be coupled with a suitably modified copy of RSAA’s existing SDSU-2 controller interface software, to take advantage of code reuse.
This document concentrates on the actual design adopted.
4 Spectrograph Detector Controller Software Design
The DC software consists of a multiple task architecture designed to optimally support the requirements of the NIFS science detector.
4.1 Overall Architecture
The DC architecture used tasks written as EPICS SNL code to interface to Gemini. We refer to these as the Control tasks, They are responsible for interfacing to the Gemini systems using EPICS channel access. The Control tasks pass commands to the Slave task for taking exposures and setting up the detector using the SDSU-2 controller. The Slave task monitors the progress of the Readout task during a detector readout and signals readout progress to the Data task. Progress is monitored through the use of an interrupt service routine (ISR) that is executed in response to an interrupt generated by the SDSU-2 VME Interface Board. The Data task will then transfer image data to the Gemini Data Handling System (DHS) for storage and quick look display. Figure 1 illustrates this architecture.
Figure 1: NIFS DC software architecture.
The architecture also shows five external systems and five internal data stores. Two of the data stores are EPICS databases for storing configuration and status information. There are also shared memory areas for storing operational parameters, copied from the EPICS CAD/CAR database, and status information before being moved to the EPICS SAD database. A shared memory area is again used for storing the detector image data at various stages of processing.
4.2 Gemini CAD/CAR/APPLY/SIR Interface – EPICS Databases
Gemini requires that instrument developers use EPICS CAD/CAR/APPLY and SIR records to interface to the Gemini observatory systems and control the instrument. We have decided to adopt the "thin-layer" EPICS approach used by the NOAO CCD detector controller group. In this approach the EPICS layer supports the setting of parameters and handling of commands through CAD and APPLY records by the OCS, but lower level code does any parameter verification and performing of required actions. The EPICS layer simply passes parameters and commands, gets back status, and does nothing else. Operational parameters are moved to the lower level shared memory area after they are modified. This is done using the CAD SNAM subroutines. This shared memory area is then accessed by the lower level tasks during the operation of the NIFS detector.
The NIFS DC recognizes all the OCS detector controller commands and uses the CAD/CAR records specified in ICDs 1a and 1b. Some of these are ignored but handled correctly by the toggling of the action state from BUSY to IDLE.
Configuration and status parameters in Gemini parameter definition format are available in ICD 1.9f/3.1. Settings for two observation modes are maintained: Idle and Run mode. In addition, SDSU-2 controllers settings are also detailed.
NIFS DC Capfast diagrams are based on the CICS DC (Beard 1997) and GMOS DC (Wolff & Ruckle 1999) hierarchy of Capfast diagrams. Changes have been made to reflect the requirements of NIFS.
4.2.1 CAD/CAR Database
Figure 2 shows the Capfast diagrams for the DC CAD/CAR database design. Note that they are arranged according to similar grouping. CADs are APPLIED in order from left-right and top-bottom.
Figure 2: Hierarchy of Capfast diagrams in the DC CAD/CAR database design.
The following describes how this database is designed (and is taken from Wolff & Ruckle 1999):
- Top level APPLY record
The DC CAD/CAR database has a top level APPLY record that can control the actions of all of the CAD records connected to it. If a CLEAR or MARK is given to the APPLY record, all of the CAD records connected to it will perform that action. If a PRESET, START, or STOP is sent, only the CAD records that are currently marked will be processed. The CAD records are marked by either sending them a MARK directive, or sending a new value to one of their inputs (A,B,C...fields).
- Lower level APPLY records
The CAD records are assembled into groups depending on their function. Each group has an APPLY record assigned to it. This lower level APPLY record gets a command from the top level APPLY and sends it along to the CAD records one by one. If the first CAD record finishes with CAD_ACCEPT, then the next one is started, otherwise, a CAD_REJECT is returned, and no other CAD records are processed. When all the CAD records under this APPLY are done, an ACCEPT or REJECT is returned to the upper level CAD. Which will either allow the processing of the next group, or stop processing with a REJECT.
- CAR records
CAR records retrieve responses from the CAD records when a START directive is received by a marked CAD record. They are set to BUSY while the CAD is processing, and set to IDLE or ERROR depending on the result of processing. The VAL, OERR, and OMSS of all the CAR records are connected to the APPLYC CAR record through a structure of GENSUB records. The GENSUB records look at the VAL fields of the CARs connected to it and pass along the value that is the greatest along with the corresponding OERR and OMSS fields.
This database includes CAD records for the reboot, init, test, park, debug, set up SDSU, set up readout sequence, set up data processing (including setDhsInfo), observe, stop, and abort commands. Other mandatory Gemini sequence commands are also included but are ignored except for the toggling of the corresponding CAR from BUSY to IDLE (verify, endVerify, guide, endGuide, endObserve, pause, continue, and datum).
4.2.2 Status and Alarm Database
The DC SAD has Capfast diagrams for system, SDSU, exposure, and data status information. Figure 3 shows this arrangement.
Figure 3: Hierarchy of Capfast diagrams in the DC SAD design.
Each of these Capfast schematics have SIR records that describe each of these areas of the DC design. A full description of all SIR records is available in ICD 1.9f/3.1. Some SIR records are updated directly from CAD output parameters when they are verified with the CAD PRESET directive. The remainder are updated from the operational NIFS C++ status object whenever they are modified. This immediate update is implemented by the use of C++ class methods that are called by the lower layer NIFS C++ code when updating a status parameter. That is, access to the SAD database by the non-EPICS layer of NIFS is encapsulated in the class implementation – each status member variable in the nifs_status class has a corresponding SIR record which is updated whenever the private member variable is updated using the appropriate class method.
4.3 SNL Code
The DC design uses three SNL state sets. These are used to interface between the Gemini EPICS layer and the NIFS VxWorks tasks. They are described fully in §5.1.1. The SNL code, typically, monitors the st.VAL field of a CADPLUS schematic for an indication that a particular CAD has received a START directive. These CADPLUS schematics are connected to the STLK forward link of a CAD record.
4.4 Support Routines
The DC CAD/CAR database CAD records call VxWorks support routines named in their INAM and SNAM fields at record initialization or in response to a directive passed in from an APPLY record. These routines are called directly from the EPICS database code. For NIFS, the INAM routines are used to write default values to a lower level C++ class object that is accessed by VxWorks tasks. The SNAM routines need to respond appropriately to each CAD directive, including verifying input parameters (PRESET directive), and starting any associated command (START directive). For NIFS, during a PRESET, parameters are copied from the CAD record field (inputs A-T) to the C++ class object, using a method of the class that will perform the actual verification and return a status result to the support routine for passing back to EPICS (see §5.1.5). The input parameters, if verified, are also copied to the CAD output fields. Note, that it is very important to maintain the correct mapping between an input CAD record field and the NIFS functional parameter. These support routines provide that mapping and therefore must be kept aligned with the CAD EPICS database generated by Capfast.
4.5 SDSU-2 IR Controller Interface
The lower layers of code running in the Slave, Readout, and Data tasks are triggered by commands from the Control task. These codes interface directly with the SDSU-2 electronics. RSAA has experience using this controller and intend to take advantage of code already in use. Code reuse issues were covered at the NIFS CoDR and will not be repeated here. We will also be making use of work done by Tim Hardy of the National Research Council of Canada, by using ideas from their sdsuLib code.
4.6 RSAA’s SDSU-2 Camera C++ Class
This section provides an overview of the RSAA Camera class and shows how this can be developed into a reusable component for Gemini cameras in a way that has been successfully deployed at RSAA.
The Camera class is a hierarchical arrangement of C++ classes that form a code infrastructure to support different electronic readout hardware. Because camera readouts are similar in most operations, this code uses common methods where possible. When specific hardware operations are required, base class methods are overridden with specific methods in the sub-class. The diagram in Figure 4 illustrates this hierarchical arrangement.
Figure 4: Camera class hierarchy.
The base level Camera class contains member variables and methods that are generic for all controller types. These include support for:
· Camera description - generic camera description information including number of controllers, controller type, number of detectors, detector type, detector size, detector location, number of readouts, readout size, readout location, and readout direction and orientation.
· Configuration - information about how the camera is currently configured.
· Regions of interest information.
· Image buffer information.
· Observation settings.
· Status detail.
· Readout progress monitoring.
· Image transfer methods, including unscrambling the data stream.
Controllers of different types make use of this common code for handling camera operations. For NIFS, the sub-classes SDSU and SDSU2_IR will model the behavior of the SDSU-2 infrared hardware. The class SDSU handles all functionality generic to this type of controller and includes support for:
· Controller communication, including device access, command passing, reply handling, and DSP code downloading.
· Generic controller initialization functions. Some initialization commands are common for the different SDSU controller types, e.g., power on and DSP code downloading.
· Controller hardware tests.
· Exception handling – including interrupts.
The SDSU2_IR class will handle functionality specific to the infrared controller. Including:
· Exposure handling.
· Infrared readout methods. Specifically, these will be linear fitting, double correlated sampling, and Fowler sampling.
The NIFS sub class will handle image processing specific to the NIFS IFU. This relates to recombining the image slices into a data cube and then compressing the cube along the spectral axis.
At RSAA, the classes Camera, SDSU, SDSU1 and SDSU2 are in place, so for NIFS, coding of the SDSU2_IR and NIFS classes are required as well as some repackaging of the base classes to handle integration.
4.6.1 The SDSU2_IR Class Specific Readout Methods
Three readout methods will be supported by NIFS. These are:
1. Double Correlated Sampling
This method resets the array and then performs an immediate read. At the end of the exposure a final read is performed and the resultant image is the difference between the two images. Parameters for setting the number of resets and the delay after a reset will be available through a CAD record.
2. Fowler Sampling
Fowler & Gatley (1990) show that read noise can be reduced by performing multiple non-destructive reads of the detector at the beginning and end of the integration ramp.
This algorithm operates by averaging N full readouts of the detector at the beginning of the exposure and then N full readouts at the end of the exposure. The resultant image is the difference between the two averaged images.
Saturated pixel and cosmic ray handling for these first two methods are limited to masking out relevant pixels during data reduction – if detected. That is, these pixels can only be detected if values are saturated by the time of the second set of reads. If so detected an array S is marked.
3. Linear Fitting
In this method, the detector is first
reset, then the array is read, non-destructively, N times over the total
exposure time at regular intervals. As the data are accumulated a least-squares
linear regression is fitted through each pixel to define the incident photon
rate. The algorithm used to calculate the slope of the regression is taken from
Chapman et al. (1990). This is shown in rewritten form to enable progressive
computation and hence progressive display of the integration as
where Vi is the voltage of sample i, n is the total number of samples, and dt is the period between NDR samples. This is a much-simplified version of the general straight line fitting equation due to the constant value of the NDR period, dt.
To compute the variance array for this fit, the calculation of E below has to be made for each pixel (see Neter & Wasserman 1974).
Memory has to be set aside to save values for SiV, SV, and SV2 for each pixel, while the other terms are constant. The time needed for this processing is of the order of 3.3 s for the full detector size on the 400 MHz SVGM5 (see §5.2.4).
Complications arise when pixels saturate or cosmic rays are encountered. These issues are described below.
4.6.1.1 Saturated Pixels
When V becomes saturated for any pixel, accumulation of the sums will cease and the data recorded up until the current time will be used to calculate the slope. This will require a recording of NDR number at which a pixel saturates, in array S. This array will be initialized to zeros and is implemented using single bytes, allowing NDR values of up to 254 to be recorded (the value 255 is reserved for bad pixels, see §4.6.1.3). NIFS is unlikely to perform more than 254 NDRs per exposure due to the accumulated effect of multiplexer glow.
4.6.1.2 Cosmic Rays
Cosmic rays can, perhaps, be detected by checking the rate of change of the calculated slope. If this goes above a threshold then accumulation of data for the pixel will cease and, in the first instance, begin again in secondary arrays SiV2, SV2, and SV22. A weighted-average of the two slopes will be used for calculating the final pixel value. To indicate whether a pixel has encountered a cosmic ray, an array C will be used to record the time. A second cosmic ray encounter for a pixel will result in data recording stopping and the array S marked, that is, the pixel will be deemed to have saturated.
4.6.1.3 Bad Pixels
Bad pixels will be recorded in the array S with the value of 255. These will be initialized from a bad-pixel file loaded from disk when the system starts up.
4.6.2 Detector Readout Linearity
Consideration has to be made of the
linearity of the detector integration. This is essentially a correction applied
to each pixel using a calibrated parameter. Setting of this parameter will be
made through the CAD database with reflection in the SAD. Final details are yet
to be determined.
4.6.3 Subtracting Out the Sky
The Readout task will subtract out the sky for each image before the image processing mentioned below (§4.6.4) is performed and the resultant image displayed in a DHS quick-look tool. It does the image subtraction by using a loaded sky frame for the particular operating mode Idle or Run. See §5.1.3 for a full description of this process.
A DHS quick-look tool will be used to perform image-subtraction for display of the raw data frame after it is transferred to the DHS.
4.6.4 NIFS Specific Image Processing
Some image processing needs to be performed before quick-look displays are useful to the astronomer because of the nature of the NIFS IFU. In addition to the raw data file (suitably unscrambled and fitted), the image data need to be reformatted into a spatial image of the object from the 29 IFU slitlets.
In the case where the flip mirror is in use (i.e., during object acquisition), the IFU slit staircase is collapsed so that each of the slitlets are aligned next to each other to form an image of the object with slitlet height on the y axis and each of the slitlets along the x axis.
In the case where the spectrum is dispersed, the data are formed into an image cube with slitlet height on the y axis, each of the slitlets along the x axis, and wavelength along the z axis. This cube is then compressed along the z axis, over a user-chosen wavelength interval, to form an image of the object.
Because each IFU slitlet is 0.103″ wide and the pixel height in the spatial direction is 0.043″ , each detector pixel is sub-divided by five and two in width and height, respectively, to form a final display pixel size of ~ 0.02″×0.02″ on the sky. The resulting image size is 145×140 pixels. This image is sent to a quick look display for preview. See the NIFS OCDD for a detailed description of this requirement.
The time needed for this processing is 0.23 s for the full detector size on the 400 MHz SVGM5 (see §5.2).
4.6.5 Internal Data Storage Requirements
The arrays described in Table 1 will be required to support the minimum data processing requirements for the NIFS readout methods. These sizes are based on pixels from four quadrants each of size 1024 × (1024 + 32) pixels, where each readout row (or column, depending on readout orientation) will usually have 32 reference samples appended.
Table 1: Data Storage Arrays – Linear Fitting.
|
Array Name |
Data Type |
Size (MB) |
Description |
|
|
|
|
|
|
pix x 2 |
U16 |
16.5 |
Raw pixels – double buffering (V). |
|
fitPix |
R32 |
16.5 |
Fitted pixels (f(V)). |
|
pixSum |
U32 |
16.5 |
Sum of raw pixels (SV). |
|
pixSqrSum |
U32 |
16.5 |
Sum of raw pixels squared (SV2). |
|
iPixSum |
U32 |
16.5 |
Sum of iteration times raw pixel (SiV). |
|
pixSumC |
U32 |
16.5 |
Sum of raw pixels after a cosmic ray (SV2). |
|
pixSqrSumC |
U32 |
16.5 |
Sum of raw pixels after a cosmic ray (SV22). |
|
iPixSumC |
U32 |
16.5 |
Sum of iteration times raw pixel after a cosmic ray (SiV2). |
|
cosmic |
U8 |
4.0 |
Cosmic ray flags (C). |
|
saturated |
U8 |
4.0 |
Saturated/bad pixels flags (S). |
|
pixCumSum |
R32 |
16.5 |
Cumulative sum for series of short exposures (Sf(V)). |
|
skyIdle |
R32 |
16.5 |
Sky image for idle mode. |
|
skyScience |
R32 |
16.5 |
Sky image for run mode. |
|
intensity |
R32 |
16.5 |
Fitted and unraveled pixels (F). |
|
variance |
R32 |
16.5 |
Variance for fitted pixels (E). |
|
Total |
|
222.5 |
|
Note that the data buffers for intensity, variance, and saturated data will be used when transferring the data to the DHS. Before transfer to the DHS begins pixels will be unraveled into correct array coordinates (see §4.7).
For the Double Correlated Sampling and Fowler Sampling methods, more than one pix array will be needed. Table 2 indicates the buffer requirements for N=16 (in the Fowler Sampling case).
Table 2: Data Storage Arrays - Double Correlated Sampling and Fowler Sampling
|
Array Name |
Data Type |
Size (MB) |
Description |
|
|
|
|
|
|
Pix x N |
U16 |
132.0 |
Raw pixels (V). |
|
fitPix |
R32 |
16.5 |
Fitted pixels ((SV)/N). |
|
pixCumSum |
R32 |
16.5 |
Cumulative sum for series of short exposures (Sf(V)). |
|
skyIdle |
R32 |
16.5 |
Sky image for idle mode. |
|
skyScience |
R32 |
16.5 |
Sky image for run mode. |
|
saturated |
U16 |
8.0 |
Saturated pixels flags (S). |
|
intensity |
R32 |
16.5 |
Fitted and unraveled pixels (F). |
|
Total |
|
222.5 |
N=16 |
Clearly, the NIFS detector controller will require more than 256 MB of RAM. The hardware that has been chosen is described in §5.2.4.
It is sometimes desirable to save all of the raw data from a particular readout method. The storage requirements in Table 1 and Table 2 assume that the data can be sent to the DHS in the time between filling the pix arrays. For linear fitting, this is the time between successive reads (i.e., the NDR period). For the other methods, it is the exposure time. Thus the data transfer rate places a lower limits on these times when all the raw data are being saved.
4.7 Unraveling Pixel Data During Readout
The NIFS detector is organized into four quadrants with one readout per quadrant. The readout orientation is rotated through ninety degrees from quadrant to quadrant. This means that two quadrants are column-oriented and the other two are row-oriented. This is illustrated in Figure 5. Pixel data will arrive in an interleaved order from the four output amplifiers. It will be necessary to unscramble these pixel data before they are sent to the DHS. It was decided not to support regions of interest (ROI) at the NIFS CoDR, so the added complication of unscrambling a selected region has been removed.
Figure 5: NIFS detector readout arrangement.
Pixels will arrive from the detector in the order
q11,1, q21,n, q3n,n, q4n,1, q12,1, q21,n-1, q3n-1,n, q4n,2, .. q1n-1,n, q2n,2, q32,1, q42,1, q1n,n, q2n,1, q31,1, q41,n
where a pixel is addressed by qNi,j with
qN quadrant number N,
i,j x and y location from the bottom-left corner of each quadrant
The Camera class needs to know how the detector is laid out to enable the unraveling of data. This is done by specifying the number, location, size, orientation, and readout direction of each output amplifier. This information is contained in a structure with the following fields:
nAmps Number of readout amplifiers
And the set of parameters listed in Table 3 for each amplifier.
Table 3: Detector Layout Parameters
|
Parameter |
Data type |
Description |
|
|
|
|
|
xo |
U16 |
x offset of first pixel |
|
yo |
U16 |
y offset of first pixel |
|
width |
U16 |
number of columns |
|
height |
U16 |
number of rows |
|
xdir |
S16 |
direction of readout – columns |
|
ydir |
S16 |
direction of readout – rows |
|
orientation |
Boolean |
orientation of readout – either row or column |
These parameters will be fixed for NIFS so will be set in an include file as constants.
The unscrambling algorithm used by the Readout task will work as described in the following pseudo-English code:
For each readout amplifier
For each row/column in detector quadrant
Compute output row/column based on readout
direction/orientation
Set index into input array to start of row/column
For each pixel in row/column
Compute output column/row position based on
readout direction/orientation
Copy pixel from input location to output
location
Increment input pixel pointer by number of
interleaved readouts
End loop over pixels
End loop over rows/columns
End loop over amplifier
The RSAA Camera class implements this algorithm as a C++ template method so that different input and output data types are easily handled.
The time needed for this processing is 0.58 s for the full detector size on the 400 MHz SVGM5.
4.8 Moving Data to the Gemini DHS and Quick-Look Displays
Data will be sent to the Gemini data handling system from the Data task:
· At the end of the readout for final storage and display.
· To quick-look displays for viewing during the integration.
· To quick-look displays during idle mode.
· After an intermediate readout (depending on method), if raw data capture is required.
4.8.1 The Data Transfer Task
The Data task waits on a signal from Readout that indicates an image is ready for sending to the DHS. Details of the image are given in an image header description and might be raw data, averaged data (from Double-Correlated Sampling or Fowler Sampling readout methods) or fitted data (from the Linear Fitting readout method). The unscrambled data are placed in a final intensity array for transport to the DHS.
4.8.2 Viewing the Image as it is Being Built
For the Linear Fitting readout method, it is possible to view the image periodically during the integration. As the detector will be read out non-destructively at regular intervals, a progressive image fit can be performed. These data will be sent to a quick-look display for viewing.
4.9 Writing Local FITS Files – the Fits Class
In the absence of the DHS or during testing and commissioning, it will be useful to record data by another method. RSAA has developed a C++ FITS class that can be used for this purpose. It is a general purpose FITS reader/writer and can be switched in when the DHS is unavailable. Whether or not this is done can be controlled by a CAD parameter.
The IOC should not be burdened with NFS writing. A suitable UNIX server process will be written for accepting FITS files and writing them to disk. This will be modeled on existing RSAA code that currently does this.
4.10 Exposure Control
Timing of an exposure will be performed by the SDSU-2 timing card. The Slave task will periodically monitor an exposure’s progress by checking that the exposure is continuing and updating status with internally counted timing values. This will be handled by a loop that sleeps for short periods and checks progress and updates status every wakeup.
4.11 Gemini Status and Alarms Database
Status and alarms will be reported through the EPICS SAD database. These SIR records will be updated from the NIFS Slave, Readout, and Data tasks through the use of a C++ class that provides a mapping between the C++ code and the EPICS database. This technique will isolate access to the EPICS database to the class and simplify the logic of the code in the NIFS tasks through encapsulation and data-hiding (see §5.1.5).
5 Spectrograph Detector Controller Software Implementation Detail
This section details the decisions made to implement the above design. In particular, the following issues will be discussed:
· Task structure and control flow.
· Data flows.
· Responsiveness.
· Performance.
· Throughput.
· Hardware issues.
5.1 Detector Controller IOC Task Diagram and Data Flows
The proposed task structure suggested in §4.1 is a broad overview of the processing design for the NIFS DC. This is expanded further in Figure 6. This diagram concentrates on the inter-process communication (IPC) design of the system. It is a client-server task model with Gemini systems forming client requests for the NIFS DC server (which consists of the IOC tasks Control (actually three SNL tasks), Slave, Readout, and Data).
Figure 6: NIFS DC task diagram.
5.1.1 The Control Tasks (dc_ss, SDSU_ss, and obs_ss)
The Control tasks are the hub of the design. They are written in SNL (based on the CICS DC SNL code) and are started when the IOC boots up and should always be running. Dc_ss initializes the lower level software environment and sets itself up to monitor the INIT, PARK, and REBOOT CAD records. SDSU_ss monitors commands that are required for setting up the SDSU controller electronics, including the TEST CAD. Obs_ss handles the OBSERVE, STOP, and ABORT CADs making sure the Gemini observation sequence requirements are met.
Dc_ss
handles initialization and re-initialization of the system sub-tasks Slave and Data, instantiates the DC
parameter and status objects, provides the system heartbeat, and sets up the Slave message queue. The parameter and status objects are implemented as C++
classes that handle the interface between EPICS CAD/SIR record values and
internal C++ data items. In the case of the status class, a set method will
ensure that changes to any data item are propagated to the EPICS SAD database.
Similarly, CAD parameter changes are passed to the parameter class using calls
from the CAD SNAM subroutines.
The state diagrams (Figure 7, Figure 8, and Figure 9) illustrate the behavior of the Control tasks.
Figure 7: Control task dc_ss state diagram.
Figure 8: Control task obs_ss state diagram.
Figure 9: Control task SDSU_ss state diagram.
A key requirement of the Control tasks is that they have to be responsive. The SNL code monitors the START and STOP links of the CAD records, enabling fast reactions to commands.
5.1.2 The Camera Task (Slave)
The Control task dc_ss starts the Slave task when it processes an INIT command (and automatically when the system is started). When Slave starts it will instantiate an SDSU2_IR camera object with constructor parameters set according to initial CAD parameter settings. Once created, the SDSU_init method is invoked to perform necessary controller initialization. Depending upon initialization configuration options, the SDSU_init method will initialize the VME controller interface, load the SDSU-2 DSP code for the timing and VME boards, set controller voltages, set the idle mode, and turn the power on.
The task then connects to a message queue for receiving further commands from the Control tasks. It then sits in its READY state. The POSIX message queue facility is used and a signal handler is set up with the mq_notify call for notifying Slave when a message is delivered by the SNL Control tasks. As commands are received they are interpreted and the requested actions are performed. The OBSERVE command will put the task into its observing MONITORING state where the observation sequence is monitored. Firstly the exposure is prepared using specified parameters. Integration then proceeds along with required readouts (depending on readout mode). Readouts are performed by the Readout task that is started in the preparation phase. The following state diagram (Figure 10) illustrates this process.
Figure 10: Slave task state diagram.
Commands to the SDSU-2 are written directly into the VME Interface Board’s command buffer by either the Slave or Readout task. The task then awaits a semaphore that indicates a reply is ready. When a command is complete, an interrupt is generated by the VME DSP code and an interrupt service routine (ISR) runs and gives the semaphore to the waiting task.
Monitoring is done in a loop that periodically sleeps for short intervals and then wakes to check if the exposure has completed. Status is updated with the exposure counter maintained using the VxWorks clock.
During this MONITORING state, Slave needs to remain responsive to Control commands so that interrupts such as ABORT, STOP, and REBOOT can be attended to.
5.1.3 The Detector Readout Task (Readout)
Readout is the task that sends the start exposure command to the SDSU-2 VME interface and waits while it proceeds. It is started by Slave and, depending on the readout method, prepares itself to run the exposure. It sets up an interrupt handler to be called in the case of either an ABORT, STOP, or REBOOT command which will cause the task to exit after appropriate cleanup.
As a readout proceeds and data is accumulated in image buffers prepared in the initialization phase, the Data task is signaled when data is ready for transfer. These signals are generated when the SDSU-2 VME DSP code executes an IOC interrupt at predetermined intervals of the readout. The ISR will place a message on the Readout task’s readout message queue and signal that a message is waiting. Readout reads the message, updates the status buffer, and processes and unravels the data in the raw pix buffer placing the results in the fitpix array. It then gives a semaphore to Data to indicate data is ready for transfer. The following state diagram (Figure 11) illustrates this process.
Figure 11: Readout task state diagram.
Timing aspects of importance in this design are:
- Responsiveness to ISR messages.
- Speed of calculating and unraveling processed data.
- Handshaking with the Data task.
These issues are discussed in §5.2.
5.1.4 The Data Transfer Task (Data)
Data is solely responsible for getting image data transferred out of the system. It handshakes with the Readout task on one side and either the DHS or a FITS server on the other. In between it prepares data for transfer by shifting it from fitpix memory to the DHS data structures. It also performs all necessary DHS data structure set up. Data also handles interaction with any quick-look displays that are in use.
Data is started by Control and waits for a semaphore to be handed it by Readout when a data buffer is ready for transfer. It interprets the contents of the data buffer and performs the transfer. Data to be transferred will include:
- Raw data and, optionally, variance and quality arrays at the end of an exposure for permanent storage and display (up to 41 MB).
- Optionally, raw data after each NDR for possible permanent storage and/or transient display (16.5 MB).
- Optionally, the compressed data cube for transient display (8 kB).
Figure 12 illustrates the Data task’s states.
Figure 12: Data task state diagram.
There are potential bottleneck problems with this process that need to be considered. These bottlenecks can occur in two places:
1. Readout can be blocked while Data holds the image semaphore.
This is unlikely as Data merely transfers the data from the fitpix array, which is already unscrambled by Readout, to the intensity array.
2. Data can be blocked by the DHS.
This, in turn, would cause a block further along the processing chain in Readout. If this happens, the blocked transfer will simply be missed and Readout will then proceed to its next step.
Both these potential blocks are discussed further in §5.2.2.
5.1.5 The Interface Between the NIFS C++ Code and EPICS.
As described earlier, NIFS will use C++ objects to store operational parameters and status information. This technique will provide a simple unified interface between the two systems. These objects include STL[1] hash dictionaries keyed by variable name. These dictionaries will have attributes for current value, EPICS process variable name, and validity checking parameters. Methods will be used to set up and maintain the dictionaries. When these objects are instantiated, the dictionaries will be populated with all the NIFS parameters and their attributes. The following code sample shows how this will be done:
nifsParameter->add(“nreads”,
”nifs:dc:seqObserve.C”, 0, min, max);
nifsParameter->add(“qlStream”,
”nifs:dc:setDhsInfo.A”, “nifsRunQL”, maxlen);
nifsParameter->add(“readMode”,
”nifs:dc:seqObserve.A”, LINEAR, readModeSet);
.
nifsStatus->add(“timeLeft”,
”nifsSad:dc:timeLeft”, 0);
Arguments to the add method are parameter name, EPICS variable name, value, and validity values.
Note, that the add method is overloaded for different data types. The data type is remembered and used for any data conversion required from the CAD input string values. Upper and lower value bounds are also specified for numeric values, maximum string length for strings, and set membership for sets. During operation these dictionaries can be updated or queried using set and get methods. CAD record INAM routines will be written for parameters that need initializing at system start up. This will be when the add method is used.
The use of these anonymous dictionaries in these classes adds much flexibility, the classes are context free, and each object can be readily extended with new parameters as they are required without code modification. Indeed, they could be easily used in other instruments requiring an interface between VxWorks code and the Gemini EPICS environment.
In the case of the operational parameters, the CAD SNAM routines will use the set method of the nifsParameter class to store and verify a CAD input parameter. An example of this use is as follows for the nreads parameter:
status
= nifsParameter->set(“nreads”, pcad->C);
If the parameter setting is valid, the
class stores nreads in an STL hash dictionary, its value will be copied
to the CAD record’s OUT field, and the return status is set to True.
Otherwise status is set to False and the value remains unchanged.
For status parameters, we take another example. The Slave task will maintain an exposure counter named timeLeft. This will be set with the following method call:
timeLeft
= exposureRequested – (currentTimestamp – startTimeStamp);
nifsStatus->set(“timeLeft”,
timeLeft);
Each parameter in nifsStatus is stored in another STL hash dictionary that is keyed by name and has a value and a SIR record name as attributes. The set method looks up the dictionary and deposits the new value in both the dictionary and the SAD database. The get method simply returns the current dictionary value.
Access to these parameter class objects is also controlled through the use of a semaphore to ensure that only one reader/writer is operating at the same time.
5.2 Performance Issues
Performance issues that need to be considered in the NIFS DC software model include:
- Command responsiveness.
- Readout mode processing.
- Inter-task communication.
- Data throughput.
- Quick look display performance.
These are examined in turn.
5.2.1 Command Responsiveness
Gemini requires NIFS to respond to all CAD commands with a CAR record within 1 s (§5.1.1). This requirement is met by use of SNL code processing of CAD commands that require action and by direct EPICS processing of no-op commands.
5.2.2 The OBSERVE Command – Processing Timeline
The OBSERVE command, in Linear Fitting readout method, proceeds according to the processing timeline illustrated in Figure 13.
Figure 13: Observe command processing timeline.
The array is read out at equal intervals during the exposure, taking the required 5 s to clock out all the pixels. At the end of this time (which we call CT) an IOC interrupt is delivered by the SDSU-2 VME DSP code. The Readout task responds by processing the data and handing a semaphore to Data. This takes time RT. Data moves the data to the transfer buffers and relinquishes the semaphore, taking time ST. It then transfers the data to the DHS and the quick look displays, taking times QT and DT.
The periodic clock out of pixels will not pause, so for all the data clocked to be processed in time, the Readout task will need to be ready to go every CT seconds.
Prototyping has revealed that RT = 3.9 s on the 400 MHz SVGM5. This means that the Readout task has some headroom for processing the data, even in the most CPU intensive linear-fitting readout method.
To alleviate the potential bottlenecks with the Data task, the design could incorporate a double buffering technique with the pix and fitpix buffers to extend the time available for processing and transferring data. There will be a limit on how far this measure can be taken because of available memory. We plan to see how the system performs before implementing extra time saving techniques.
This picture is slightly different for the Double-Correlated Sampling and Fowler Sampling readout methods. The time between successive read outs of the array, E, can reduce to zero for bright objects (i.e., an exposure time of 5 s). Then there is no time for the Readout task to perform any data processing. In this case, the processing required is averaging (for Fowler Sampling) and subtracting images. We will incorporate a second pix array that can be filled while the first one is processed. This will make available an additional time of 5 s which is greater than RT.
5.2.3 Data Transfer Throughput
· Gemini DHS Performance Specification
We need to know how fast the DHS can take data delivered over the Fast Ethernet link. Fast Ethernet itself will restrict this rate to a maximum of around 8 MB/sec but this might well be an upper limit. Current information from IGPO indicates that the DHS has only been tested to handle 2.8 MB/sec but further optimizations might be able to be made (Boyer, priv. Comm. Jan 2001).
· Quick-Look Display Performance Specification
IGPO have