Interactive DPS/Mosflm/CCP4 Data Processing
Version 2.01


General information, and starting the system

This system may be used to process images from a variety of area detectors (see Detector types for a list). The user interface has been developed through a collaboration between Area Detector Systems Corp. and Cornell and Purdue Universities. The working routines include "DPS" programs from Purdue and Cornell, an image display program from ADSC, and the Mosflm/CCP4 package from Andrew Leslie and the CCP4 group.

Development of the software is continuing. This documentation is for Version 2.01, released August 2001. Comments on the programs themselves, and on this documentation, are welcome; send them to Marian Szebenyi.

To process a set of data, type "process" for an initial run. If you already have a parameter file (created by a previous run of the program), you can type "process name", where "name.param" is the name of the parameter file. Process is a script that copies the necessary shell scripts ("command files") from a standard location to the current directory (if they're not already there), starts up processing_gui, and loads the specified parameter file, or a standard prototype if no filename is given. The main interactive window, part of which is shown here, will come up.

Processing_gui stores information in a parameter file, whose name generally (but not necessarily) has a ".param" extension. This file is updated during processing and can be read in on a subsequent run. Area (1) of the main window allows you to specify a parameter file, load parameters from the file, and save parameters to the file. If you specify a non-existent file and click "SAVE", a new file will be created. It is good practice to make a new parameter file for each dataset: start by loading a file from a similar dataset, or the prototype, change parameters as necessary (see below), change the file name, and save it.

A number of "command files" are needed to run the various programs which are part of the system. These are kept in a standard location and copied to the current working directory before use. This is done automatically by the process script, but can also be done manually by clicking the "Get Cmd Files" button (2). You may need to do this if you have edited the command files and want to reset to the standard values, for instance. To edit command files before executing them, click "Edit Cmd File Before Execution". See command files section for much more about command files.

Area (3), and the "Directories:Images" field above it, specifies the image files to be processed. Files are sought for in the "Images" directory, and filenames are of the form "prefix_run_sequence.suffix", where "prefix" is given by the "Image Prefix" field, "run" by the "Run No." field, suffix by the "Filename suffix" field in the "Detector Parameters" window (see below), and "sequence" runs from "Start Image No." to "End Image No.". Alternatively, the "Run No." field may be left blank and filenames are of the form "prefix_sequence.suffix". For indexing, and initial refinement, you will probably use only a subset of the complete set of frames (as specified in the indexing and refinement windows); "sequence" gives the range that will be integrated.

Area (4) contains a few basic parameters for this dataset. Other general parameters are shown, and can be changed, in the windows which you can pop up using the buttons at the top right (5). In particular, be sure to set the correct detector type and direction of spindle rotation in the "Machine Parameters" window. To change a parameter, type the new value in the appropriate text field. To save updated parameters to the current parameter file, click "SAVE". To save them to a new parameter file, enter a new name in "Parameter File" and click "SAVE".

To stop a process which has been started from the GUI, click "STOP". To exit from the GUI itself, click "CLOSE" (6).

Follow the flow chart across the middle of the GUI: indexing, refinement, integration, scaling, final output. The display and strategy steps are optional; an image display is produced automatically at various stages, and the strategy calculation is only useful when preliminary processing is done prior to data collection.


Detector types

Supported detector types are listed in the menu which appears on clicking the "Detector Type" button in the "Machine Parameters" window. Currently, they include: Clicking "Show Detector Parameters" posts a window containing information about the current detector: number of pixels and pixel size in each direction, origin and orientation of scan axes, size of header, and suffix for image file name. When a new detector type is selected from the menu, standard values for that type will be set in the "Detector Parameters" window. These may be updated from the file header when the first data file is read for indexing, and can also be changed manually. For the generic detector, the appropriate values must be entered manually; no header information will be read even if the files have headers.

When setting the detector type, be sure to also set the direction of rotation and orientation of the spindle axis, in the "Machine Parameters" window. These depend on the particular camera on which the detector is mounted. At CHESS, for example, the spindle orientation is always horizontal, but the rotation direction is clockwise at the F-1 and F-2 stations and anticlockwise at A-1.


Indexing

Before indexing, experimental parameters such as crystal-to-detector distance, wavelength, and angular settings must be set in the "Machine Parameters" window. If the first image file (as specified by the image prefix, run number, and starting image number given in the main window) exists, values may be obtained from the file header by clearing a text field and hitting C/R. If files from the current detector type have no header, or have an unreasonable value in the header, a value of 0 will be returned. Manually update any fields with missing or incorrect values. Note the following about the machine parameters:

The indexing process includes visual inspection of one or more diffraction patterns, location of diffraction spots, and determination of the unit cell parameters and orientation of the crystal that produced the image(s). Two means are available for this operation; they are described below under the headings of Index:DPS and Index:MOSFLM. DPS indexing usually works well and requires only one image (but often gives more accurate results with multiple images). Mosflm indexing gives the options of DPS or REFIX indexing. DPS indexing via mosflm differs from direct DPS indexing in the peak search routine used, and in the pre-refinement which mosflm does immediately after indexing; REFIX-style indexing is more touchy and often requires more than one image to succeed.

In either case, activation of an indexing button first checks for the existence of the image file specified by the "Images" (directory), "Image Prefix", "Run No.", "Start Image No.", and "Filename suffix" (in the "Detector Parameters" window) fields. The filename is of the form "prefix_run_startno.suffix" (run number given) or "prefix_startno.suffix" (run number field blank), where "startno" must be 3 digits long. If the file does not exist in the image directory, an error message will appear in the message window. Following the check, the appropriate window is brought up - see details below for the individual indexing programs. Indexing determines the unit cell and orientation of the crystal - these are stored in a mosflm-format matrix file, and the unit cell is also kept in the parameter file. If you know the unit cell very well and it differs slightly from the one determined by indexing, you can force a change by entering the desired cell in the "Cell Parameters" window and clicking "Update orientation matrix with this cell". This only works for small cell changes - you cannot swap axes or change symmetry this way.

After indexing, proceed to refinement.


Index: DPS

DPS indexing is a Fourier-based procedure which searches for periodicities along directions in reciprocal space. It works on one or more images. By default, it will use the image specified in the "Start Image No." field, and the starting phi and delta phi values from the Machine Parameters window. If this is a good image and the program is using the correct distance, wavelength, and direct beam position, a good solution can generally be obtained. If not, you can select a different image, or use multiple images. At the top of the Autoindexing window is a field labeled "Frame[:phi]'s to Use". In this field specify the images you wish to use, separated by spaces. Each image can be given as either a single number or as two numbers separated by a colon(:). If two numbers are given, the first is the image number and the second is the starting phi value for that image. If one number is given, the starting phi is calculated from the image number, assuming that all images have the same delta phi, there are no discontinuities in the sequence of images, and the image specified in "Start Image No." has the starting phi given in the Machine Parameters window.

If you know the unit cell of your crystal, you may enter it in the "target cell" field and turn on "Use target cell". The indexing routine will not strictly enforce the given cell, but will favor solutions near to that cell. If no good solutions are close to the target cell, "Use target cell" has no effect.

To proceed with indexing, follow the sequence of buttons below the frame selection field.

"Find peaks" starts up dps_display to find and display peaks (shown as blue diamonds) for the first image. The default set of peaks found will be appropriate for most images, but if it is not you can use the "Peaks" option of the dps_display "Edit" menu to bring up a window showing the peak search parameters: change as needed and use "Find peaks" to get a new set of peaks. In difficult cases it may be necessary to manually add or remove peaks; turn on the appropriate toggle button and use the mouse to select peaks. Indexing works well with 500-1000 peaks, provided that there are that many good spots on the image; a small unit cell or poorly diffracting crystal may give fewer.

If you are indexing on frame(s) other than the first, change the display to each of the other frames in turn by typing the new frame number in the "Frame:" field, or by using the up and down arrows to step one frame at a time. As long as the "Peaks" toggle button is on, new peaks will be automatically calculated whenever a new frame is loaded. Peaks for each file are stored in a file image_name.peaks, where image_name is the name of the particular image file; this file is kept until explicitly deleted or until overwritten by another peak search on the same image. This means that if you want to reindex with a different selection of files, you need only do a peak search on files which have not been used in any previous indexing.

The grayscale for image display can be changed using the sliders to the right of the "Predictions" button: adjust the left slider to change the bottom of the white-to-black range and the right slider to change the top of the range.

Be sure the cross indicating direct beam position is on the direct beam spot. To check it, zoom in on the area around the direct beam by clicking on the beam spot with the right mouse button. Change the magnification in the zoom window using the text field and/or arrows at the top of the window. If the cross is not centered on the spot, click on the correct position with the middle mouse button. Confirm the new position in the window that pops up. Note - the new coordinates will be written to the current parameter file but to update the "Machine Parameters" window you must click "LOAD" on the main GUI. Failure to click "LOAD" will result in the beam position being reset to the old values.

Set the "Maximum Cell Edge" field to be comfortably larger than the longest unit cell edge you expect to have (e.g. for a longest cell edge of 250 Angstroms, enter 300). If you don't know the unit cell parameters, but don't expect any very long ones, the default value of 300 is a good place to start. Adjust the resolution limits for indexing if necessary: usually it is appropriate to have a wide range and use all the picked peaks, but you may have to limit resolution if you can't come up with peak-picking parameters that find enough real peaks without picking up too many noise peaks, or to avoid using spurious peaks from ice rings. Then click "Autoindex". After dps_index completes its run, possible solutions in different symmetries will be listed:

Primitive Cubic        :    6.48 :  55.98  62.22 129.92    90.25  90.84  89.79
                                    82.71  82.71  82.71    90.00  90.00  90.00

I Centered Cubic       :   11.82 : 140.71  83.85 143.80    71.92  34.13  75.25
                                   122.79 122.79 122.79    90.00  90.00  90.00

F Centered Cubic       :   14.58 : 154.92 153.70 154.00   132.29  65.49 137.46
                                   154.21 154.21 154.21    90.00  90.00  90.00

Primitive Rhombohedral :    6.32 :  62.22  55.98 130.04    89.16  89.72  89.90
                                    82.75  82.75  82.75    89.59  89.59  89.59

R Centered Hexagonal   :    6.32 :  83.54 140.71 155.55   128.98  87.06 105.04
                                   112.12 112.12 155.55    90.00  90.00 120.00

Prim Trigonal/Hexagonal:    5.70 :  55.98  62.22 129.92    89.75  90.84  90.21
                                    59.10  59.10 129.92    90.00  90.00 120.00

Primitive Tetragonal   :    1.27 :  55.98  62.22 129.92    90.25  90.84  89.79
                                    59.10  59.10 129.92    90.00  90.00  90.00

I Centered Tetragonal  :    4.07 :  55.98  62.22 271.98    76.98  78.89  89.79
                                    59.10  59.10 271.98    90.00  90.00  90.00
Primitive Orthorhombic :    0.27 :  55.98  62.22 129.92    90.25  90.84  89.79
                                    55.98  62.22 129.92    90.00  90.00  90.00
C Centered Orthorhombic:    1.23 :  83.85  83.54 129.92    89.62  90.75  83.96
                                    83.85  83.54 129.92    90.00  90.00  90.00

I Centered Orthorhombic:    4.43 :  55.98  62.22 271.98   103.02 101.11  89.79
                                    55.98  62.22 271.98    90.00  90.00  90.00

F Centered Orthorhombic:    3.14 :  83.85  83.54 271.98    92.22 107.21  83.96
                                    83.85  83.54 271.98    90.00  90.00  90.00

Primitive Monoclinic   :    0.12 :  55.98  62.22 129.92    90.25  90.84  89.79
                                    55.98  62.22 129.92    90.00  90.84  90.00

C Centered Monoclinic  :    1.11 :  83.85  83.54 129.92    90.38  90.75  96.04
                                    83.85  83.54 129.92    90.00  90.75  90.00

Primitive Triclinic    :    0.00 :  55.98  62.22 129.92    89.75  89.16  89.79
                                    55.98  62.22 129.92    89.75  89.16  89.79

Expect a penalty under 0.5 for the correct symmetry, for a good image and correct machine parameters. If the image is weak, or there is an error in distance or direct beam position, a penalty in the range of 0.5 to 1.0 may be obtained for the correct symmetry (see below for a more complete discussion of this point). In this example, primitive orthorhombic is the most likely symmetry.

The entry for the symmetry currently specified in the parameter file will be highlighted. In addition, a display of predicted reflections for the first image using that solution will be produced. In the display, fully-recorded reflections are green, partials are yellow, and overlapping reflections are red. Partiality is determined using the current mosaicity (in the main GUI window) and overlaps are determined using the current minimum spot separation (in the Integration Parameters window, which pops up when you click "Integrate:MOSFLM"). To change to a different symmetry, click on the appropriate entry. This will update the symmetry entries in the parameter file and the Cell Parameters window, and display predictions using the new cell. To save the results from autoindexing directly to a matrix file (whether or not you have changed symmetries), click "Save result without refining" (in the Autoindexing window). To do some refinement before writing a matrix file, click "Refine result (DPS refine)". If you do a direct save now, you will still be able to refine later.

Problems

Note that it is possible for the correct symmetry to have a high penalty if the distance, wavelength, or direct beam position supplied is wrong. If you know the symmetry and you get a high penalty for it, check these parameters. Be sure that the direct beam position has been entered in the correct coordinate system - note switch at top of "Machine Parameters" window.

Try using different image(s). Some images, particularly from a mosaic crystal, do not provide enough information for accurate determination of all cell and orientation parameters. Use of multiple images will alleviate this problem; they should be separated by at least 10 degrees or so in phi - be sure to associate the correct phi values with each frame.

Other things that may help are to to pick more or fewer peaks on each image, to change the resolution limits for indexing, or to vary the "Maximum Cell Edge" value. In particular, for large cell dimensions it can help to increase the maximum cell edge, to as much as twice the actual value.


Index: MOSFLM

Mosflm 6.10 has two indexing options; the "new" indexing uses the same basic algorithm as dps_index, but your results may differ from those of "Index:DPS" due to differences in the peak search routine and in the "pre-refinement" of a few parameters that mosflm does after indexing.

The alternative "old", or "REFIX", or "Kabsch", option uses a different algorithm based on assigning inter-peak vectors to reciprocal space basis vectors. It uses one or more images. It is less reliable than DPS indexing for a single frame but will generally succeed if you can give it enough images which are separated by at least several degrees in spindle angle.

To use mosflm for indexing, click on "Index:MOSFLM". An xterm titled "Xmosflm" will appear. At the MOSFLM => prompt, type "@index.inp". A large window will appear with a display of the first image and a number of control areas. Note that the image orientation differs from that in dps_display; the spindle rotation axis is horizontal in the dps_display display but vertical in the mosflm display if it is "Parallel with: Horizontal" in the "Machine Parameters" window.

Image display parameters are controlled by the panel above the image. To change minimum or maximum value, click on the number, type a new value, and hit C/R. To adjust the contrast, use the slider. To change the color scheme, click on the triangle with the right mouse button and select from the menu. The "Mag" menu refers to the magnification in the small image at the top left of the main one, which is a display of the area immediately around the mouse's current position. Note - this area can be enlarged by clicking in it with the middle mouse button. To zoom in on an area, click on Zoom and define the rectangle to be displayed by putting the mouse on its top left corner, holding down the left button, and dragging to its lower right corner. To unzoom, click once on Zoom to change it to Full, and then again (on the button now labeled Full) to execute.

Clicking on a spot in the image displays its coordinates in the Output block to the lower left of the image. The red circle indicates the beamstop position, as specified in the parameter file. The processing parameters down the lefthand side come from the parameter file. They can be changed by clicking on the value to change, typing the new number, and hitting C/R. Buttons in the "Main menu" block control actions.

To get the direct beam location: zoom in on the central region and click on the direct beam spot. The XC,YC coordinates in the "Output" panel are the values you need. Transfer them to the "Beam X" and "Beam Y" fields in the "Processing params" area of the mosflm display, and also to the "X Beam" and "Y Beam" fields in the "Machine Parameters" window of the main GUI. Be sure that the mosflm coordinate system is selected in "Machine Parameters" when doing this.

For REFIX-style indexing, it is suggested that you use two images (if available), preferably well-separated in phi. For DPS-style indexing, a single image can be used but two or three will usually give a better result. Starting from the display of the first image, proceed as follows:

Click on "Find spots". An "xdl_io_window" will come up with some information about spot-finding. A narrow red box will appear on the image showing where data for a radial background correction will be taken. If the box overlaps an unusable strip between CCD modules, answer "n" to the question in the xdl_io_window, change the "Y offset" parameter, and click "Find spots" again. If box location is correct, answer "y" to the question.

A summary of spots found on the first image will appear in the window. Hit C/R to dismiss the window and continue. Picked spots show as red crosses on the image display. If the number of spots is too large or small, change "Threshold" and try again.

To index using spots from a single frame, click "Autoindex" now. To use spots from a second frame also, first do this:

Click "Read image". In the window that comes up, request another image (e.g. 10). Accept the default phi values, unless there was a discontinuity in data collection and they are not right. Repeat "Find spots" for this frame. If desired, repeat "Read image" and "Find spots" for additional frame(s). Then click "Autoindex".

In the window that comes up when "Autoindex" is activated, answer "y" or "n" to "use new indexing?", depending on whether you want DPS or Kabsch indexing. Then answer a few more questions; generally the default values are fine. Confirm that it is OK to proceed. Another window will come up with results of indexing.

List of possible Laue groups, sorted on penalty index.
The lower the PENALTY, the better
Only solutions with PENALTY less than 200 are listed, a complete list is given in the terminal window
 No PENALTY LATT    a        b        c     alpha  beta gamma  Possible spacegroups
 10 122     oC    84.41    84.42   130.81    89.9  90.2  84.1   C222,C2221
  9 122     tP    56.51    62.71   130.81    90.1  90.2  90.0   P4,P41,P42,P43,P422,P4212,P4122,P41212,P4222,P42212,P4322,P43212
  8 119     mC    84.41    84.42   130.81    90.1  90.2  95.9   C2
  7 119     mC    84.41    84.42   130.81    89.9  90.2  84.1   C2
  6   6     oP    56.51    62.71   130.81    90.1  90.2  90.0   P222,P2221,P21212,P212121
  5   6     mP    56.51   130.81    62.71    90.1  90.0  90.2   P2,P21
  4   5     mP    62.71    56.51   130.81    90.2  90.1  90.0   P2,P21
  3   2     mP    56.51    62.71   130.81    90.1  90.2  90.0   P2,P21
  2   0     aP    56.51    62.71   130.81    89.9  89.8  90.0   P1
  1   0     aP    56.51    62.71   130.81    90.1  90.2  90.0   P1

Select a solution AND a spacegroup from list above (eg 3 p42) or 0 to abandon or T to change min I/sig(I):6 P212121
Running refix again with this symmetry imposed
New orientation matrix written to (Keyword NEWMAT):test_1.mat
Using 1706 indexed reflections, final sd in spot positions is 0.05mm and in phi 0.24 degrees
Refined cell parameters   56.47   62.71  130.74  90.00  90.00  90.00

Do you want to update cell parameters (Y):

Do you want to accept this solution (Y) :

Pick the solution of highest symmetry with a reasonable penalty - generally there will be an obvious break between high-penalty bad solutions and low-penalty good solutions. Type the number of the line with the highest symmetry good solution and a space group name, e.g. "6 P212121". (If there is no reasonable solution, type "0" to reject all solutions and see "Problems" section below). A little more processing will be done and you will be asked whether to accept the results. The dialog above is from a Kabsch indexing run; the sequence is slightly different for DPS indexing, but in both cases the final question is "accept this solution?". If you answer "no", you will have a chance to reindex with different spot-selection criteria. If "yes", a matrix file containing cell and orientation information will be written.

Use the "Predict" button to generate a predicted diffraction pattern. If picked spots are confusing the display, use "Clear spots" to remove them; answer "y" to the "simply remove spots from display?" question. Predictions may be removed and replaced as often as desired using the "Clear predictions" and "Predict" buttons.

Check crystal mosaicity by comparing real and predicted spots along a lattice row. If there are extra predicted spots at the end of the row, the mosaicity being used is too high; if there are real spots which are not being predicted, the mosaicity used is too low. To estimate the correct value, change the "Mosaicity" field in the "Processing params" list and click "Predict" again. Repeat until predicted and real spots correspond as well as possible. For integration, use a mosaicity value slightly above this.

The effect on the predictions of changing other parameters than mosaicity may also be examined. For a parameter in the "Processing params" list, click on the old value, type in a new value, and hit C/R to store it. For a parameter not on the list, click "Keyword Input" to bring up a dialog window, type the parameter name and value (e.g. "SEPARATION 0.4 0.4"), hit C/R, then type "GO" to close the window and return control to the mosflm interface. Click "Predict" to see the prediction using the new parameters.

To continue after indexing, use the "EXIT" button to close the display window. A window will come up to ask "save current parameters?" If you answer "y" and accept the default file name, the unit cell, symmetry, wavelength, direct beam position, two theta, and distance will be passed to the main GUI; if you answer "n", or give another file name, you must manually transfer any needed information from the mosflm display to the windows of the main GUI. Note that any other parameters that you may have changed will not be transferred - you must modify settings for minimum spot separation, etc. manually. Answer "y" to the "exit?" question. In the "Xmosflm" window, run a strategy calculation if you wish (see "Strategy" section below). Then type "end" (the main GUI will be unresponsive until you have terminated mosflm).

Problems

Autoindexing can fail to find a solution (the message "Autoindexing failed" appears), or can find a poor solution (predictions do not match real spots). Try the following to improve matters:


Strategy for data collection

Once an image has been indexed, it is possible to calculate an optimum strategy for data collection. For a single-axis goniostat such as the ones at CHESS, this means determining the start and end of a total sweep in phi that will give data as complete as the crystal orientation and machine parameters allow. Mosflm has a "STRATEGY" option to do this. To use it, click "Strategy: MOSFLM" on the main GUI flow chart. In the "Xmosflm" window which appears, type "strat.inp". If you have just done mosflm indexing and have closed the image display but not removed the "Xmosflm" window, you can get the same result by typing the following in that window:

 MOSFLM => strategy
Strategy option will be run in automatic mode.
 Speedup factor will be calculated automatically (use keyword SPEEDUP to set explicitly).
 Type GO to continue, or ABORT to stop strategy run.
 MOSFLM => go

Output from the strategy option looks like this:

 Default speedup set to     46.0
                       .
                       .
                       .
 Optimum rotation gives  97.8% of unique data
 This corresponds to the following rotation range(s):
 From  132.0 to 162.0 degrees
 Type "STATS" for full statistics.
                       .
                       .
                       .
 STRATEGY =>
Optionally type "stats" to get detailed statistics about completeness and redundancy over the calculated range. Type "end" to return to the "MOSFLM =>" prompt.

To get statistics for a particular range of spindle angles (for example, 0 to 90), do this:

 MOSFLM => strategy start 0 end 90
 MOSFLM => go
 Default speedup set to     46.0
                       .
                       .
                       .
 COMPLETE option
 ===============
 Give the segments of data to be tested, in the form:
 START 0 END 20 (RUN 1)
                       .
                       .
                       .
 STRATEGY =>
Enter the range of interest, which must be contained within the initial range given (0 to 90 in this case) as "start 0 end 45", for instance. Then type "go" to start the calculation. This sequence may be repeated for other ranges of angle. When done checking ranges, type "end" to exit strategy mode.

To get information about appropriate oscillation widths for your frames, you can use the TESTGEN option:

 MOSFLM => testgen start 0 end 90
 MOSFLM => go
                       .
                       .
                       .
 TESTGEN OPTION
 ==============
 Generating from phi =   0.00 (START) to  90.00 (END) testing overlaps
 every  5.00 degrees (STEP).
 At each phi value, the oscillation angle which results in less than
   0.0% overlaps (OVERLAP) will be determined, subject to a minimum oscillation
 angle of 0.20 degrees (MINOSC) and a maximum of  5.00 (MAXOSC)
 Testing at phi   0 degrees
 Testing at phi   5 degrees
                       .
                       .
                       .
 Suggested data collection stratgey for a maximum spot overlap of  0.0%

 Phi start  Phi end   no of images  oscillation angle  %age overlaps %age fulls
    0.0       5.0            10           0.50             0.0          23.8
    5.0       9.9             9           0.55             0.0          23.8
    9.9      19.7            15           0.65             0.0          32.3
   19.7      24.9             7           0.75             0.0          37.5
   24.9      34.7            15           0.65             0.0          31.4
   34.7      44.9            12           0.85             0.0          43.8
   44.9      50.1             5           1.05             0.0          50.7
   50.1      54.9             5           0.95             0.0          46.8
   54.9      65.1            12           0.85             0.0          41.5
   65.1      74.9            28           0.35             0.0           8.0
   74.9      84.8             2           4.95             0.0          86.6
   84.8      90.8             7           0.85             0.0          42.2

 ***** IMPORTANT *****
 The suggested values depend critically on the estimates of mosaic spread
 and the SEPARATION parameters. These values MUST be realistic, or you may
 not end up with a complete dataset.
 In particular, the spot separation parameters should be at least as large
 as the spot size in the centre of the image.
 Current values: Mosaic spread 0.11  Spot separation  0.60  0.60mm

 These are the MAXIMUM possible oscillation angles.
 A better signal to noise may be achieved by using a smaller oscillation
 angle if the oscillation angles suggested are greater than the rocking width
 (mosaic spread plus beam divergence)

 MOSFLM => 
Exit from mosflm by typing "end" at the "MOSFLM =>" prompt.


Refinement

Refinement of distance, direct beam position, and other "camera constants" is automatically carried out during integration. However, it is usually desirable to refine unit cell parameters, crystal orientation, and mosaicity in a preliminary run before integrating the whole set of images. There are three ways to do this, which are described below under the headings Refine (dps_refine), Refine (mosflm), and Refine (postchk). If you have only a single frame, or a few non-contiguous frames, only dps_refine can be used. If you have a contiguous series of frames, either dps_refine or mosflm, or both, can be used; see details below for the different strengths and limitations of the two programs. Postchk refinement can be tried if the others fail; it is fairly robust, but may give poor results if the frames used do not cover a large enough spindle angle range or are of poor quality. It also requires that the initial indexing be good enough to locate and integrate spots on a series of frames, before any refinement of cell and oreintation.

To start refinement, use either the "Refine result (DPS refine)" button in the "Autoindexing" window or the "Refine" button on the main GUI. Select the desired program in the "Refinement Control" window, select the items to be refined and the frames to be used, and click "START REFINEMENT". Follow refinement progress by watching the message window of the main GUI, which monitors the log file being generated by the process. When refinement is complete, new parameter values appear in the "Results:" area of the Refinement Control window, and dps_display is started to show new predictions for the first image. Refined parameter values are saved to the parameter and matrix files when you click one of the "Save" buttons - usually "Save Everything", but it is possible to save a subset of the refined parameters. Refinement can be repeated as often as desired: after saving refined parameters, change options (program to use, parameters to refine, frames to use) if desired, click "START REFINEMENT" again. When you are satisfied with the results, use "CLOSE" to close the Refinement Control window and proceed to integration.

Note that for integration it is wise to reset the mosaicity (on the main GUI window) somewhat higher than the final refined value, to allow for residual errors in cell and orientation parameters, and for anisotropy in mosaic spread.

Details for the three refinement options are given below.


Refine (dps_refine)

Dps_refine follows the Rossmann approach of cyclic refinement of the "Q" and "A" matrices, followed by a final step of mosaicity estimation. The Q-matrix incorporates information on beam position, distance, and detector rotations, while the A-matrix depends on unit cell and crystal orientation. Shifts in Q-matrix elements, and in cell parameters, depend on the discrepancy between observed and calculated spot positions. Crystal orientation, however, is adjusted to maximize the total integrated intensity of all reflections.

Single or multiple frames may be used. The program default for "Frames to use in refinement" is the first frame only, but better results may be obtained using several frames, e.g. "1 5 10". Typically, frames used should range over 10-30 degrees in phi; a larger range gives more information on cell parameters, but if the range is too large the initial indexing may not be good enough to locate spots on some frames.

The current algorithm used by dps_refine for mosaicity estimation is unsatisfactory - you will do better to refine mosaicity using mosflm (if possible) or just estimate it by comparing observed and predicted spot patterns. The recommended initial set of items to refine is "Unit Cell", "Xtal Orientation", and "Beam/Distance". On later cycles, "Detector Rotns" may also be refined.

Note - for Fuji imaging plate (or other offline scanner) data - you can only use one image, as there is currently no provision for providing multiple direct beam positions and detector rotations.

Click "START REFINEMENT" to start up the actual refinement process. If you want to take a closer look at the log file during refinement, you can turn off window updating using the "Pause" button in the corner of the message window; then use the scroll bar to move around in the file. To resume updating, use the "Resume" button.

When refinement is complete, the log file window will show a series of matrices, followed by "Done.", for normal termination. A display of the new predictions for the first image will be presented, and a summary of the results will appear in the "Results:" panel of the "Refinement Control" window.

If "Done." does not appear, something went wrong. Scroll back the text in the window to find the error message describing the problem; then see "Problems" below for what to try next.

If refinement is successful, predicted reflections on the display should match actual spots better than before refinement. Changes in unit cell and detector parameters should be relatively small.

The missetting angles sometimes look very large even when refinement is successful, due to the fact that dps_refine works with a set of primitive vectors which do not always correspond to the standard ones defining the cell. When these vectors are used to generate a final orientation matrix, the program sometimes comes up with a matrix related by a symmetry transformation to the original one. This annoying "feature" will be fixed in a future release. As long as predicted spots match real ones better than before refinement, the refinement has succeeded.

Check new predictions for other frames by changing the "Frame:" field in the display window. If predictions match real spots worse than before, or parameter changes are unreasonable, there is a problem - see below for suggestions. If values look reasonable, clicking "Save Everything" will save them all (any parameters that were not being refined will be unaffected). This is usually the right thing to do, but if necessary you can save subsets of values by clicking the appropriate buttons: "Save Unit Cell", "Save Misset Angles", or "Save Beam/Distance".

Note that updating cell and/or angles alters the orientation matrix file - you cannot get back to the original file except by indexing again.

If parameter changes are much different from zero (except for the missetting angles, as mentioned above), you should probably do another round of refinement. Save everything, and hit "START REFINEMENT" again. You can repeat the "Save" - "START" sequence as often as needed, until parameter changes are negligible. If you used only one frame for the initial refinement, or were keeping some parameters fixed, you may want to add more images or refine more parameters for later cycles. If refinement has converged to a reasonable looking set of parameters, that predict spot positions well, you are ready to proceed to integration.

Problems

If refinement does not complete normally, look at the log file to determine the problem: scroll back in the message window, or use an editor to view the file. The log file name is constructed from the image prefix, run number (if any), and first and last sequence numbers: test_1_1_10_dps_refine.log, for example. The most common problem is not enough usable reflections.

"Usable reflections" are those above a minimum I/sigma(I) value. You can increase their number by including more frames, and/or by changing the resolution range and/or mosaicity (set the latter two in the main GUI window and click "SAVE"). If a continuous sequence of frames is available, you may have better luck with mosflm or postchk. Or, try very careful indexing followed directly by integration; it may be necessary to split up the frames into groups and index each group separately. DPS scaling can then be used to be sure all groups have the same indexing, and to refine crystal parameters.

If refinement completes but parameter changes are large:

  1. If the crystal-detector distance or direct beam position changes by more than a few tenths of a mm., go back and reindex with the correct values; then try refinement again.
  2. Missetting angle changes greater than about 0.5 degrees (but not large enough to indicate selection of a symmetry-related orientation) are possible if the initial indexing was not very good, but if the spot predictions before refinement looked good and angle changes are large, something is probably wrong. You can try saving the refined cell (and camera constants if they changed substantially) and clicking "START REFINEMENT" again - two or more passes may be necessary for convergence of parameters.
  3. Be sure you are starting with a reasonable estimate of mosaicity, i.e. the number of predicted spots is similar to the number of real spots.
  4. Consider using a different set of images for refinement. Accurate refinement of all cell parameters may not be possible with certain sets of images, particularly if symmetry is low. Also, if the initial indexing is not very good predictions will be poor for frames far away from the one which was used for indexing - try using just the frames used for indexing, or a subset of them, first. If refinement works on these, other frames may be added for subsequent cycles.
  5. If changes are suspiciously large but the above fixes don't help, try going on to integration anyway, perhaps with post-refinement turned off. You may simply have marginal data which can be processed but refine poorly.
  6. If you're still not happy with the results, try using one of the other refinement programs.


Refine (mosflm)

Mosflm refinement uses the "POSTREF SEGMENT" option, which works by comparing calculated and observed intensities for partial reflections. Camera constants (beam position, distance, and detector rotations) are refined for each individual frame, after which information from all specified frames is used to refine unit cell, crystal orientation, and mosaicity.

The process requires one or two sets of 2-5 consecutive images. Two sets separated by 10-30 degrees in phi are usually suitable. Designate the sets in the "Frames to use in refinement" field, e.g. "1-3 28-30". Initially, the program will set the field to specify the first 3 and last 3 frames in the range from "Start Image No." to "End Image No.". If the total range is large, it is better to use a second segment only about 20 degrees away from the first, or else mosflm may be unable to locate reflections in the second segment from the initial orientation matrix. If the initial indexing is not very good, you may need to use only a single segment close to the frame(s) used for indexing, e.g. "1-5".

With good images and low to moderate mosaicity, mosflm can do a good job of refining all parameters, and all items in the "Refine:" list should be selected. If mosaicity is high, i.e. greater than about 1.5 times the oscillation width, it will not refine well and should be held fixed. If you know the unit cell or direct beam position and distance very well, you may want to keep them fixed. It may also be necessary to keep the cell fixed if only a small range of spindle angle is covered by the frames, particularly if symmetry is low. Unless you know that mosaicity is too high to be refined, a good approach is to first try refining everything. If results are poor, then try rerunning with some parameters fixed.

Note - for Fuji imaging plate (or other offline scanner) data - you must turn on "Edit Cmd File Before Execution" and supply direct beam positions for images to be used in refinement. See command file section for details.

Click "START REFINEMENT" to start up the actual refinement process. If you want to take a closer look at the log file during refinement, you can turn off window updating using the "Pause" button in the corner of the message window; then use the scroll bar to move around in the file. To resume updating, use the "Resume" button.

When refinement is complete, the log file window will show "**** END OF PROCESSING ****" for normal termination, followed by the new matrix file. A display of the new predictions for the first image will be presented, and a summary of the results will appear in the "Results:" panel of the "Refinement Control" window.

If a new matrix file does not appear, something went wrong. Scroll back the text in the window to find the "*** FATAL ERROR ***" message describing the problem; scroll back farther to see details of the refinement process. Then see "Problems" below for what to try next.

If refinement is successful, predicted reflections on the display should match actual spots better than before refinement, and changes in crystal and detector parameters should be relatively small. Check new predictions for other frames by changing the "Frame:" field in the display window. If predictions match real spots worse than before, or parameter changes are unreasonable, there is a problem - see below for suggestions. If values look reasonable, clicking "Save Everything" will save them all (any parameters that were not being refined will be unaffected). This is usually the right thing to do, but if necessary you can save subsets of values by clicking the appropriate buttons: "Save Unit Cell", "Save Mosaicity", "Save Misset Angles", "Save Beam/Distance", or "Save Detector Rotns".

Note that updating cell and/or angles alters the mosflm matrix file - you cannot get back to the original file except by indexing again.

If parameter changes are much different from zero, you should probably do another pass of refinement. Save everything, or at least the new cell and beam position/distance (if mosaicity has gone negative, don't save that!), and hit "START REFINEMENT" again. You can repeat the "Save" - "START" sequence as often as needed, until parameter changes are negligible. If you used only one set of frames for the initial refinement, or were keeping some parameters fixed, you may want to add more images or refine more parameters for later cycles. If refinement has converged to a reasonable looking set of parameters, that predict spot positions well, you are ready to proceed to integration.

Problems

If refinement does not complete normally (no new matrix file printed in the message window), look at the log file to determine the problem: scroll back in the message window, or use an editor or xloggraph to view the file. The log file name is constructed from the image prefix, run number (if any), and first and last sequence numbers: test_1_1_10_mosflm_refine.log, for example. The most common problems are (1) not enough usable reflections and (2) too big a shift in some parameter.

"Usable reflections" are medium-to-strong partials extending across two frames. You can increase their number by adjusting the starting mosaicity (set it in the main GUI window and click "SAVE"), by changing the resolution range, or by changing the rejection parameters for refinement. For a highly mosaic or weakly diffracting crystal, it may not be possible to find sufficient usable reflections for mosflm refinement to work. Try dps_refine or postchk. Or, try very careful indexing followed directly by integration; it may be necessary to split up the frames into groups and index each group separately. DPS scaling can then be used to be sure all groups have the same indexing, and to refine crystal parameters.

Overlarge shifts could be due either to a bad initial indexing or to use of unsuitable images.

  1. Check whether the distance or direct beam position changed substantially on the initial frame-by-frame refinement of camera constants. If so, try reindexing with the corrected values.
  2. If the shift in some parameter is just slightly too big, try increasing the "maxresid" parameter in the post-refinement section of the "Integration Parameters" window. Use the "SAVE" button of the main GUI to save the new value to the parameter file, and try refinement again.
  3. Try keeping the mosaicity fixed (toggle off the "Mosaicity" entry in the "Refine:" section of the Refinement Control window) and see if refinement is more stable.
  4. Try using dps_refine or postchk refinement instead.
If refinement completes but parameter changes are large:
  1. If the crystal-detector distance or direct beam position changes by more than a few tenths of a mm., go back and reindex with the correct values; then try refinement again.
  2. If the mosaicity goes very small or negative but distance and direct beam are OK, try starting with a larger value for mosaicity. If that doesn't help, try refining with mosaicity fixed.
  3. Missetting angle changes greater than about 0.5 degrees are possible if the initial indexing was not very good, but if the spot predictions before refinement looked good and angle changes are large, something is probably wrong. You can try saving the refined cell (and camera constants if they changed substantially) and clicking "START REFINEMENT" again - two or more passes may be necessary for convergence of parameters.
  4. Consider using a different set of images for refinement. Mosflm has difficulty refining all cell parameters if one of the unit cell axes is close to the x-ray beam direction; avoid using frames for which this is the case. Also, if the initial indexing is not very good predictions will be poor for frames far away from the one which was used for indexing - try using just one set of frames, close to the initial one, at first. If refinement is successful on these, the range may be broadened for subsequent cycles.
  5. If changes are suspiciously large but the above fixes don't help, try going on to integration anyway, perhaps with post-refinement turned off. You may simply have marginal data which can be processed but refine poorly.
  6. If you're still not happy with the results, try using one of the other refinement programs.


Refine (postchk)

To perform Postchk refinement, first integrate 10-20 frames with no post-refinement: in "Integration Parameters", disable post-refinement. Set the starting and ending image numbers to include 10-20 frames. Then click "Integrate:MOSFLM". A running display of the integration's progress will appear. When integration is complete, click "Postchk" to bring up the Refinement Control window (or click "Refine" and then choose program "POSTCHK"). Turn on "Unit Cell", "Mosaicity", and "Xtal Orientation" in the "Refine:" list. Select the frames to use (the ones you just integrated), and click "START REFINEMENT" to start the actual refinement.

Normal output in Refinement Control window:
             ---Cell---
            Old:      New:
     a :   55.874    56.210
     b :   62.026    62.410
     c :  129.745   129.770
 alpha :   90.000    90.000
  beta :   90.000    90.000
 gamma :   90.000    90.000

     ---Missetting Angles---
            Old:      New:
Phi(X) :    0.000    -0.305
Phi(Y) :    0.000     0.000
Phi(Z) :    0.000     0.012

            ---Mosaicity---
            Old:      New:
            0.100     0.120
Normal output in main message window:
                                                   .
                                                   .
                                                   .
                                               DISTRIBUTION OF 
                                       CALCULATED-OBSERVED FRACTIONS
  PACKS   PHIS    PHIE   NREF    PHIX     PHIY     PHIZ   ETA      A      B      C   rmsr     -5   -2   2    5
   1- 10   1.00    9.00   1081  -0.145    0.001   -0.038  0.04  55.86  62.39 130.37   0.04  461  174  539  227  107
Postchk finished.

Refined values for unit cell, missetting angles, and beam mosaicity are shown. For a successful refinement, changes in cell and mosaicity should be relatively small. Missetting angles determined by postchk are inaccurate, particularly if the images used cover a wide range of spindle angles, and shifts of several tenths of a degree do not necessarily indicate a problem. If results look reasonable, save the cell parameters ("Save Unit Cell"). Save mosaicity ("Save Mosaicity"), or (preferably) update the value in the main window to be slightly larger than the refined value.

Problems

If postchk crashes (no "Postchk finished" in the main GUI message window), look first at the log file from the integration run. The name of this file is constructed from the image prefix, run number (if any), and first and last sequence numbers: test_1_1_10_mosflm.log, for example.
  1. If there are large changes in camera constants in the frame-by-frame refinement, go back and reindex with the correct values.
  2. If early frames look OK but things get progressively worse during the integration, try integrating with post-refinement of orientation only turned on.
  3. If refinement of camera constants is using only a few reflections, try adjusting the spot-rejection parameters - see "Integration" section.
If integration proceeds smoothly but postchk does not, try using dps_refine or mosflm refinement, or skip refinement entirely.

If postchk completes, but changes in cell parameters or mosaicity are unreasonable, check for problems with integration, as above. If none are found, try saving the new cell and integrating the same set of 10-20 frames, with post-refinement of orientation only, or no post-refinement. If this succeeds, go ahead and process the full set. If not, try another refinement program, or integration with no refinement.


Integration

Integrate: MOSFLM

Integration is carried out with mosflm. The principal integration parameters are set using the GUI. Other parameters normally default to reasonable values; they can be changed in unusual cases, as described below. Important parameters include:

  1. Image files to be processed. Set with the "Image Prefix", "Run No.", "Start Image No.", and "End Image No." fields on the main GUI window. Discontinuous sequences of files must be integrated separately.

  2. Starting phi value for the first frame to be integrated, in the "Machine Parameters" window. This must be consistent with the phi values used during indexing! When you change the "Start Image No." field (in the main window), be sure to change "Starting Phi" in "Machine Parameters" to correspond.

  3. Resolution limits. Set with the "Resolution" fields of main GUI.

  4. "Camera constants" - beam position, crystal-detector distance, etc. Taken from the parameter file, where they should be correct after indexing and preliminary refinement. Current values are displayed in the "Machine Parameters" window.

  5. Crystal parameters. Symmetry is taken from the parameter file; it was set at the indexing stage, and is displayed in the "Cell Parameters" window. Unit cell and crystal orientation information are incorporated in the matrix file ("test_1.mat", for example) generated during indexing and updated during preliminary refinement; cell parameters are also kept in the parameter file and displayed in the window. If you change unit cell or missetting angles in "Cell Parameters", they will override the matrix file values - there is generally no reason to do this.

  6. Beam divergence and mosaicity. Taken from the parameter file; displayed in "Machine Parameters" and main window, respectively. Usually the divergence parameters are left at the default values and the mosaicity is set a little higher than the refined value from mosflm or postchk refinement, or the value from visual estimation.

  7. Spot separation and profile fitting parameters. These are displayed in "Integration Parameters". Since mosflm optimizes the box used for profile-fitting, it is normally not necessary to change most of these parameters. The exception is "Spot sep X,Y", which is the minimum distance between spot centers, in mm., for a pair of spots to be considered non-overlapping. Different minima may be specified in X and Y to allow for spots elongated along one dimension. The default values are 8 pixels, i.e. 0.65 mm. for the Quantum-4, in each direction, but these may need to be changed for particularly large or small spots. To check whether overlaps may be a problem, bring up an image display using "(Re)Display image" and turn on predictions. Predicted overlaps are shown in red. Use "Integrate:MOSFLM" to bring up the "Integration Parameters" window, adjust spot separation values, click "SAVE" on the main GUI, click "(Re)Display image" to see new predictions. Modify spot separation values until predicted overlaps agree with true overlaps.

    If you need to change profile fitting parameters, use the mosflm on-line help facility (type "help" at the MOSFLM => prompt in the Xmosflm window that you get by clicking "Index:MOSFLM") for an explanation of their significance.

  8. Detector parameters - gain, overload value, number of regions for calculating profiles. These can be changed in the "Integration Parameters" window. Default values for gain and overloads should be used except in rare cases. "No. of profile regions" must be a square; it is normally 16 for a 4x4 grid, but can be reduced to 9, i.e. a 3x3 grid, if spots are sparse or increased to 25 if spots are very dense and spot shape varies greatly with position on the detector. "Add Partials", in the same area of the "Integration Parameters" window, refers to adding partial reflections across frames before scaling, and is only appropriate when the x-ray dose per frame is constant (i.e. data were not collected at a synchrotron source such as CHESS) and the direct beam position is fixed (i.e. you are not using Fuji plates).

  9. "Post-refinement" parameters. Set these in the "Integration Parameters" window. If post-refinement is enabled, crystal orientation will be refined for each frame, using the information from intensities of partials on that frame and the next in the series. Refinement of mosaicity, and of unit cell parameters, may also be done, if requested. Generally, the unit cell is not refined during integration, as it should have been well determined by indexing and preliminary refinement. Orientation should be refined, especially if a data set extending over a large range of spindle angles is being processed.

    Mosaicity may be refined ("Mosaicity Refinement: ISOTROPIC" selected), particularly if the real crystal mosaicity is anisotropic or varying in time. A reasonably constant mosaicity is also a check on the good behavior of the post-refinement. "Start Mos for Frame: CURRENT" means that the initial divergence for each frame is set to the refined value for the previous frame. Only set this if the refinement is very well behaved and you expect a substantial drift in divergence with time. "Start Mos for Frame: INITIAL" begins the refinement for each frame with the original value; this is usually preferred.

    To refine unit cell parameters, set "Fix Cell?" to "NO". These parameters are refined for a group of images rather than a pair, where the "Width" parameter specifies the number of frames in the group. The "Maxresid" and "Maxshift" parameters tell how much change in the cell parameters is acceptable, and "Repeat" specifies the maximum number of times that refinement of a group will be repeated in an attempt to converge on sensible values.

  10. Output file name and "batch number" offset, set in "Integration Parameters" window. The default output file name is constructed from the image prefix, run number (if any), and first and last image numbers, "test_1_1_10.mtz", for example. If you blank the "Output file name" field and hit C/R, this name will be generated.

    Each "batch number" in the output file is given by the image number plus the offset. The offset is normally 0, but should be set to some positive value if multiple runs with overlapping image numbers are to be merged; batch numbers of all batches to be scaled together must be unique.

  11. Position and size of beam stop shadow, specified in "Machine Parameters". A circular area representing the shadow of the beam stop is omitted from processing. Set its center (approximately the same as the direct beam position) and radius in the appropriate fields, display an image to check.
If you need to change the defaults for any other mosflm parameters, turn on the "Edit Cmd File Before Execution" button on the main GUI.

Click "Integrate:MOSFLM". The "Integration Parameters" window comes up, so you can check the current parameter values. Then click "START INTEGRATION" to proceed. If you are editing the command file, it will come up in a window now. Make changes and exit the editor.

When you start integration, the message window of the main GUI monitors the log file which is being generated (test_1_1_10_mosflm.log, e.g.). A sampling of images, with predictions, is displayed during integration. On the display, fulls are green, partials are yellow, overlaps are red, and reflections rejected by mosflm are magenta. Reasons for rejection include bad profile, too many background pixels rejected, and pixels greater than the overload value. When integration completes normally, the monitor window will show the "**** END OF PROCESSING ****" message from mosflm, followed by output file header information and a summary of the results:

                                                         .
                                                         .
                                                         .
 IMAGE CCX CCY CCOM DIST YSCALE TILT TWIST ROFF TOFF RESID WRESID F P O N B I/sig(I) I/sig(I)_outer Rsym Nsym SDRAT $$ $$
     1  0.01  0.01  0.01 151.4 1.000  -13     2  0.00  0.00 0.021    1.3   1313   919   47   52   0   32.6    7.0   0.036  724   2.7 
     2  0.01  0.01  0.01 151.5 1.000  -17     4  0.00  0.00 0.021    1.0   1322   854   47   53   0   32.8    6.8   0.043  392   2.9 
     3  0.01  0.01  0.01 151.5 1.000  -15     3  0.00  0.00 0.021    1.4   1367   811   38   69   0   33.0    6.8   0.039  254   3.3 
     4  0.01  0.01  0.01 151.5 1.000  -16     3  0.00  0.00 0.020    1.1   1289   936   34   63   0   32.7    7.0   0.043  174   3.4 
     5  0.01  0.01  0.01 151.5 1.000  -18     4  0.00  0.00 0.020    1.2   1339   964   41   68   0   32.6    7.0   0.035  146   3.0 
     6  0.01  0.01  0.00 151.4 1.000  -16     7  0.00  0.00 0.020    1.3   1345   938   43   74   0   31.5    6.6   0.036  104   2.8 
     7  0.01  0.01  0.00 151.4 1.000  -15     6  0.00  0.00 0.023    1.4   1358   965   43   86   0   31.3    5.7   0.044  106   3.1 
     8  0.01  0.01  0.00 151.4 1.000  -14     2  0.00  0.00 0.020    1.3   1394   979   36   79   0   30.5    6.6   0.047   92   3.4 
     9  0.01  0.01 -0.01 151.5 1.000  -15     4  0.00  0.00 0.021    1.3   1410   944   30   91   0   30.0    7.0   0.056   80   2.7 
    10  0.00  0.01 -0.01 151.4 1.000  -20     3  0.00  0.00 0.023    1.3   1462   974   36   79   0   28.3    6.2   0.058   66   2.6 
 $$
  $TABLE: Post refinement:
 $GRAPHS :Missets phix phiy phiz v image:A:1,2,3,4:  
 :Cell parameters A,B,C v image:A:1,5,6,7:  
 :Cell angles alpha beta gamma v image:A:1,8,9,10:  
 :Mosaic spread v image:A:1,11:  
 :Beam divergences v image:A:1,12,13: $$  
 Image      PHIX    PHIY    PHIZ      A      B      C    ALPHA   BETA  GAMMA  MOSAIC  DIVH  DIVV  Resid   NR $$ $$
    1      0.00    0.03    0.06     56.29  62.50 130.16  90.00  90.00  90.00    0.14  0.10  0.02  0.010  315                         
    2      0.00    0.03    0.05     56.29  62.50 130.16  90.00  90.00  90.00    0.13  0.10  0.02  0.007  269                         
    3      0.00    0.03    0.05     56.29  62.50 130.16  90.00  90.00  90.00    0.12  0.10  0.02  0.009  278                         
    4      0.00    0.02    0.05     56.29  62.50 130.16  90.00  90.00  90.00    0.13  0.10  0.02  0.009  359                         
    5      0.00    0.01    0.05     56.29  62.50 130.16  90.00  90.00  90.00    0.13  0.10  0.02  0.010  288                         
    6      0.00    0.01    0.05     56.29  62.50 130.16  90.00  90.00  90.00    0.12  0.10  0.02  0.010  312                         
    7      0.00    0.00    0.05     56.29  62.50 130.16  90.00  90.00  90.00    0.12  0.10  0.02  0.008  315                         
    8      0.00    0.00    0.05     56.29  62.50 130.16  90.00  90.00  90.00    0.12  0.10  0.02  0.009  295                         
    9      0.00   -0.01    0.05     56.29  62.50 130.16  90.00  90.00  90.00    0.13  0.10  0.02  0.010  289                         
 $$
 Mosflm finished.
 Please examine the information printed to the
 screen from the summary file; also examine the log
 file:  test_1_1_10_mosflm.log
Sorting results
Sortmtz finished.

The summary shows the refined values of the camera constants for each frame and also the results of post-refinement (if enabled). Check for reasonable and consistent values. For more detailed information, examine the log file, by scrolling back in the window or using an editor or xloggraph.

Problems

Failures can occur during:


Scaling and merging

The output from integration is an MTZ file containing observations from each of the images in the data set. Sortmtz is automatically run after integration to sort the observations by reduced Miller indices. The data must then be scaled and merged together, using "Scaling:DPS" or "Scaling:SCALA".
Scaling:DPS

The DPS scaling program is a new program written by Robert Bolotovsky at Purdue. Important features include:

More information is available from the
DPS Web site.

Click on "Scaling:DPS" to bring up the "Scaling Parameters" window. Important parameters include:

  1. Preparation of sorted and merged input. Usually required, but may be bypassed (turn on "Bypass initial sort/merge") if you are rescaling a data set, for instance with a different resolution range or different scaling options.

  2. Starting values for refinable parameters. The "default" setting starts all scale factors at 1.0 and relative B's at 0.0. To use results from a previous scaling run, select one of the "output from previous" options. To rerun the previous run (with different settings) when that run was not using default input, select "input to last run".

  3. Resolution range. Usually set to the full range used for integration, but may be decreased.

  4. Mode of refinement. The default is to refine scale factors separately for each frame ("Scale refinement: by individual frame") but to smooth B-factors by fitting the values to a polynomial function ("B-factor refinement: polynomial fit"). If refinement is unstable, you can turn off B-factor refinement entirely, or smooth scales as well as B's. If smoothing scales, the frames should be grouped into "runs" (see below) such that there is no large discontinuity in scale factor within a run.

  5. Maximum number of refinement cycles. The default value is 10; it may need to be increased if parameters vary greatly for different parts of the data set. This limit is usually applied to both scaling and post-refinement steps; see below for how to change the post-refinement limit separately.

  6. Refinement of mosaicity (during the scaling pass). Turn on or off with the "Refine mos" toggle button. Mosaicity can only be refined if summed partials are not being used (see below).

  7. Post-refinement of crystal parameters. Turn on or off with the "Post-refine" toggle button. On by default, but may not be necessary if a successful refinement was carried out before integration. The post-refinement step can take quite a long time.

  8. Summed partials. If "Use summed partials" is selected for scaling and/or merging, full reflections are constructed by adding partials from adjacent frames, rather than scaling up individual partial observations, for the corresponding step(s). Summed partials are less sensitive to errors in crystal parameters, while use of scaled partials avoids (1) problems due to adding partials across a discontinuity in scale factors and (2) loss of data due to missing parts of reflections. Scaling up partials also provides greater redundancy of observations, although some noise is introduced by scaling up weak signals.

  9. Anomalous data. The default is to combine anomalous mates, but if there is an anomalous signal present they should be kept separate. At present, anomalous mates are combined or kept separate throughout the scaling and merging process.

  10. Output format, and name of output file. The default format is "normal", i.e. a list of h, k, l, I, and sig(I) for each merged reflection. For anomalous data, I and sig(I) are replaced by I+, sig(I+), I-, and sig(I-). Other options include "complete" to include unmeasured reflections in the specified resolution range as zeroes, "unmerged" to list each observation separately, and "none" to omit output entirely. The output file is a formatted ASCII file.

  11. "Runs". Normally, all frames from a single input file are processed together, as a single "run". However, multiple runs are needed if:

    Each line in the "Definitions of runs" table defines one run. You may specify the input file either by its name or by image prefix, run number, start number, and end number. In the latter case, the file name will be constructed from the other items.

    If the "Start no." and "End no" fields are blank, all frames in the file will be used. If not, the specified range of batch numbers (frame number plus offset) will be used. The same input file may be specified multiple times with different ranges of batch number; these ranges must not overlap. Ranges from different files may overlap without confusion, as the program will assign a unique internal sequence number to each frame.

A number of other parameters, which usually do not need to be changed, are shown in the "Additional Scaling Parameters" window. These include:
  1. Rejection list. A list of individual observations to be rejected can be read in, and the program can output a list of reflections which should probably be rejected. Normally, no reject list is read or written; fill in a file name in the appropriate place if you want to use this feature.

  2. Parameters for scaling and post-refinement steps:

  3. Parameters to be post-refined. Default is to refine all crystal rotations, mosaicity, and unit cell. If only a small range of spindle angle is covered, it may be better not to refine the Z rotation. Some poorer quality data sets may require holding other parameters fixed for a stable refinement.

  4. Constraints on post-refined parameters. These can be varied independently for each frame or constrained to be the same for the entire set or for subgroups within the set. For grouped constraint, specify the groups by ranges of sequence number, e.g. "1-5 6-20 21-30". These are the internal sequence numbers, which start at 1 for the first batch number in the first file and go up to n, where n is the total number of files being scaled.

  5. Output statistics. Statistics given in the output log file can range from minimal to voluminous. The default "brief" setting is suitable for most cases.

  6. Reindexing. Normally off. The "auto" setting is useful if a set of frames recorded from a single crystal has been split up into groups and indexed separately. In this case, the various indexing solutions will be related by symmetry but need not be identical - automatic reindexing attempts to place all reflections on a common lattice. Groups of frames for reindexing are normally specified by the entries in the "Runs" window; when different grouping is necessary, bring up the "Still More Parameters" window and enter ranges of sequence numbers in the "Auto-reindexing groups" field.

    "Explicit" reindexing allows you to specify any transformation of indices you want: in the "Explicit reindexing parameters" window of "Still More Parameters", define groups by filename and batch number range; define transform as new indices, e.g. "-h -k l" for a 2-fold rotation about c*; enter "Y" or "N" under "Mat?" for whether or not to update the orientation matrix as well as the indices.

  7. Fixed parameters for specified frames. In "Still More Parameters". For any parameters to be held fixed for certain frame(s), turn on the switch and enter frame numbers (groups such as "1-5" are allowed) in the adjacent field. It is not usually necessary to fix parameters, but may sometimes improve convergence.

  8. Post-refinement parameters different from scaling. To set different resolution limits and/or number of cycles for post-refinement and scaling, use the fields in "Still More Parameters".
To change other scaling parameters, activate "Edit Cmd File Before Execution". Click "START SCALING" to start the DPS scaling program; if "Edit Cmd File" is on, there will be a chance to edit the command file first.

Output from DPS scaling goes to a log file (test_1_dps_scale.log, for example), which is monitored during the run. Xloggraph can be used to make various useful plots of the results.

Problems

Scaling can have various problems, including:

  1. Insufficient symmetry-related reflections to determine scale factors. If you are using summed partials, try switching to calculated partials. If more frames are available, include them. Additional restraints on scales and B's may help.
  2. Initial parameters too far off from true values. If you are using calculated partials, try switching to summed partials.
  3. High R-factors. It is normal for R's using calculated partials to be higher than those using summed partials, because of the noise introduced in scaling up weak measurements, and simply because of having more observations per unique reflection. You can adjust the "minimum fraction" parameter, in the "Additional Scaling Parameters" window, to reject more of the weak reflections. You may want to use calculated partials in scaling but summed partials in merging.


Scaling:SCALA

Scala is a versatile scaling program from the CCP4 suite. Click on "Scaling:SCALA" to bring up the "Scaling Parameters" window.

Important parameters include:

  1. Preparation of sorted and merged input. Usually required, but may be bypassed (turn on "Bypass initial sort/merge") if you are rescaling a data set, for instance with a different resolution range or different scaling options.

  2. Starting values for refinable parameters. This function is not yet working; leave it at "default".

  3. Resolution range. Usually set to the full range used for integration, but may be decreased. Note that changing the integration range (in the main GUI window) does not automatically update the range for scaling.

  4. Mode of refinement. The default is to refine scale factors separately for each frame ("Scale refinement: by individual frame") but to smooth B-factors, calculating B values at 5-degree intervals and interpolating between them ("B-factor refinement: by phi, Phi interval 5.0"). If refinement is unstable, you can use a larger interval for B, turn off B-factor refinement entirely, or refine scales "by phi". If smoothing scales, the frames should be grouped into "runs" (see below) such that there is no large discontinuity in scale factor within a run (this is a good idea in general, but becomes even more important when smoothing).

  5. Maximum number of refinement cycles. The default value is 10; increase it if scaling seems to be working, but slowly, so that convergence is not attained in 10 cycles. With good data, convergence typically takes 3 - 4 cycles, but larger numbers are possible if there is a large variation in scale factors.

  6. Partials. Normally included in both scaling and merging steps, but if mosaicity is small relative to oscillation range and redundancy is high, you may get somewhat better data by excluding partials from one or both stages.

  7. Anomalous data. The default is to combine anomalous mates for both scaling and output. If there is an anomalous signal present, keep anomalous mates separate for output. If the signal is large and data are complete, you can keep them separate for scaling as well.

  8. TAILS correction. Turn this on if your data are badly affected by diffuse scattering. See scala documentation for an explanation of this parameter.

  9. Output format, and name of output file. Normal format is "averaged". Other options are possible if you want to keep all observations separate ("unmerged"), you plan to do a later scaling run using the output from this run as input ("separate"), or you are going to run the postref post-refinement program ("postref"). Postref is an unsupported CCP4 program which has been mostly superseded by the post-refinement functions of mosflm; see CCP4 documentation for more information. The output file is a (binary) MTZ file; for the "unmerged" case a formatted ASCII file is also generated, in a form suitable for input to programs such as solve.

  10. "Runs". Normally, all frames from a single input file are processed together, as a single "run". However, multiple runs are needed if:

    Each line in the "Definitions of runs" table defines one run. You may specify the input file either by its name or by image prefix, run number, start number, and end number. In the latter case, the file name will be constructed from the other items, so the integration run that produced the file must have used the default output file name, and a batch number offset of 0.

    If the "Start no." and "End no" fields are blank, all frames in the file will be used. If not, the specified range of batch numbers (frame number plus offset) will be used. The same input file may be specified multiple times with different ranges of batch number; these ranges may not overlap. Be sure all batch numbers to be scaled together are unique.

  11. Criteria for rejection. Several parameters, shown in the "Additional Scaling Parameters" window, may be adjusted. See scala documentation for more details.
To change other scaling parameters, activate "Edit Cmd File Before Execution". Click "START SCALING" to start scala; if "Edit Cmd File" is on, there will be a chance to edit the command file first.

Output from scala goes to a log file (test_1_scala.log, for example), which is monitored during the run. When the run is complete, it is advisable to examine the log file using a browser such as netscape or xloggraph, a program from CCP4 that allows for generating plots from tabular information in the file, as well as displaying the file itself.

Full documentation is available as part of the CCP4 documentation set.

Problems

Scaling can fail for various reasons, including:

  1. Insufficient symmetry-related reflections to determine scale factors. Check the chart (in the scala log file) of the number of reflections in common between pairs of images. Include more frames, if available. Check whether observations are being rejected, and alter rejection parameters if appropriate. If the number of observations is marginal, it may stabilize the refinement to keep B factors fixed, and/or to restrain scale factors (but be sure to group frames so that you are not restraining scale factors over a jump in scales, such as from the end of a fill at CHESS to the beginning of the next).
  2. All partial reflections. This situation can cause scale factors to refine to alternately high and low values, easily seen as a sawtooth pattern in the plot of K vs. batch. Try restraining the scale factors. Or use DPS scaling with scaled-up rather than summed partials.
  3. Poor integration - if images look good and you are sure of the space group, and you get reasonable scales and B factors but high R(sym), go back and carefully check the log file from integration. Check whether predictions actually match well with spots.
  4. Or maybe your crystal is crummy - twinned, for example. Have you carefully checked a selection of your images, with predictions?


Final output

I to F: TRUNCATE (CCP4)

DPS scaling produces an ASCII file of scaled and merged intensities, which can go directly into a phasing program.

When scala is used, its output is an MTZ file containing scaled and merged intensities. To produce final files of F's, in both binary (MTZ) and ASCII formats, click on the "I to F:TRUNCATE(CCP4)" button. The programs used are truncate and mtz2various, from CCP4. There is normally no need to change any parameters from the defaults, but if necessary it can be done by editing the command file.


Command files

Description of command files
Format of command files
Editing files
Refinement parameters
Integration parameters
Scaling parameters
Truncate parameters


Description of command files

The data processing GUI executes the "working" code for indexing, refinement, etc. through the use of shell scripts ("command files"). The master file called "Mosflm_control.com" selects an action based on a keyword passed to it by the GUI and calls one or more other scripts to perform the action. The name of the parameter file maintained by the GUI is passed as an argument to all the scripts, so that information in the file is available to the working routines.

The control file may be edited to change the flow of operations resulting from activation of a button on the GUI. This is seldom necessary. Individual command files, however, may be edited routinely to change processing parameters which have no representation in the GUI windows.

Return to top of section


Format of command files

The first part of Mosflm_control.com looks like this:

#!/bin/csh -f 
#
set PARAM_FILE = $1
set OPTION = $2
#
set CWD = `get_gdb $PARAM_FILE Working_Dir`                 ^ 
set EDIT_SCRIPT = `get_gdb $PARAM_FILE Edit_Script`         | <= Part 1
set IMAGE_PREFIX = `get_gdb $PARAM_FILE Image_Prefix`       V
echo "Running option: $OPTION"
if("$EDIT_SCRIPT" == "1") then                              ^
  set user_editor = `printenv GUI_EDITOR`                   |
  if ("$user_editor" == "") then                            | <= Part 2
    set user_editor = "vi"                                  |
  endif                                                     |
endif                                                       V
#
cd $CWD
goto $OPTION
#
Indexing:                                                   ^
#                                                           |
echo "Running autoindexing interactively with Mosflm"       |
if ( "$EDIT_SCRIPT" == "1") then                            |
  set editor_result = `xterm -geom 80,24 -title MosflmIndex.com -e $user_editor MosflmIndex.com`
endif                                                       | <= Part 3
                                                            |
csh -v MosflmIndex.com $PARAM_FILE ${DISPLAY}               |
exit                                                        V
This code includes (1) extraction of some values from the parameter file, (2) selection of an editor based on the GUI_EDITOR environment variable, and (3) execution of the (mosflm) Indexing option. The rest of the command file includes sections for the other options.


Here is part of the command file for DPS indexing:

#
set PARAM_FILE = $1                                            ^
#                                                              |
set run_no =     `get_gdb $PARAM_FILE Run_Number`              |
set name =       `get_gdb $PARAM_FILE Image_Prefix`            |
set StartImNo =  `get_gdb $PARAM_FILE Start_Image_Number`      |
set EndImNo =    `get_gdb $PARAM_FILE Last_Image_Number`       |
set file_dir =   `get_gdb $PARAM_FILE Image_Dir`               |
set resolution = `get_gdb $PARAM_FILE Resolution`              | <= Part 1
set beam =       `get_gdb $PARAM_FILE Beam_X Beam_Y`           |
set wavelength = `get_gdb $PARAM_FILE Wavelength`              |
set distance =   `get_gdb $PARAM_FILE Distance`                |
set twotheta =   `get_gdb $PARAM_FILE Twotheta`                |
set deltaphi =   `get_gdb $PARAM_FILE Delta_Phi`               |
set StartAngle = `get_gdb $PARAM_FILE Start_Angle`             |
set lattice_name = `get_gdb $PARAM_FILE Lattice_Name`          |
set cell_max = `get_gdb $PARAM_FILE DPS_Longvec`               |
set low_res = `get_gdb $PARAM_FILE DPS_Lowres`                 |
set high_res = `get_gdb $PARAM_FILE DPS_Highres`               |
set spindle_dir = `get_gdb $PARAM_FILE Spindle_dir`            |
set spindle_vert = `get_gdb $PARAM_FILE Spindle_vert`          |
set crystal_system = `get_gdb $PARAM_FILE Crystal_System_Name` |
set detector = `get_gdb $PARAM_FILE Detector_Type`             |
set suffix = `get_gdb $PARAM_FILE File_Suffix`                 |
set use_target = `get_gdb $PARAM_FILE DPS_Use_Target`          |
set target_cell = `get_gdb $PARAM_FILE DPS_Target_Cell`        |
                                                               |
if ("${run_no}" != "") then                                    |
  set long_name = "${name}_${run_no}"                          |
else                                                           |
  set long_name = "${name}"                                    |
endif                                                          V
                            .
                            .
                            .
#
set st_3dig =    `make_3digit $StartImNo`                     ^
#                                                             |
#  Check first image.                                         |
#                                                             |
set first_img = ${file_dir}/${long_name}_${st_3dig}.img       | <= Part 2
if(-e ${first_img}) then                                      |
  goto first_img_ok                                           |
endif                                                         V
#
first_img_ok:
#                           .                      Part 3
                            .                        |
                            .                        V
switch ("$detector")
  case ADSC_Quantum:
  case R-axis:
  case MAR:
  case MAR_IP:
  case MAR_CCD:
    set rsizef_head = `mh_st rastersize1 ${detector} < ${first_img}`
    set rsizes_head = `mh_st rastersize2 ${detector} < ${first_img}`
    set imsizef_head = `mh_st imsize1 ${detector} < ${first_img}`
    set imsizes_head = `mh_st imsize2 ${detector} < ${first_img}`
    if (${rsizes_head} != '0.0000') then
      if (${rsizes_head} != ${rastersizes}) then
        echo "**** Warning - rastersize ${rastersizes} in parameter file disagre
es with value ${rsizes_head} in file header."
        echo "     File header value is used."
        set rastersizes = ${rsizes_head}
      endif
    endif
                            .
                            .
                            .
    if ($wavelength == '0.0000') then
      set wavelength = \
       `mh_st wavelength ${detector} < ${first_img}`
      echo "Wavelength set to ${wavelength}, from file header."
      set status = `put_gdb $PARAM_FILE Wavelength ${wavelength}`
    else
      set wave_head = \
       `mh_st wavelength ${detector} < ${first_img}`
      if (${wave_head} != ${wavelength}) then
        echo "**** Warning - wavelength ${wavelength} in parameter file disagree
s with value ${wave_head} in file header."
        echo "     Parameter file value is used."
      endif
    endif
                            .
                            .
                            .
set raster_rs = `mos2dps_coord $detector $beam $rastersizef $rastersizes $imsizef $imsizes`
#
set dps_2th = `mulf $twotheta '-1.0'`
                            .
                            .
                            .
/bin/rm control_index                                        ^
                                                             |
if ($use_target == '1') then                                 |
  echo "mode target_cell" > control_index                    |
else                                                         |
  echo "mode automatic" > control_index                      |
endif                                                        |
echo "##parameter_file $PARAM_FILE" >> control_index         |
echo "resolution ${low_res} ${high_res}" >> control_index    |
echo "cell ${cell_max}" >> control_index                     |
echo "wavelength ${wavelength}" >> control_index             |
echo "rastersize ${rastersize}" >> control_index             | <= Part 4
echo "distance ${distance}" >> control_index                 |
echo "twotheta ${dps_2th}" >> control_index                  |
echo "beamstop ${raster_rs}" >> control_index                |
echo "film_rotation ${film_rotation}" >> control_index       |
echo "sscale $y_scale" >> control_index                      |
echo "peaks peaks.file 0.0 ${deltaphi}" >> control_index     |
echo "omfile ${long_name}.om" >> control_index               |
#                                                            |
dps_index < control_index > dps_index.log1                   |
/bin/mv PEAKS_WITH_PHIS NEW_PEAK_FILE                        V

if(-e POST_ERROR) then                                         ^
    echo "dps_index: FAILED.  SKIP FURTHER INDEXING PROGRAMS." |
    exit                                                       |
endif                                                          |
                                                               | <= Part 5
/bin/rm PEAKS_WITH_PHIS                                        |
/bin/rm NEW_PEAK_FILE                                          |
dps_analyseOM ${name}_${run_no}.om analyze.log                 | 
dps_bravais < control_bravais > bravais.log                    V
                            .
                            .
                            .
#
This includes (1) extraction of values from the parameter file, (2) a check for existence of a data file, (3) extraction of values from an image file header (if there is one; choices are made based on detector type coordinates from another file, (4) construction of a control file and execution of the indexing program, and (5) execution of a series of secondary programs.

Return to top of section


Editing individual command files

When the "Edit Cmd Files Before Execution" button on the main GUI window is on, there will be an opportunity to edit each command file before it is executed. An xterm window will appear, with the file open in the editor selected by the GUI_EDITOR environment variable. If GUI_EDITOR is not set, "jot" will be used on SGI machines and "vi" on others. Use the editor to alter the file and save it. When you exit from the editor, the script will be executed. As the updated command file overwrites the original one in this process, any changes made will stay in effect for future runs, unless you use the "Copy Cmd Files" button to reload the standard files.

To change a processing parameter, find the place in the command file where it is being written to a control file and alter the write statement as necessary. You can make other changes in a command file, as long as the result is a valid shell script, but this will not generally be necessary.

Return to top of section


Refinement parameters

Mosflm command lines for "Refine:MOSFLM" that can be changed only by editing the command file "MosflmRefine.com" include:

"SYNCHROTRON POLAR 0.89"
"DISPER 0.0025"               Polarization and dispersion of the x-ray beam

"LIMITS RMIN 5 RMAX $rmax YMAX $rmax XMAX $rmax xscan $rmax yscan $rmax"
"BACKSTOP RADIUS 5 CENTRE $beam"      Beam stop radius and position, and
                                      low resolution limit

"NULLPIX ..."                   Detector parameter

"REFINEMENT CYCLES 2 FIX YSCALE RESID 15 IMIN 20 5 NSIG 6"
"REFINEMENT INCLUDE PARTIALS 0.5"
"REFINEMENT LIMIT $ref_limit"
                Parameters for refinement of camera constants
The refinement parameters are the ones most likely to need changing. The default values specify:
CYCLES 2 - 2 cycles of refinement; more may be helpful.
FIX YSCALE - do not refine the ratio of Y to X pixel dimensions; for an image plate detector, refinement may be appropriate.
RESID 15 - maximum residual after refinement; this is already pretty generous, but could be changed.
IMIN 20 5 - minimum I/sig(I) for spots in the outer region of the image, and the amount to increase the limit as one goes in towards the center of the image; could be reduced for waek images.
LIMIT ... - radius, in mm., of the inner circle used for preliminary refinement, set to a standard value for each type of detector; could be increased if there are few central spots.
NSIG 6 - minimum variation in pixel values over a spot relative to a standard value, for spots in the central region; could be reduced for weak images.
INCLUDE PARTIALS 0.5 - include partials with a calculated recorded fraction of 0.5 or higher in the refinement; could set limit higher or lower depending on number of partials present.

Excluding an area

One or more regions of an image (for example, those shadowed by some equipment in front of the detector) may be excluded by inserting a line of the form:

LIMITS EXCLUDE 10.0 25.0 40.0 50.0

where the two pairs of numbers specify the upper left and lower right corners of a rectangular area to be excluded from processing. The coordinates are in mm. and are relative to the scan origin. Multiple areas may be specified, up to a maximum of 10.

Offline scanners

When an offline scanner, such as the Fuji BAS2000, is used, the direct beam position varies from frame to frame. For reliable operation of mosflm in this case, the beam positions must be specified for each frame. Determine the positions by using the mosflm display, or with some other program such as getbeam (available from MacCHESS). Then edit the command file to replace the "PROCESS" line with a series of lines, e.g.:
PROCESS 1 TO 3 ADD 0 START 0.0 ANGLE 1.0

becomes:

BEAM 111.3 100.5
PROCESS 1 TO 1 ADD 0 START 0.0 ANGLE 1.0
BEAM 110.3 102.3
PROCESS 2 TO 2 ADD 0 START 1.0 ANGLE 1.0
BEAM 112.2 101.6
PROCESS 3 TO 3 ADD 0 START 2.0 ANGLE 1.0

Return to top of section


Integration parameters

Mosflm command lines for "Integrate:Mosflm" that can be changed only by editing the command file "MosflmIntegrate.com" include all those listed above for refinement plus those created by the following lines:

set pr_0 = {"POSTREF SDFAC 3 SHIFTFAC 10.0 BEAM "}{$sw_beamtyp}{" USEBEAM FIX ALL"}
set pr_1 = {"POSTREF MAXRESID "}{$pref_resid}{" MAXSHIFT "}{$pref_shift}{" WIDTH "}{$pref_width}{" REPEAT "}{$pref_rep}
                     .
                     .
                     .
set prof_line_0 = "PROFILE OPTIMIZE"
set prof_line_1 = {"PROFILE BOUNDARY "}{$p_bound}{" RATIO "}{$p_ratio}{"TOLERANCE "}{$p_tol}{" RMSBG 10."}
                     .
                     .
                     .    
MATRIX ${name}_$run_no.mat   

$pr_0                         Post-refinement control
$pr_1

$prof_line_0                  Profile parameters
$prof_line_1         

PROCESS $StartImNo to $EndImNo ADD $add_no START $StartAngle ANGLE $deltaphi BLOCK $m_block

Most of the parameters on these lines can be set using the GUI, except for:

SDFAC 3 - minimum I/sig(I) to use reflection in post-refinement; could set lower for weak images or higher for strong images with many spots.
SHIFTFAC 10.0 - maximum shift/sig(shift) for cell parameters. If shift is greater, images are reprocessed and post-refinement repeated. This value is already large and should not need changing.
RMSBG 10. - maximum r.m.s. variation in background, on a scale of 0-255, for acceptable standard profile; could be increased if background is noisy.

Offline scanners

When an offline scanner, such as the Fuji BAS2000, is used, the direct beam position varies from frame to frame. For reliable operation of mosflm in this case, the beam positions must be specified for each frame. The routine make_fuji_list is supplied to construct the proper series of "BEAM" and "PROCESS" lines to do this, but the user must supply an input file containing a list of image file names and direct beam positions, in the form:
file_001.img 110.3 100.7
file_002.img 111.6 99.8
        .
        .
        .
These beam positions are in mosflm coordinates - determine them by using the mosflm display, or with some other program such as getbeam (available from MacCHESS). Name the file "prefix_start_end.beam", or "prefix_run_start_end.beam" if "Run No." is not blank, where prefix is the image prefix, start is the first frame number of the series to be integrated, and end is the last frame number.

Return to top of section


Scaling parameters

Some parameters used for merging, and for producing statistics, for DPS scaling can only be changed by editing the command file.

The only command line supplied to scala which can not be changed from the GUI is:

SDCORRECTION  1.25  0.03
This gives correction factors for sigmas; other values may be more appropriate for some data sets.

Some other settable parameters have their default values. For information on them, see the scala documentation.

Return to top of section


Parameters for truncate

At present, parameters for final output can be changed only by editing the command file.

The standard truncate parameters are:

ANOMALOUS YES
TRUNCATE YES
FALLOFF YES
LABOUT F(+)=F(+) SIGF(+)=SIGF(+) F(-)=F(-) SIGF(-)=SIGF(-)

These specify: output anomalous differences (if they are present in input), convert I's to F's using the "truncation" procedure of French and Wilson, check for anisotropy in falloff of intensity with resolution, label output columns F(+), SIGF(+), etc. To merge anomalous pairs, use "ANOMALOUS NO". To use F = SQRT(I), set "TRUNCATE NO". See CCP4 documentation for other options.

Return to top of section


Getting help

To follow what the GUI program is doing in detail, look at the command files DpsIndex.com, MosflmIntegrate.com, etc. Various input and log files are produced when the program is run; examination of these may help.

For help on mosflm, read "mosflm_user_guide.doc", in the /use/ccp4/mosflm directory on MacCHESS Alphas. Also, whenever an Xmosflm window is up, you can access mosflm interactive help by typing "help".

CCP4 documentation is available.

Help on the adxv display program is available using the "Help" button on the menu bar of each of its windows.

See the "Problems" section for descriptions of some miscellaneous problems that may occur when executing processing_gui.


Problems using the GUI

The problems listed here are occasionally encountered when using processing_gui. They appear to be related to the particular X server and operating system used, and are difficult to reproduce and hence to correct. The authors would appreciate hearing about any additional problems of this type that you may experience on your system.

Marian Szebenyi

dms35@cornell.edu

August 22, 2001