Processing serial synchrotron crystallography (SSX) datasets with xia2

xia2 is able to processes serial synchrotron data through a SSX pipeline built upon data analysis programs from DIALS.

To process serial data from images to a merged MTZ file, the minimal recommended example command is:

xia2.ssx template=../lyso_1_###.cbf space_group=P43212 \
  unit_cell=79.1,79.1,38.2,90,90,90  reference../lyso.mtz

i.e. the minimal required information is the location of the images, the expected space group and unit cell. A suitable reference file, containing a reference set of intensities or pdb model (.mtz, .cif, .pdb) is also recommended to enable data reduction using a reference. Typically, a reference geometry and mask should also be provided, as described below.

Table of contents of xia2 SSX processing documentation

Data Integration

To integrate data, the images must be provided, plus the space group to use for processing and an estimate of the unit cell. If the space group or unit cell are not known, the pipeline can be initially run without these; spotfinding and indexing will be performed, and the most populous unit cell clusters will be presented to the user.

The location of the image data must be specified by using one of the options image=, template= or directory= (multiple instances of one option is allowed). Here image corresponds to an image file (e.g. .cbf or .h5), template is a file template for a sequence of images (e.g. for files lyso_1_001.cbf to lyso_1_900.cbf, the matching template is lyso_1_###.cbf), and directory is a path to a directory containing image files. For data in h5 format, the option image=/path/to/image_master.h5 is recommended. For cbf data, the file template option is recommended, e.g.:

xia2.ssx template=../lyso_1_###.cbf space_group=P43212 \

The overall sequence of the data integration part of the pipeline is as follows. In the absence of an explicitly provided reference geometry, spotfinding and indexing will be run on batches of 1000 images at a time, until a threshold number of crystals have been indexed (250 by default, as specified by the geometry_refinement.n_crystals parameter). Then joint refinement of the detector geometry is run to determine an improved geometry model. Following this, the refined geometry is used to perform spotfinding, indexing and integration of all images, with feedback provided on batches of 1000 images at a time. This batch size can be adjusted with the batch_size parameter. The number of available processes will be determined automatically (optionally a value for nproc can be given), and parallel processing will be performed within the batch.

A DIALS reference geometry file (refined.expt) can be provided as input with the option reference_geometry=, which will be used instead of performing a joint refinement on the data. A mask file created from the DIALS image viewer can also be provided with the option mask=, which will be used in spotfinding and integration. A few other common options to set are the max_lattices parameter, which determines the number of lattices to search for on each image in indexing, and d_min which controls the resolution limit for spotfinding and integration.

To see the full list of options and their descriptions, run xia2.ssx -ce2 -a1. Change the number after -ce to a value from 0 to 3 to see different “expert levels” of program parameters. Note that a phil options file can be provided for each of the DIALS programs, to allow further customisation of the options for the individual programs. Additionally, stepwise processing can be performed by running the program multiple times with the option steps=find_spots, then steps=index and finally steps=integrate.

Data Reduction

Following data integration, data reduction (reindexing, scaling and merging) will be performed. The data reduction can be run separately to the full pipeline through the command xia2.ssx_reduce, taking integrated data as input, e.g.:

xia2.ssx_reduce ../xia2_ssx/batch_*/integrated*.{expt,refl}

To run only the data integration without reduction, use the option steps=find_spots+index+integrate (i.e. omit +reduce) when running xia2.ssx.

The data reduction process consists of unit cell filtering, followed by indexing ambiguity resolution in batches (if ambiguities are possible due to lattice and space group symmetries), followed by scaling and merging.

If a reference dataset/PDB model is provided with the option reference=, then reindexing and scaling is performed in parallel in batches of at least reduction_batch_size crystals, using intensities extracted/generated from the reference as a reference when reindexing and scaling. It is recommended to use a high-quality reference set of intensities in preference to genering a set of intensities from a PDB model, to give a higher accuracy. If generating intensities from a PDB model, the default bulk solvent parameters (k_sol and b_sol) should be adjusted to suitable values. If there is no reference given, the scaling is not performed in parallel. Other important options are setting anomalous=True/False and specifying a d_min value. To evaluate the success of indexing ambiguity resolution, it is important to inspect the html output from dials.cosym jobs in the data_reduction\reindex folder. To see the full list of data reduction parameters and their descriptions, run xia2.ssx_reduce -ce3 -a2. The output of the data reduction pipeline is a merged MTZ file which can be taken onwards for structure determination.

xia2.ssx_reduce also supports quick remerging of already processed data by using the option steps=merge. This can be useful for aggregating multiple processing jobs processed using the same reference model, or generating MTZ files with specified resolution cutoffs e.g.:

xia2.ssx_reduce ../xia2_ssx_reduce/DataFiles/scaled*.{expt,refl} steps=merge d_min=1.8

xia2.ssx_reduce steps=merge ../{chip1,chip2}/DataFiles/scaled*.{expt,refl}

Overcoming difficulties in geometry refinement

The first step of processing with xia2.ssx is to improve the detector geometry in order to improve the indexing rate. The starting assumption is that the initial geometry is good enough to successfully index a fraction of the images. When this is not the case, it will be necessary to change some of the program parameters. Examples and suggestions of how to tackle trickier processing cases are described in the link below.

Merging in groups

xia2.ssx_reduce also supports merging in groups to handle more complex experiments such as dose series experiments. This is described in more detail in the links below.