GSoC 2026: Week 5 & 6 - Stress Testing and Parallelization

Based on meeting discussions, it was suggested to add periodic boundary box info and energies to the Universe as well. This was done and charges have also been added on a “if exists” basis.

These weeks have been spent stress testing the integration as well as manually checking some of the MDAnalysis tools on the output Universe. Some of the tools tested were RMSD, RMSF, AlignTraj, PCA, DistanceMatrix, DensityAnalysis and LinearDensity. All of these tools successfully ran on the generated Universe, provided the given WESTPA simulation output had the required parameters.

Coming to parallelization, MDAnalysis uses its AnalysisBase backend which uses the split-apply-combine scheme to efficiently split groups to workers, compute observables, and combine all the objects from all the workers which then gets merged together giving a simular result to serial analysis.
This poses a major challenge for HDF5 files. HDF5 file handles are bound to the operating system and cannot be serialized (pickled) to be sent across different worker processes. If we try to pass an open file to a worker, Python will crash.

This is solved by:

Stripping File Handles

When MDAnalysis decides to split the workload, it uses Python’s pickle library to package up your reader and send it to the workers. Right before this happens, it call __getstate__ which intercepts this process, explicitly closes all open HDF5 file handles (self._current_h5 and self._h5), and sets their variables to None. This strips away the “unpicklable” OS-level pointers, allowing the pure data (like the frame index) to transmit safely.

def __getstate__(self):
        if self._current_h5 is not None:
            self._current_h5.close()
        if self._h5 is not None:
            self._h5.close()

        state = self.__dict__.copy()
        state['_h5'] = None
        state['_current_h5'] = None
        state['_current_path'] = None
        return state

Rebuilding the Worker’s State

When the worker process receives the package and wakes up, it calls __setstate__. The reader uses this moment to cleanly reconstruct itself in the new memory space. It re-opens the main west.h5 file restores all the attributes that sometimes gets wiped out during multiprocessing.

def __setstate__(self, state):
        self.__dict__.update(state)
        self._h5 = h5py.File(self.filename, 'r')
        self._current_h5 = None
        self._current_path = None
        if hasattr(self, 'ts') and hasattr(self, '_reader_dt'):
            self.ts.dt = self._reader_dt

With this, we can use backend=multiprocessing and give ‘n’ number of workers to perform tasks in parallel, which may run certain analysis much faster than running in serial. Checking this on InterRDF:

With this in place, all goals for the midterm deadline are complete. The next task is to allow users to save back analysis data back into their westpa file as well as start work on the CLI tool which wraps the entire parser and reader and allows users to effortlessly use this tool.

Summary

• Stress tested the reader to improve on speed.
• Manually checked some MDAnalysis tools on the generated Universe.
• Added periodic box info, energies and charges on a “if exists” basis.
• Added support for running analysis using MDAnalysis’s parallelization backends.
• Added some more testcases for this.