brimfile

  1"""
  2## What is brimfile?
  3
  4*brimfile* is a Python library to read from and write to brim (**Br**illouin **im**aging) files,
  5which contain both the spectra and analysed data for Brillouin imaging.
  6More information about the brim file format can be found [here](https://github.com/brillouin-imaging/Brillouin-standard-file).
  7
  8Briefly, a brim file can contain multiple data groups,
  9typically corresponding to imaging of the same sample at different timepoints/conditions.
 10Each data group contains the spectral data as well as the metadata and
 11the results of the analysis on the spectral data (which can be many in case multiple reconstruction pipelines are performed).
 12
 13The structure of the *brimfile* library reflects the structure of the brim file and the user can access the data,
 14metadata and analysis results through their corresponding classes.
 15
 16- [File](#file): represents a brim file, which can be opened or created.
 17- [Data](#data): represents a data group in the brim file, which contains the spectral data and metadata.
 18- [Metadata](#metadata): represents the metadata associated to a data group (or to the whole file).
 19- [AnalysisResults](#analysisresults): represents the results of the analysis of the spectral data.
 20- [Calibration](#calibration): represents the calibration data associated to a data group, which contains the calibration spectra and relevant metadata.
 21
 22
 23## Install brimfile
 24
 25We recommend installing *brimfile* in a [virtual environment](https://docs.python.org/3/library/venv.html).
 26
 27After activating the new environment, simply run:
 28
 29```bash
 30pip install brimfile
 31```
 32
 33If you also need the support for exporting the analyzed data to OME-TIFF files,
 34you can install the optional dependencies with:
 35
 36```bash
 37pip install "brimfile[export-tiff]"
 38```
 39
 40For accessing remote data (i.e. S3 buckets), you need `remote-store`:
 41
 42```bash
 43pip install "brimfile[remote-store]"
 44```
 45
 46## Quickstart
 47
 48The following code shows how to:
 49- open a .brim file 
 50- get an image for the Brillouin shift 
 51- get the spectrum at a specific pixel
 52- get the metadata.
 53
 54```Python
 55from brimfile import File, Data, Metadata, AnalysisResults
 56Quantity = AnalysisResults.Quantity
 57PeakType = AnalysisResults.PeakType
 58
 59filename = 'path/to/your/file.brim.zarr' 
 60f = File(filename)
 61
 62# get the first data group in the file
 63d = f.get_data()
 64
 65# get the first analysis results in the data group
 66ar = d.get_analysis_results()
 67
 68# get the image for the shift
 69img, px_size = ar.get_image(Quantity.Shift, PeakType.average)
 70
 71# get the spectrum at the pixel (pz,py,px)
 72(pz,py,px) = (0,0,0)
 73PSD, frequency, PSD_units, frequency_units = d.get_spectrum_in_image((pz,py,px))
 74
 75# get the metadata 
 76md = d.get_metadata()
 77all_metadata = md.all_to_dict()
 78
 79# close the file
 80f.close()
 81```
 82
 83## Store types
 84
 85Currently brimfile supports zip, zarr and S3 buckets as a store.
 86When opening or creating a file, the storage be selected by using the brimfile.file_abstraction.StoreType enum; zip and zarr can be used both for reading and writing while S3 only for reading. 
 87
 88Although it is possible to write directly to zip, this will create duplicated entries in the archive (see [GitHub issue](https://github.com/zarr-developers/zarr-python/issues/1695)).
 89
 90A possible workaround is to create a .zarr store instead and zip the folder afterwards.
 91Importantly the root of the archive should not contain the folder itself, i.e. you should go inside the .zarr folder, select all the elements there, right click on them to create a .zip archive.
 92
 93
 94## Use brimfile
 95
 96### File
 97
 98The main class is `brimfile.file.File`, which represents a brim file.
 99It can be used to create a new brim file (`brimfile.file.File.create`) or to open an existing one (`brimfile.file.File.__init__`).
100
101```Python
102import brimfile as brim
103
104filename = 'path/to/your/file.brim.zarr'
105
106# Open an existing brim file
107f = brim.File(filename)
108
109# or create a new one
110f = brim.File.create(filename)
111```
112
113### Data
114
115You can then get a `brimfile.data.Data` object representing the data group in the brim file
116by opening an existing one (`brimfile.file.File.get_data`).
117
118```Python
119# Get the first data group in the file
120data = f.get_data()
121```
122
123To add a new data group to the file, you can use the `brimfile.file.File.create_data_group` method,
124which accepts a 4D array for the PSD with dimensions (z, y, x, spectrum),
125a frequency array which might have the same size as PSD or be 1D, in case the frequency axis is the same for all the spectra.
126```Python
127# or create a new one
128data = f.create_data_group(PSD, freq_GHz, (dz, dy, dx), name='my_data_group')
129```
130Alternatively you can use `brimfile.file.File.create_data_group_sparse` for sparse data, which lets you directly assign the correspondence
131between the spatial positions and the spectra through the `scanning` dictionary.
132
133Once you have an istance of `brimfile.data.Data`, you can get the spectrum corresponding to a pixel in the image
134by calling the `brimfile.data.Data.get_spectrum_in_image` method:
135```Python
136PSD, frequency, PSD_units, frequency_units = data.get_spectrum_in_image((pz,py,px))    
137```
138
139### Metadata
140
141You can then get a `brimfile.metadata.main.Metadata` object by simply calling the `brimfile.data.Data.get_metadata` method on a previously retrieved `Data` object.
142The returned Metadata object contains all the metadata associated with the file and the specific data group.
143```Python
144metadata = data.get_metadata()
145```
146The list of available metadata is defined [here](https://github.com/brillouin-imaging/Brillouin-standard-file/blob/main/docs/brim_file_metadata.md) and can also be printed in the terminal with the `brimfile.metadata.schema.print_schema` method, which also allows to print the description of each metadata field:
147```Python
148brim.metadata.print_schema(include_description=True)
149```
150For metadata fields which require an enum, it can be imported from `brimfile.metadata`, e.g. `from brimfile.metadata import ImmersionMedium`.
151
152New metadata can be added to the current data group (or to the whole file) by calling the `brimfile.metadata.main.Metadata.add` method.
153```Python
154import datetime
155
156Attr = Metadata.Item
157datetime_now = datetime.now().isoformat()
158temp = Attr(22.0, 'C')
159    
160metadata.add(Metadata.Type.Experiment, {'Datetime':datetime_now, 'Temperature':temp},local=True)
161```
162When adding metadata fields which require an enum, the enum value or the string representation of the enum member (case-insensitive and ignoring underscores and spaces) can be used, e.g. `brim.metadata.SignalType.spontaneous` or 'spontaneous' can be used for the `Signal_type` field.
163
164A single metadata item can be retrieved by indexing the `Metadata` object, which takes a string in the format 'group.object', e.g. 'Experiment.Datetime'.
165```Python
166datetime = metadata['Experiment.Datetime']
167```
168A dictionary containing all metadata can be retrieved by calling the `brimfile.metadata.main.Metadata.all_to_dict` method.
169```Python
170metadata.all_to_dict()
171```
172
173### AnalysisResults
174
175The results of the analysis can be accessed through the `brimfile.analysis_results.AnalysisResults` object, obtained by calling the `brimfile.data.Data.get_analysis_results` method on a previously retrieved `Data` object:
176``` Python
177analysis_results = data.get_analysis_results()
178```
179or create a new one by calling the `brimfile.data.Data.create_analysis_results_group`:
180``` Python
181ar = data.create_analysis_results_group({'shift':shift_GHz, 'shift_units': 'GHz',
182                                        'width': width_GHz, 'width_units': 'GHz'},
183                                        {'shift':shift_GHz, 'shift_units': 'GHz',
184                                        'width': width_GHz, 'width_units': 'GHz'},
185                                        name = 'my_analysis_results')
186```
187
188`AnalysisResults` also exposes a method to retrieve the images of the analysis results (`brimfile.analysis_results.AnalysisResults.get_image`):
189
190``` Python
191ar_cls = AnalysisResults
192img, px_size = analysis_results.get_image(ar_cls.Quantity.Shift, ar_cls.PeakType.average)
193```
194
195### Calibration
196
197Calibration data can be accessed through the `brimfile.calibration.Calibration` object, obtained by calling `brimfile.data.Data.get_calibration`.
198You can create a new calibration group with `brimfile.data.Data.create_calibration_group` by providing one or more calibration materials
199(each with `spectra`, `shift`, and optional `shift_units`) and, when needed, an `index` array mapping image coordinates to calibration spectra.
200If another data group should reuse an existing calibration, you can pass `same_as=<data_group_index>` when creating the calibration group.
201
202```Python
203# create calibration for data group d0
204N = np.prod(PSD.shape[:-1])
205index = np.arange(N).reshape(PSD.shape[:-1])
206cal_d = {'spectra': np.empty((N, 50)), 'shift': 7.0, 'shift_units': 'GHz'}
207d0.create_calibration_group(index=index, calibration_data=[cal_d])
208
209# retrieve calibration and get a spectrum at (z, y, x)
210cal = d0.get_calibration()
211spectrum, shift = cal.get_spectrum_at_coor((1, 2, 3))
212
213# optionally reuse calibration from data group 0 in another data group
214d1.create_calibration_group(same_as=0)
215```
216
217## List the contents of a brim file
218
219The *brimfile* library provides methods to list the contents of a brim file.
220
221To list all the data groups in a brim file, you can use the `brimfile.file.File.list_data_groups` method.
222
223Once you have a `Data` object, you can list the analysis results in it by calling the `brimfile.data.Data.list_AnalysisResults` method.
224
225Once you have an `AnalysisResults` object, you can determine:
226- if the Stokes and/or anti-Stokes peaks are present by calling the `brimfile.analysis_results.AnalysisResults.list_existing_peak_types` method;
227- the available quantities (e.g. shift, linewidth, etc...) in the analysis results by calling the `brimfile.analysis_results.AnalysisResults.list_existing_quantities` method.
228
229## Subtypes
230
231For subtype-specific data and utilities (for example SinglePoint_VIPA helpers), see ``brimfile.subtypes``.
232
233## Example code
234
235Here is a simple example which creates a brim file with a data group and some metadata and then reads it back.
236
237We first write a function to generate some dummy data:
238
239``` Python
240import numpy as np
241
242def generate_data():
243    def lorentzian(x, x0, w):
244        return 1/(1+((x-x0)/(w/2))**2)
245    Nx, Ny, Nz = (7, 5, 3) # Number of points in x,y,z
246    dx, dy, dz = (0.4, 0.5, 2) # Stepsizes (in um)
247    n_points = Nx*Ny*Nz  # total number of points
248
249    width_GHz = 0.4
250    width_GHz_arr = np.full((Nz, Ny, Nx), width_GHz)
251    shift_GHz_arr = np.empty((Nz, Ny, Nx))
252    freq_GHz = np.linspace(6, 9, 151)  # 151 frequency points
253    PSD = np.empty((Nz, Ny, Nx, len(freq_GHz)))
254    for i in range(Nz):
255        for j in range(Ny):
256            for k in range(Nx):
257                index = k + Nx*j + Ny*Nx*i
258                #let's increase the shift linearly to have a readout 
259                shift_GHz = freq_GHz[0] + (freq_GHz[-1]-freq_GHz[0]) * index/n_points
260                spectrum = lorentzian(freq_GHz, shift_GHz, width_GHz)
261                shift_GHz_arr[i,j,k] = shift_GHz 
262                PSD[i, j, k,:] = spectrum
263
264    return PSD, freq_GHz, (dz,dy,dx), shift_GHz_arr, width_GHz_arr
265```
266
267Now we can use this function to create a brim file with a data group and some metadata:
268
269``` Python
270    import brimfile as brim
271    from brimfile import File, Data, Metadata, StoreType
272    from datetime import datetime
273
274    filename = 'path/to/your/file.brim.zarr' 
275
276    f = File.create(filename, store_type=StoreType.AUTO)
277
278    PSD, freq_GHz, (dz,dy,dx), shift_GHz, width_GHz = generate_data()
279    
280    d0 = f.create_data_group(PSD, freq_GHz, (dz,dy,dx), name='test1')
281    
282    # Create the metadata
283    Attr = Metadata.Item
284    datetime_now = datetime.now().isoformat()
285    temp = Attr(22.0, 'C')
286    md = d0.get_metadata()
287    
288    md.add(Metadata.Type.Experiment, {'Datetime':datetime_now, 'Temperature':temp})
289    md.add(Metadata.Type.Optics, {'Wavelength':Attr(660, 'nm')})
290    # enums can be added using the enum value or the string representation of the enum member (case-insensitive and ignoring underscores and spaces)
291    md.add(Metadata.Type.Brillouin, {'Signal_type': brim.metadata.SignalType.spontaneous, 
292                                          'Phonons_measured': 'longitudinal',})
293    # Add some metadata to the local data group   
294    temp = Attr(37.0, 'C')
295    md.add(Metadata.Type.Experiment, {'Temperature':temp}, local=True)
296
297    # create the analysis results
298    ar = d0.create_analysis_results_group({'shift':shift_GHz, 'shift_units': 'GHz',
299                                             'width': width_GHz, 'width_units': 'GHz'},
300                                             {'shift':shift_GHz, 'shift_units': 'GHz',
301                                             'width': width_GHz, 'width_units': 'GHz'},
302                                             name = 'test1_analysis')
303    f.close()
304```
305and we can read it back:
306``` Python
307    from brimfile import File, Data, Metadata, AnalysisResults
308
309    filename = 'path/to/your/file.brim.zarr' 
310
311    f = File(filename)
312
313    # check if the file is read only
314    f.is_read_only()
315
316    #list all the data groups in the file
317    data_groups = f.list_data_groups(retrieve_custom_name=True)
318
319    # get the first data group in the file
320    d = f.get_data()
321    # get the name of the data group
322    d.get_name()
323
324    # get the number of parameters which the spectra depend on
325    n_pars = d.get_num_parameters()
326
327    # get the metadata 
328    md = d.get_metadata()
329    all_metadata = md.all_to_dict()
330    # the list of metadata is defined here https://github.com/brillouin-imaging/Brillouin-standard-file/blob/main/docs/brim_file_metadata.md
331    time = md['Experiment.Datetime']
332    time.value
333    time.units
334    temp = md['Experiment.Temperature']
335    md_dict = md.to_dict(Metadata.Type.Experiment)
336
337
338    #get the list of analysis results in the data group
339    ar_list = d.list_AnalysisResults(retrieve_custom_name=True)
340    # get the first analysis results in the data group
341    ar = d.get_analysis_results()
342    # get the name of the analysis results
343    ar.get_name()
344    # list the existing peak types and quantities in the analysis results
345    pt = ar.list_existing_peak_types()
346    qt = ar.list_existing_quantities()
347    # get the image of the shift quantity for the average of the Stokes and anti-Stokes peaks
348    img, px_size = ar.get_image(AnalysisResults.Quantity.Shift, AnalysisResults.PeakType.average)
349    # get the units of the shift quantity
350    u = ar.get_units(AnalysisResults.Quantity.Shift)
351
352    # get a quantity at a specific pixel (coord) in the image
353    coord = (1,3,4)
354    qt_at_px = ar.get_quantity_at_pixel(coord, AnalysisResults.Quantity.Shift, AnalysisResults.PeakType.average)
355    assert img[coord]==qt_at_px
356    
357    # get the spectrum in the image at a specific pixel (coord)
358    PSD, frequency, PSD_units, frequency_units = d.get_spectrum_in_image(coord)    
359
360    f.close()
361```
362
363## Export the data to a different format
364
365### OME-TIFF
366
367You can export a specific quantity in the analyzed data to OME-TIFF files using the `brimfile.analysis_results.AnalysisResults.save_image_to_OMETiff` method on an instance `ar` of the `AnalysisResults` class.
368``` Python
369ar_cls = AnalysisResults
370ar.save_image_to_OMETiff(ar_cls.Quantity.Shift, ar_cls.PeakType.average, filename='path/to/your/exported_tiff' )
371```
372"""
373
374try:
375    from ._version import __version__
376except ImportError:
377    __version__ = "unknown"
378
379import os
380# Default: normal imports enabled.
381# Override with env var: BRIMFILE_IMPORT_VALIDATION_ONLY=1
382_IMPORT_VALIDATION_ONLY = os.getenv("BRIMFILE_IMPORT_VALIDATION_ONLY", "").lower() in {
383    "1", "true", "yes"
384}
385
386if not _IMPORT_VALIDATION_ONLY:
387    from .file import File
388    from .data import Data
389    from .analysis_results import AnalysisResults
390    from .calibration import Calibration
391    from .metadata import Metadata
392    from .file_abstraction import StoreType