brimfile

  1"""
  2## What is brimfile?
  3
  4*brimfile* is a Python library to read from and write to brim (**Br**illouin **im**aging) files,
  5which contain both the spectra and analysed data for Brillouin imaging.
  6More information about the brim file format can be found [here](https://github.com/brillouin-imaging/Brillouin-standard-file).
  7
  8Briefly, a brim file can contain multiple data groups,
  9typically corresponding to imaging of the same sample at different timepoints/conditions.
 10Each data group contains the spectral data as well as the metadata and
 11the results of the analysis on the spectral data (which can be many in case multiple reconstruction pipelines are performed).
 12
 13The structure of the *brimfile* library reflects the structure of the brim file and the user can access the data,
 14metadata and analysis results through their corresponding classes.
 15
 16- [File](#file): represents a brim file, which can be opened or created.
 17- [Data](#data): represents a data group in the brim file, which contains the spectral data and metadata.
 18- [Metadata](#metadata): represents the metadata associated to a data group (or to the whole file).
 19- [AnalysisResults](#analysisresults): represents the results of the analysis of the spectral data.
 20
 21
 22## Install brimfile
 23
 24We recommend installing *brimfile* in a [virtual environment](https://docs.python.org/3/library/venv.html).
 25
 26After activating the new environment, simply run:
 27
 28```bash
 29pip install brimfile
 30```
 31
 32If you also need the support for exporting the analyzed data to OME-TIFF files,
 33you can install the optional dependencies with:
 34
 35```bash
 36pip install "brimfile[export-tiff]"
 37```
 38
 39For accessing remote data (i.e. S3 buckets), you need `remote-store`:
 40
 41```bash
 42pip install "brimfile[remote-store]"
 43```
 44
 45## Quickstart
 46
 47The following code shows how to:
 48- open a .brim file 
 49- get an image for the Brillouin shift 
 50- get the spectrum at a specific pixel
 51- get the metadata.
 52
 53```Python
 54from brimfile import File, Data, Metadata, AnalysisResults
 55Quantity = AnalysisResults.Quantity
 56PeakType = AnalysisResults.PeakType
 57
 58filename = 'path/to/your/file.brim.zarr' 
 59f = File(filename)
 60
 61# get the first data group in the file
 62d = f.get_data()
 63
 64# get the first analysis results in the data group
 65ar = d.get_analysis_results()
 66
 67# get the image for the shift
 68img, px_size = ar.get_image(Quantity.Shift, PeakType.average)
 69
 70# get the spectrum at the pixel (pz,py,px)
 71(pz,py,px) = (0,0,0)
 72PSD, frequency, PSD_units, frequency_units = d.get_spectrum_in_image((pz,py,px))
 73
 74# get the metadata 
 75md = d.get_metadata()
 76all_metadata = md.all_to_dict()
 77
 78# close the file
 79f.close()
 80```
 81
 82## Store types
 83
 84Currently brimfile supports zip, zarr and S3 buckets as a store.
 85When opening or creating a file, the storage be selected by using the brimfile.file_abstraction.StoreType enum; zip and zarr can be used both for reading and writing while S3 only for reading. 
 86
 87Although it is possible to write directly to zip, this will create duplicated entries in the archive (see [GitHub issue](https://github.com/zarr-developers/zarr-python/issues/1695)).
 88
 89A possible workaround is to create a .zarr store instead and zip the folder afterwards.
 90Importantly the root of the archive should not contain the folder itself, i.e. you should go inside the .zarr folder, select all the elements there, right click on them to create a .zip archive.
 91
 92
 93## Use brimfile
 94
 95### File
 96
 97The main class is `brimfile.file.File`, which represents a brim file.
 98It can be used to create a new brim file (`brimfile.file.File.create`) or to open an existing one (`brimfile.file.File.__init__`).
 99
100```Python
101import brimfile as brim
102
103filename = 'path/to/your/file.brim.zarr'
104
105# Open an existing brim file
106f = brim.File(filename)
107
108# or create a new one
109f = brim.File.create(filename)
110```
111
112### Data
113
114You can then get a `brimfile.data.Data` object representing the data group in the brim file
115by opening an existing one (`brimfile.file.File.get_data`).
116
117```Python
118# Get the first data group in the file
119data = f.get_data()
120```
121
122To add a new data group to the file, you can use the `brimfile.file.File.create_data_group` method,
123which accepts a 4D array for the PSD with dimensions (z, y, x, spectrum),
124a frequency array which might have the same size as PSD or be 1D, in case the frequency axis is the same for all the spectra.
125```Python
126# or create a new one
127data = f.create_data_group(PSD, freq_GHz, (dz, dy, dx), name='my_data_group')
128```
129Alternatively you can use `brimfile.file.File.create_data_group_sparse` for sparse data, which lets you directly assign the correspondence
130between the spatial positions and the spectra through the `scanning` dictionary.
131
132Once you have an istance of `brimfile.data.Data`, you can get the spectrum corresponding to a pixel in the image
133by calling the `brimfile.data.Data.get_spectrum_in_image` method:
134```Python
135PSD, frequency, PSD_units, frequency_units = data.get_spectrum_in_image((pz,py,px))    
136```
137
138### Metadata
139
140You can then get a `brimfile.metadata.main.Metadata` object by simply calling the `brimfile.data.Data.get_metadata` method on a previously retrieved `Data` object.
141The returned Metadata object contains all the metadata associated with the file and the specific data group.
142```Python
143metadata = data.get_metadata()
144```
145The list of available metadata is defined [here](https://github.com/brillouin-imaging/Brillouin-standard-file/blob/main/docs/brim_file_metadata.md) and can also be printed in the terminal with the `brimfile.metadata.schema.print_schema` method, which also allows to print the description of each metadata field:
146```Python
147brim.metadata.print_schema(include_description=True)
148```
149For metadata fields which require an enum, it can be imported from `brimfile.metadata`, e.g. `from brimfile.metadata import ImmersionMedium`.
150
151New metadata can be added to the current data group (or to the whole file) by calling the `brimfile.metadata.main.Metadata.add` method.
152```Python
153import datetime
154
155Attr = Metadata.Item
156datetime_now = datetime.now().isoformat()
157temp = Attr(22.0, 'C')
158    
159metadata.add(Metadata.Type.Experiment, {'Datetime':datetime_now, 'Temperature':temp},local=True)
160```
161When adding metadata fields which require an enum, the enum value or the string representation of the enum member (case-insensitive and ignoring underscores and spaces) can be used, e.g. `brim.metadata.SignalType.spontaneous` or 'spontaneous' can be used for the `Signal_type` field.
162
163A single metadata item can be retrieved by indexing the `Metadata` object, which takes a string in the format 'group.object', e.g. 'Experiment.Datetime'.
164```Python
165datetime = metadata['Experiment.Datetime']
166```
167A dictionary containing all metadata can be retrieved by calling the `brimfile.metadata.main.Metadata.all_to_dict` method.
168```Python
169metadata.all_to_dict()
170```
171
172### AnalysisResults
173
174The results of the analysis can be accessed through the `brimfile.analysis_results.AnalysisResults` object, obtained by calling the `brimfile.data.Data.get_analysis_results` method on a previously retrieved `Data` object:
175``` Python
176analysis_results = data.get_analysis_results()
177```
178or create a new one by calling the `brimfile.data.Data.create_analysis_results_group`:
179``` Python
180ar = data.create_analysis_results_group({'shift':shift_GHz, 'shift_units': 'GHz',
181                                        'width': width_GHz, 'width_units': 'GHz'},
182                                        {'shift':shift_GHz, 'shift_units': 'GHz',
183                                        'width': width_GHz, 'width_units': 'GHz'},
184                                        name = 'my_analysis_results')
185```
186
187`AnalysisResults` also exposes a method to retrieve the images of the analysis results (`brimfile.analysis_results.AnalysisResults.get_image`):
188
189``` Python
190ar_cls = AnalysisResults
191img, px_size = analysis_results.get_image(ar_cls.Quantity.Shift, ar_cls.PeakType.average)
192```
193
194## List the contents of a brim file
195
196The *brimfile* library provides methods to list the contents of a brim file.
197
198To list all the data groups in a brim file, you can use the `brimfile.file.File.list_data_groups` method.
199
200Once you have a `Data` object, you can list the analysis results in it by calling the `brimfile.data.Data.list_AnalysisResults` method.
201
202Once you have an `AnalysisResults` object, you can determine:
203- if the Stokes and/or anti-Stokes peaks are present by calling the `brimfile.analysis_results.AnalysisResults.list_existing_peak_types` method;
204- the available quantities (e.g. shift, linewidth, etc...) in the analysis results by calling the `brimfile.analysis_results.AnalysisResults.list_existing_quantities` method.
205
206## Example code
207
208Here is a simple example which creates a brim file with a data group and some metadata and then reads it back.
209
210We first write a function to generate some dummy data:
211
212``` Python
213import numpy as np
214
215def generate_data():
216    def lorentzian(x, x0, w):
217        return 1/(1+((x-x0)/(w/2))**2)
218    Nx, Ny, Nz = (7, 5, 3) # Number of points in x,y,z
219    dx, dy, dz = (0.4, 0.5, 2) # Stepsizes (in um)
220    n_points = Nx*Ny*Nz  # total number of points
221
222    width_GHz = 0.4
223    width_GHz_arr = np.full((Nz, Ny, Nx), width_GHz)
224    shift_GHz_arr = np.empty((Nz, Ny, Nx))
225    freq_GHz = np.linspace(6, 9, 151)  # 151 frequency points
226    PSD = np.empty((Nz, Ny, Nx, len(freq_GHz)))
227    for i in range(Nz):
228        for j in range(Ny):
229            for k in range(Nx):
230                index = k + Nx*j + Ny*Nx*i
231                #let's increase the shift linearly to have a readout 
232                shift_GHz = freq_GHz[0] + (freq_GHz[-1]-freq_GHz[0]) * index/n_points
233                spectrum = lorentzian(freq_GHz, shift_GHz, width_GHz)
234                shift_GHz_arr[i,j,k] = shift_GHz 
235                PSD[i, j, k,:] = spectrum
236
237    return PSD, freq_GHz, (dz,dy,dx), shift_GHz_arr, width_GHz_arr
238```
239
240Now we can use this function to create a brim file with a data group and some metadata:
241
242``` Python
243    import brimfile as brim
244    from brimfile import File, Data, Metadata, StoreType
245    from datetime import datetime
246
247    filename = 'path/to/your/file.brim.zarr' 
248
249    f = File.create(filename, store_type=StoreType.AUTO)
250
251    PSD, freq_GHz, (dz,dy,dx), shift_GHz, width_GHz = generate_data()
252    
253    d0 = f.create_data_group(PSD, freq_GHz, (dz,dy,dx), name='test1')
254    
255    # Create the metadata
256    Attr = Metadata.Item
257    datetime_now = datetime.now().isoformat()
258    temp = Attr(22.0, 'C')
259    md = d0.get_metadata()
260    
261    md.add(Metadata.Type.Experiment, {'Datetime':datetime_now, 'Temperature':temp})
262    md.add(Metadata.Type.Optics, {'Wavelength':Attr(660, 'nm')})
263    # enums can be added using the enum value or the string representation of the enum member (case-insensitive and ignoring underscores and spaces)
264    md.add(Metadata.Type.Brillouin, {'Signal_type': brim.metadata.SignalType.spontaneous, 
265                                          'Phonons_measured': 'longitudinal',})
266    # Add some metadata to the local data group   
267    temp = Attr(37.0, 'C')
268    md.add(Metadata.Type.Experiment, {'Temperature':temp}, local=True)
269
270    # create the analysis results
271    ar = d0.create_analysis_results_group({'shift':shift_GHz, 'shift_units': 'GHz',
272                                             'width': width_GHz, 'width_units': 'GHz'},
273                                             {'shift':shift_GHz, 'shift_units': 'GHz',
274                                             'width': width_GHz, 'width_units': 'GHz'},
275                                             name = 'test1_analysis')
276    f.close()
277```
278and we can read it back:
279``` Python
280    from brimfile import File, Data, Metadata, AnalysisResults
281
282    filename = 'path/to/your/file.brim.zarr' 
283
284    f = File(filename)
285
286    # check if the file is read only
287    f.is_read_only()
288
289    #list all the data groups in the file
290    data_groups = f.list_data_groups(retrieve_custom_name=True)
291
292    # get the first data group in the file
293    d = f.get_data()
294    # get the name of the data group
295    d.get_name()
296
297    # get the number of parameters which the spectra depend on
298    n_pars = d.get_num_parameters()
299
300    # get the metadata 
301    md = d.get_metadata()
302    all_metadata = md.all_to_dict()
303    # the list of metadata is defined here https://github.com/brillouin-imaging/Brillouin-standard-file/blob/main/docs/brim_file_metadata.md
304    time = md['Experiment.Datetime']
305    time.value
306    time.units
307    temp = md['Experiment.Temperature']
308    md_dict = md.to_dict(Metadata.Type.Experiment)
309
310
311    #get the list of analysis results in the data group
312    ar_list = d.list_AnalysisResults(retrieve_custom_name=True)
313    # get the first analysis results in the data group
314    ar = d.get_analysis_results()
315    # get the name of the analysis results
316    ar.get_name()
317    # list the existing peak types and quantities in the analysis results
318    pt = ar.list_existing_peak_types()
319    qt = ar.list_existing_quantities()
320    # get the image of the shift quantity for the average of the Stokes and anti-Stokes peaks
321    img, px_size = ar.get_image(AnalysisResults.Quantity.Shift, AnalysisResults.PeakType.average)
322    # get the units of the shift quantity
323    u = ar.get_units(AnalysisResults.Quantity.Shift)
324
325    # get a quantity at a specific pixel (coord) in the image
326    coord = (1,3,4)
327    qt_at_px = ar.get_quantity_at_pixel(coord, AnalysisResults.Quantity.Shift, AnalysisResults.PeakType.average)
328    assert img[coord]==qt_at_px
329    
330    # get the spectrum in the image at a specific pixel (coord)
331    PSD, frequency, PSD_units, frequency_units = d.get_spectrum_in_image(coord)    
332
333    f.close()
334```
335
336## Export the data to a different format
337
338### OME-TIFF
339
340You can export a specific quantity in the analyzed data to OME-TIFF files using the `brimfile.analysis_results.AnalysisResults.save_image_to_OMETiff` method on an instance `ar` of the `AnalysisResults` class.
341``` Python
342ar_cls = AnalysisResults
343ar.save_image_to_OMETiff(ar_cls.Quantity.Shift, ar_cls.PeakType.average, filename='path/to/your/exported_tiff' )
344```
345"""
346
347try:
348    from ._version import __version__
349except ImportError:
350    __version__ = "unknown"
351
352import os
353# Default: normal imports enabled.
354# Override with env var: BRIMFILE_IMPORT_VALIDATION_ONLY=1
355_IMPORT_VALIDATION_ONLY = os.getenv("BRIMFILE_IMPORT_VALIDATION_ONLY", "").lower() in {
356    "1", "true", "yes"
357}
358
359if not _IMPORT_VALIDATION_ONLY:
360    from .file import File
361    from .data import Data
362    from .analysis_results import AnalysisResults
363    from .metadata import Metadata
364    from .file_abstraction import StoreType