Note

This page was generated from a Jupyter notebook. Not all interactive visualization will work on this web page. Consider downloading the notebooks for full Python-backed interactivity.

# holoviews initialization

2. Setting Up

The cells under this section should be executed every time the kernel is restarted.

2.1. load modules

Loads the minian modules, usually this cell should not be modified.

%%capture
%load_ext autoreload
%autoreload 2
import itertools as itt
import os
import sys

import holoviews as hv
import numpy as np
import xarray as xr
from dask.distributed import Client, LocalCluster
from holoviews.operation.datashader import datashade, regrid
from holoviews.util import Dynamic
from IPython.core.display import display

2.2. set path and parameters

Set all of the parameters that control the notebook’s behavior. Ideally, the following cell is the only part of the code the user will have to change when analyzing different datasets. Here we briefly introduce only some of the initial parameters that are necessary to start the pipeline, and leave the discussion of specific parameters for later.

• minian_path is the path that contains the minian folder , where the minian codebase (.py files) reside. The default value "." means “current folder”, which should work in most cases, unless you want to try out another version of minian that is not in the same folder as this notebook.

• dpath is the folder that contains the videos to be processed.

• interactive controls whether interactive plots will be shown for parameters exploration. Interactive plotting requires CPU/memory usage, and thus could require some time (in particular, those steps where video is played). In principle, the user might want to visualize interactive plots during the initial parameters exploration, once the parameters are set and ready for batch processing, the user will set interactive as False to reduce processing time.

• output_size controls the relative size of all the plots on a scale of 0-100 percent, though it can be set to values >100 without any problem.

• param_save_minian specifies the destination folder and format of the saved data. dpath is the folder path where the data will be saved. meta_dict is a dictionary that is used to construct meta data for the final labeled data structure. overwrite is a boolean value controlling whether the data is overwritten if a file already exists. We set it to True here so you can easily play with the demo multiple times, but use caution with this option during actual analysis. In addition to erasing prior data that may be important to you, overwritting data may cause compatibility issues with existing data from the same minian dataset folder. If you want to re-analyze a video from scratch using different parameters, it is recommended that you delete existing data first.

folder structure

The defult meta_dict in param_save_minian assumes output minian datasets are stored in heirarchiically arranged folders, as shown below:

mice1
│
└───session1
│   │
│   └───minian
│       │   Y.zarr
│       │   A.zarr
│       │   ...
│
└───session2
    │
    └───minian

The default value can be read as follows: The name of the last folder (-1) in dpath (the folder that directly contains the videos) will be used to designate the value of a metadata dimension named "session". The name of the second-to-last folder (-2) in dpath will be used to designate the value for "animal" and so on. Both the keys (name of metadata dimension) and values (numbers indicating which level of folder name should be used) of meta_dict can be modified to represent your preferred way of data storage. Note that the metadata are determined by the folder structure of saved minian datasets, not by those of input movie data.

# Set up Initial Basic Parameters#
minian_path = "."
dpath = "./demo_movies/"
minian_ds_path = os.path.join(dpath, "minian")
intpath = "./minian_intermediate"
subset = dict(frame=slice(0, None))
subset_mc = None
interactive = True
output_size = 100
n_workers = int(os.getenv("MINIAN_NWORKERS", 4))
param_save_minian = {
    "dpath": minian_ds_path,
    "meta_dict": dict(session=-1, animal=-2),
    "overwrite": True,
}

# Pre-processing Parameters#
param_load_videos = {
    "pattern": "msCam[0-9]+\.avi$",
    "dtype": np.uint8,
    "downsample": dict(frame=1, height=1, width=1),
    "downsample_strategy": "subset",
}
param_denoise = {"method": "median", "ksize": 7}
param_background_removal = {"method": "tophat", "wnd": 15}

# Motion Correction Parameters#
subset_mc = None
param_estimate_motion = {"dim": "frame"}

# Initialization Parameters#
param_seeds_init = {
    "wnd_size": 1000,
    "method": "rolling",
    "stp_size": 500,
    "max_wnd": 15,
    "diff_thres": 3,
}
param_pnr_refine = {"noise_freq": 0.06, "thres": 1}
param_ks_refine = {"sig": 0.05}
param_seeds_merge = {"thres_dist": 10, "thres_corr": 0.8, "noise_freq": 0.06}
param_initialize = {"thres_corr": 0.8, "wnd": 10, "noise_freq": 0.06}
param_init_merge = {"thres_corr": 0.8}

# CNMF Parameters#
param_get_noise = {"noise_range": (0.06, 0.5)}
param_first_spatial = {
    "dl_wnd": 5,
    "sparse_penal": 0.01,
    "update_background": True,
    "size_thres": (25, None),
}
param_first_temporal = {
    "noise_freq": 0.06,
    "sparse_penal": 1,
    "p": 1,
    "add_lag": 20,
    "jac_thres": 0.2,
}
param_first_merge = {"thres_corr": 0.8}
param_second_spatial = {
    "dl_wnd": 5,
    "sparse_penal": 0.01,
    "update_background": True,
    "size_thres": (25, None),
}
param_second_temporal = {
    "noise_freq": 0.06,
    "sparse_penal": 1,
    "p": 1,
    "add_lag": 20,
    "jac_thres": 0.4,
}

os.environ["OMP_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MINIAN_INTERMEDIATE"] = intpath

2.3. import minian

The following cell loads minian and usually should not be modified. If you encounter an ImportError, check that you followed the installation instructions and that minian_path is pointing to the right place.

%%capture
sys.path.append(minian_path)
from minian.cnmf import (
    compute_AtC,
    compute_trace,
    get_noise_fft,
    smooth_sig,
    unit_merge,
    update_spatial,
    update_temporal,
)
from minian.initialization import (
    gmm_refine,
    initA,
    initbf,
    initC,
    intensity_refine,
    ks_refine,
    pnr_refine,
    seeds_init,
    seeds_merge,
)
from minian.motion_correction import apply_transform, estimate_motion
from minian.preprocessing import denoise, remove_background
from minian.utilities import (
    TaskAnnotation,
    get_optimal_chk,
    load_videos,
    open_minian,
    save_minian,
)
from minian.visualization import (
    CNMFViewer,
    VArrayViewer,
    generate_videos,
    visualize_gmm_fit,
    visualize_motion,
    visualize_preprocess,
    visualize_seeds,
    visualize_spatial_update,
    visualize_temporal_update,
    write_video,
)

2.4. module initialization

The following cell handles initialization of modules and parameters necessary for minian to be run and usually should not be modified.

dpath = os.path.abspath(dpath)
hv.notebook_extension("bokeh", width=100)

2.5. start cluster

cluster = LocalCluster(
    n_workers=n_workers,
    memory_limit="2GB",
    resources={"MEM": 1},
    threads_per_worker=2,
    dashboard_address=":8787",
)
annt_plugin = TaskAnnotation()
cluster.scheduler.add_plugin(annt_plugin)
client = Client(cluster)

1. Preface 3. Pre-processing