Pinpoint

Pinpoint is under active development, there is no requirement to maintain backwards compatibility with version changes.

Organization

Unity

The Trajectory Planner is built out of a single main repository which holds the Unity codebase. Code that is shared across multiple VBL projects is stored in the vbl-core repository which you should add to the main repo as a git submodule. The full pull sequence for the entire repository is:

git clone https://github.com/VirtualBrainLab/Pinpoint
git submodule add https://github.com/dbirman/vbl-core Assets/vbl-core

If you use Github Desktop the vbl-core repository will automatically be pulled. If you are working on multiple VBL projects you should add the repository as a separate repo in Desktop and give it an alias, e.g. Pinpoint-vbl-core.

Additional assets

Some of the asset folders are too large to push to Github. We store these separately in a Google Drive folder. Download these and unzip them into the /Assets/ folder in Pinpoint.

Branch structure

The latest minor release version (e.g. X.Y.Z) is kept on the main branch and released to the /Pinpoint/ folder.

The latest development build is kept on the develop branch and released to the /Pinpoint-develop/ folder.

Incremental features are built on separate branches off of develop and are not released to the public. Outdated builds will be kept available in their corresponding /Pinpoint_X.Y.Z/ folder on the server. Until v1.0.0 we are not making any promises of backward compatibility or maintenance.

Build process

A Windows (win64) and WebGL build configuration are included

To run a new build there are three steps:

  1. Re-build Addressables Storage (if needed) and push this to the server. a) If you modify the storage files you need to update the version number in Pinpoint

  2. Re-build the Addressables build for Pinpoint and push this to the server. a) You can copy the files without over-writing the existing files, to maintain backward compatibility

  3. Build the WebGL build and again push these files to the server in the htdocs/Pinpoint/ folder. We recommend labeling the folder with the version number.

Once you confirm that the new version works you can re-name the folders appropriately. The current stable version is always labeled Pinpoint and older versions have the version number appended, e.g. Pinpoint_0_1_0.

IL2CPP

You may have to install optional visual studios features, specifically the Windows 10 SDK and MSVC, see: https://forum.unity.com/threads/unable-to-build-il2cpp-in-2021-2.1189441/

Assets

Most of the VBL assets are shared across projects, these are accessed from a shared Addressables Storage repository. Shared assets are accessed via the AddressablesRemoteLoader class in vbl-core, see here.

Scenes

When contributing to a VBL project you should avoid modifying the main scenes, except to add or remove Prefab objects. Merges between scenes are very difficult to handle in git and should be avoided.

Core components

Code

The trajectory planner runs out of a central class TrajectoryPlannerManager or tpmanager which coordinates loading the main functionality of the planner.

Managers

Manager classes handle the high-level coordination of functionality in Pinpoint. In general, creating and using Manager classes that share an AssemblyDefinition with TrajectoryPlannerManager should be avoided as much as possible.

Function code

Function classes handle the isolated functionality of specific features in Pinpoint. For example, the code allowing control of the Camera is an isolated functionality. Other examples include the Accounts system, Ephys Link, the Rig code, etc.

Modules

Modules contain code that needs to be re-used across Functions. For example, the ReferenceAtlas/AtlasTransform classes, or the ProbeInsertion class.

Assembly definitions

We have a three level assembly definition hierarchy. The top level is trajectoryplanner which contains *Manager classes. The second level is trajectoryplanner.function where function refers to a specific functionality in Pinpoint. The bottom level is trajectoryplanner.function.module where module is a set of scripts that may be re-used across functions. For example, trajectoryplanner.core.utils holds a set of convenience functions for parsing data files. The folder structure in the Scripts/ folder should reflect the assembly definition hierarchy.

Because of this hierarchy, Managers can see all Functions, but Functions cannot see Managers or other Functions. All code can access modules.

UnityEvent callbacks

There are four ways that Manager and Function classes might need to communicate.

  1. A Manager needs to send data to a Function. Because Manager classes can be linked directly to Functions, you can define a function that takes a variable as input in your Function class.

  2. A Manager needs to request data from a Function. Because Manager classes can be linked directly to Functions, you define a function with a return type in your Function class.

  3. A Function needs to send data to a Manager or another Function. Functions cannot see Manager or other Function classes. Instead, to send data to a Manager you should expose a UnityEvent that the Manager/Function can subscribe to.

  4. A Function needs to request data from a Manager or another Function. Again, Functions cannot see Manager or other Function classes. Instead, to request data you need to create a UnityEvent that the other code subscribes to. In the case of a Manager, the UnityEvent can trigger a callback that pushes data to the Function. In the case of another Function, the Unityevent should trigger a second UnityEvent that the original Function can be subscribed to.

Datasets

Allen CCF

At the core of the planner is the Allen CCF annotation dataset. You can learn more about the dataset on the Allen’s documentation page. We use the 25 um atlas which has dimensions 528 x 320 x 456, the nrrd file downloaded from the Allen.

The annotation dataset is stored internally as a Texture3D object for GPU access as well as in an AnnotationDataset object.

The 3D models are controlled via the CCFModelControl component. Models are loaded from the shared addressables system and the mesh is saved locally to render the models.

Rat atlas

[todo]

Probes

Probe models are stored as prefabs and instantiated by the tpmanager. Each probe is controlled by a ProbeManager component which also handles probe movement. When a probe is moved it updates its associated probe panel prefabs, which are run by the TP_ProbePanel and ProbeUIManager components. The probe panels are interpolated using a custom shader which is a variant of the in-plane slice shader.

Probe Insertion Coordinates

The ProbeInsertion class represents a target coordinate in space. At the lowest level a probe has a tip coordinate (Vector3: ap, ml, dv) space and a set of probe angles (Vector3: azimuth, elevation, spin). Note that a ProbeInsertion must be defined with both a ReferenceAtlas and AtlasTransform, which define the relationship between the insertion’s coordinates/angles and Unity world space.

Using an AnnotationDataset it’s possible to recover the entry/surface coordinate of an insertion.

Coordinate Spaces and Transforms

Each Atlas space (e.g. mouse CCF, or rat Waxholm) has its own set of axes and coordinates. Even within a single species different researchers may have used different conventions to refer to directions in space. To deal with this, we introduce three concepts: Unity World Space, Coordinate Spaces, and Coordinate Transforms.

Unity “World” space is the ground truth space inside of Unity. It has an X, Y, and Z axis and units are measured in millimeters. The (0,0,0) coordinate is at the center of the space. All of the objects in the Unity scene are placed in Unity World coordinates. This is the only real coordinate system in the Unity Editor. What this means is that when you reference a Transform on a GameObject the Transform.position value is returning the object’s coordinates in Unity World space. If you want to know where this coordinate is in a particular Coordinate Space, you would need to use the World2Atlas function. If you want to know what direction a unit vector points in a ReferenceAtlas, you would use the World2AtlasAxisRotation function.

overview image

A ReferenceAtlas defines an axis rotation and a relative offset for the (0,0,0) coordinate. For example, the CCF space defines a (13.2 x 11.4 x 8 mm) rectangle with the (0,0,0) coordinate in the “front, left, top” corner. The axes are rotated so that the +Z axis becomes +AP, the +X axis becomes -ML, and the +Y axis becomes -DV. In Pinpoint, we then override the relative offset by moving it to Bregma at (+5.4, +5.7, +0.33) in CCF Space.

A AtlasTransform is necessary to represent situations where a ReferenceAtlas has been further rotated or warped. For example, you might want to know your coordinates in Paxinos space but visually see the Allen CCF annotations. To go back and forth between these spaces we use a AtlasTransform. The simplest transform is an AffineTransform which can stretch or shrink each axes and/or reverse them and can apply rotations. For example, the MRITransform flips the directions in CCFSpace, applies a scaling to each axis, and pitches the brain up by five degrees. More complex non-linear transforms are also possible.

World Space is Transformed

The most important thing to keep in mind is that objects in the scene are positioned according to their transformed coordinates. What the heck does that mean!?! This means that the Transform.position of a 3D model in the scene is computed by calculating: RefrenceAtlas.Atlas2World(AtlasTransform.Transform2AtlasAxisChange(coordinate)). Note that we do not un-transform the cooordinate by using AtlasTransform.Transform2Atlas, it’s only correct to do this when you need to know where a Transformed coordinate is in a different AtlasTransform.

The reason that we represent objects in their Transformed coordinates is that this means that objects in the Unity scene obey euclidian geometry, i.e. the distance between two points in the Unity scene (and therefore in the Transformed space) can be calculated with Vector3.Distance and the angle between two vectors in the scene (and therefore in the Transformed space) can be calculated using Vector3.Angle, these distances are angles are the correct distances and angles in the Transformed Coordinate Space. If it’s not obvious why this is important ask Dan to explain it.

The next question you should ask yourself now is: if points in the Unity scene are transformed, then

Moving between spaces and transforms

Lets work through some examples.

Where should I place the tip of a probe in Unity World space: Because coordinates in the Unity scene represent the transformed coordinates we need to first take the coordinate and find it’s position in its ReferenceAtlas by calculating: AtlasTransform.Transform2AtlasAxisChange, then to find the corresponding point in the scene we use ReferenceAtlas.Atlas2World.

How do I interpolate annotations along a probe insertion: Here we are going to need to translate between two ReferenceAtlass: from the Space the insertion is defined in and to the space where the annotations are defined. In addition, we need two points a start and an end coordinate, so that we can keep track of directions as we do our transforms. In general, to translate between ReferenceAtlass you need to use Unity World space as an intermediate. So the full set of steps here would be:

  1. Un-transform the start/end coordinates: AtlasTransform.Transform2Atlas

  2. Move into World Space: ReferenceAtlas.Atlas2World

  3. Move into the new coordinate space: ReferenceAtlas.World2Atlas

  4. Transform into the space’s transform (if there is one): AtlasTransform.Atlas2Transform

How do I move vectors between tranforms and spaces? Vectors are a special case, since we don’t want to scale them and they have no origin, but we still want to make sure they get rotated correctly. Both ReferenceAtlas and AtlasTransform have special AxisChange functions that apply all rotations but skip scaling and origin changes, for use with vectors. If you pass a unit vector to these functions, they should return a unit vector.

How do I know the coordinates of a probe insertion in a different AtlasTransform: This happens when you change the active AtlasTransform in the scene and all the probes need to be updated. This is as simple as going back into un-transformed space and then transforming into the new one. Hopefully it’s clear how you do that by now!

Coordinate naming conventions

Because we can’t attach metadata to every Vector3 we pass around representing an (AP, ML, DV) coordinate, we need a way to remind ourselves about whether the coordinate is in World or Reference Atlas space, and then whether the coordinate is transformed or not. Note that being transformed is independent of what space you are in!

When you name position variables use a sensible name like position as a prefix. Then, append World or Atlas according to what space the data is in and then append U or T according to whether the coordinate is in an un- or transformed space.

Building new features and fixing bugs

We use Github Issues to track development. Any changes you make should first be posted as an issue and flagged either as a bug or a feature and assigned to the relevant developer. When you complete adding a new feature you should either merge it to the corresponding development branch (if it’s minor) or submit a pull request (if it’s a major feature change). In either case, you should link to the relevant commit in the issue.

We use a Project to track development on the planner and milestones to assign bugs/features to releases.

When a bug/feature is closed it should be added to the incremental update list on the next Release page, or added to the current release as a hotfix.

Builds

Pinpoint can build to WebGL (primary), Windows, and Linux. Building to Mac is more complex than we can handle right now.

Addressables

Any assets that don’t need to be immediately loaded with the trajectory planner should be marked as Addressable and loaded asynchronously.

If you modified the Addressables assets you need to re-build these and deploy them on the asset server before you build to targets. In the Addressables Groups window go to Build > New Build > Default Build Script, or use Build > Update a Previous Build if you only modified existing assets but did not add new ones.

Remote Build and Load Paths

Addressables should be built to the remote build and load paths, which should point to ServerData and http://data.virtualbrainlab.org/NPTraj/version/[BuildTarget] respectively. These can be modified in the Addressables Profiles window.

We use a versioning system for Addressable builds to avoid over-writing guid values by accident. Each new Addressables build should target a new version number.

WebGL

WebGL builds function without extra steps, just zip the output directory, move it to the asset server, and swap it for the existing file. Note that there is an .htaccess file in the Build directory which needs to be manually copied to the new release.

Windows

Windows builds function without extra steps, just zip the output directory and upload it as either a hotfix to the Releases page (make sure to increment the hotfix number) or as a new release.

Linux

Linux builds function but you need to load the build on a Linux machine and mark the output file as executable.

Mac builds

Mac builds require extra steps to certify the build, we don’t have the capacity right now to deal with these.

Release information

As of 2023/08/31 Pinpoint has been accessed by:

79 unique users on Steam 170 unique downloads from Github Releases Unknown users via WebGL: estimate of 200 active users from survey

Github releases

Run this command:

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <token>" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/repos/VirtualBrainLab/Pinpoint/releases

Using a token with VBL access, which can be created from: https://github.com/settings/tokens?type=beta

Steam download information

Accessed from the Steam financials page