Important Notes and Disclaimer

All code is provided as is and without any warranty of functionality or fitness for a given task.

The framework is based on caffe (http://bvlc.eecs.berkeley.edu). The caffe framework can run entirely on the CPU or use GPU acceleration. If available, it is highly recommended to use GPU acceleration. By using GPU acceleration the computation times are drastically reduced by a factor of 20-100 (i.e. computations take minutes instead of hours).

Please file bug reports to maid@informatik.uni-freiburg.de including information about your system and hardware.

Content

Installation Instructions

Prerequisites

You need a computer for runnning the backend (Cell Net) and a computer for running the frontend (ImageJ with our U-Net plugin). You can run the frontend on the same computer as the backend if desired.

Backend (Server) requirements:

Frontend (Client) requirements:

Backend (Cell Net) Setup

Setup on Public Cloud

This option is well suited to test-drive Cell Net for your specific images as it allows to make predictions with a minimal financial (about one USD) and minimal time (about one hour) investment, while fully benefiting from GPU accelerated code execution.

The proposed setup is reasonably safe. The authentication happens through an RSA key and communication between your local client PC and the AWS cloud instance is encrypted. However, be aware that you will transmit your images to an external location, possibly located outside your country. Be sure to comply with your organizational data storage rules.

The Amazon Machine Image (AMI) that includes the operating system and additional software is provided by Amazon. Note, that with the proposed procedure you are at no point required to enter a password, except to access the Amazon AWS web page.

Setup Amazon AWS

Setting up an AWS account requires a valid credit card and a callable phone number. The cloud instances (i.e. the virtual servers) are bound to a region. Not every region provides the same infrastructure. During the setup process you will be asked to generate a key pair. This key pair is specific to a region and since we later are going to reserve resources that are not available in all regions, make sure you create the key in the region eu-west-1.

Follow all instructions under this link and select region eu-west-1 when creating the key pair: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/get-set-up-for-amazon-ec2.html.

Run a virtual machine with GPU

Bare in mind that as soon as an instance is running, you will be charged an hourly cost. Typically, the hourly cost will be below 1 USD per hour. Make sure to terminate your session(s) when done. The virtual machine with GPU will play the role of the CellNet server.

We will use a spot instance, which is a temporary instance that will be at your disposal as long as the market price is below a threshold set by you. Unlike spot instances, the regular on-demand instances have a fixed price and can be paused. However, no on-demand instances are available to new AWS users, thus we use spot instances for this procedure. Follow the instructions in the link: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#using-spot-instances-request to launch a spot instance. First, change to region EU (Ireland). Choose g2.2xlarge as instance type and ami-0d77397e (Ubuntu 16.04) as root image. Note, that above AMI will only work in region eu-west-1.

Connect to your EC2 instance

Once the spot instance is running, log on to the instance as described at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html#ec2-connect-to-instance-linux and copy the IP address of your running EC2 instance to your clipboard or write it down. The username is ubuntu and the authentication is done by the saved key. The way to connect depends on your operating system.

Run Cell Net setup

Once you are logged in, run following commands:

cd ~
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.44-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_8.0.44-1_amd64.deb
sudo apt-get update
sudo apt-get install -y cuda unzip
sudo apt-get clean
wget --user ***** --password ***** http://lmb.informatik.uni-freiburg.de/people/maid/cellnet_supplemental/caffe_unet_package_16.04_gpu_no_cuDNN.zip
unzip caffe_unet_package_16.04_gpu_no_cuDNN.zip
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/ubuntu/caffe_unet_package_16.04_gpu_no_cuDNN/libs" | cat - ~/.bashrc > tmp
echo "export PATH=$PATH:/home/ubuntu/caffe_unet_package_16.04_gpu_no_cuDNN/bin" | cat - tmp > ~/.bashrc
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-8.0/lib64" | cat - ~/.bashrc > tmp
mv tmp ~/.bashrc

Use CellNet Client

While the EC2 instance is running, follow the section Frontend (Client) Setup further below. On your local client computer enter the IP address of your EC2 instance, choose ubuntu as user name and choose the private RSA key specified earlier.

Save Snapshot for Future Use (Optional)

IMPORTANT NOTE: Storing a snapshot of your image will cost approximately 0.05 dollars per month per GB. The proposed image has a size of 16GB, thus will cost 0.4 dollars per month to be stored. Saving a snapshot allows to re-start the Cell Net server instance later and skip the configuration steps. Here is a link describing how to save a snapshot: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html.

In order to start an instance based on a snapshot, it needs to be converted to an AMI (subsection Creating a Linux AMI from a Snapshot).

Terminate instance

IMPORTANT NOTE: You will be charged for each hour that your instance is running, regardless whether you are interacting with it or not. To terminate the instance follow the instruction in http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html.

On-Premise Setup

We recommend using the current Ubuntu 16.04 on a computer with a compatible GPU with at least 4GB of vRAM. The tutorial will assume the use of Ubuntu 16.04. If you use the older Ubuntu 14.04 release, replace the string "16.04" everywhere with "14.04". If you don't have a gpu replace "gpu" with "cpu" and skip the installation of CUDA.

We describe the setup process for the fictional user "maid" who wants to install the caffe U-Net backend in the directory "/home/maid/cellnet" on the host with IP "192.168.0.10" and hostname "backendserver". He runs both backend and frontend on the same machine with ubuntu 16.04 and a GTX 980M GPU with 8GB. Installing and Plugin usage instructions are also provided as a video tutorial.

Installation of CUDA 8.0

Download the CUDA 8.0 library:

cd /tmp
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.44-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_8.0.44-1_amd64.deb
sudo apt-get update
sudo apt-get install -y cuda unzip
sudo apt-get clean
rm cuda-repo-ubuntu1604_8.0.44-1_amd64.deb
cd ~

Set the environment by adding the following line to the top of your .bashrc

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-8.0/lib64/

Installation of the Cell Net package

Download caffe_unet_package_16.04.zip and caffe_models_2d.zip to /home/maid/cellnet.

cd /home/maid/cellnet
unzip caffe_unet_package_16.04_gpu_no_cuDNN.zip
unzip caffe_models_2d.zip

Edit your ~/.bashrc file to set up the environment for the caffe U-Net software (You can combine the LD_LIBRARY_PATH statement below with the one above):

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/maid/cellnet/unet_package_16.04_gpu/libs
export PATH=$PATH:/home/maid/cellnet/unet_package_16.04_gpu/bin

Test, if it works: start a new shell and execute

caffe

A usage message should appear on the screen.

Test whether execution over ssh works:

ssh localhost caffe

A usage message should appear on the screen.

Frontend (Client) Setup

  1. (optional) Check, if you can connect to the server and execute a program there:
  2. Installation of the Fiji U-Net plugin.

The plugin needs to be installed on the client computers only. It has no included segmentation capabilities, instead it interfaces existing caffe installations on the local machine or a backend server that is accessible via secure shell (ssh) and secure file transfer (sftp).

Using the FiJi U-Net plugin with the pretrained 2D Cell Net

Walk-through example

  1. Open a gray valued image, e.g. /home/maid/cellnet/caffe_unet_package_16.04_gpu/2dcellnet_finetune/valdata_rbc/48hr-003-DIC.tif
  2. Use Image->Properties... to set the pixel extents
    Unit:micrometer
    Pixel width:0.45
    Pixel height:0.45
    Voxel depth:1

    (For different images exchange width and height values by the actual pixel-size of your image)

  3. Setup a new segmentation with Plugins->U-Net->U-Net Segmentation Manager->New Segmentation
    Model:2D Cell Net v0*
    Weight file:/home/maid/cellnet/caffe_models_2d/2d_cell_net_v0_snapshot_iter_150000.caffemodel.h5
    Process Folder:/home/maid/cellnet
    Use GPU:GPU0
    Memory (MB):8000
    Host:192.168.0.10
    Port:22
    Username:maid
    Password:********

    *see below how to select the Folder to chose Model from

  4. Click "OK"

Segmentation progress will be shown in a new row of the job table. After the caffe_unet binary on the backend server has finished the "Cancel" button on the right will change to "Show". Click it to show the segmentation result.

U-Net Segmentation parameters

Model:
Use the "Select folder" icon on the right of the "Model:"-line to select the local folder containing the Cell net model (2D or 3D) (e.g. /home/maid/cellnet/caffe_models_2d). Then select the model you want to use for segmentation in the combo box.
Weight file: (Path on the backend server)
The weight file contains the weights of the trained Cell Net. The weights file must be available via the given path on the backend (server) running caffe_unet.
Process folder: (Path on the backend server)
During the segmentation process intermediate files will be created and stored in the given folder on the backend server running caffe_unet. These files are: the model definition, the normalized image data, and the segmentation result. After closing Fiji, these temporary files will be removed.
Use GPU:
Select the GPU that is used for the segmentation. In CPU-only mode, select "None".
Tiling Layout:
Depending on the selected model, there are various options of defining the tiling layout. If available (which is the case for 2D Cell Net) we recommend to use the "Memory (MB)" option with which you can define the available amount of memory on the GPU. The options "Tile shape (px)" and "Grid (tiles)" are always available and let you define the maximum tile size or the tiling layout. The given values are upper bounds and will be adjusted to appropriate network input sizes. "#Tiles" uses the given amount of tiles and automatically defines the tile shape for optimal performance. "#Pixels/Tile" let's you set the number of input pixels of each tile. See also "caffe_unet --help" for further details.
Use remote host:
Check this to use a backend server for segmentation. If you want to use the local machine for segmentation, please uncheck.
Host:
The hostname of the backend server on which caffe_unet is installed.
Port:
The SSH port of the backend server (Default is 22).
Username:
Your username on the backend server.
Password:/RSA key:
Your SSH password on the backend server.
Keep original:
If checked, the original image is retained, otherwise it is replaced by the normalized image in processing resolution.
Output scores:
If checked, the output scores of the network are output instead of the binary segmentation masks. The output scores contain as many channels as there are classes. The segmentation can be obtained through the scores via a pixel-wise arg max operation over the channels. Scores are especially useful during training and fine-tuning to get an idea of the training progress.

Troubleshooting

If using GPU acceleration, make sure the graphics card is being recognized. Execution of

nvidia-smi

should show a table indicating the NVIDIA GPUs installed on the system.

When logging in to the server, the command

caffe_unet

should display a usage message by caffe. If not, make sure the $PATH and $LD_LIBRARY_PATH environment variables are set correctly. The same usage message must also appear when you run the command with a non interactive shell from a remote server:

ssh user@backendserver caffe_unet

If .bashrc contains an expression such as

case $- in
*i*);;
*) return;;
esac

or

[ -z "$PS1" ] && return

all instructions after that line will be ignored, thus the specification of the environment variables must be placed before this expression. Ensure that no outputs to standard output are generated in your .bashrc in non-interactive mode, otherwise file upload/download via sftp fails!

The selected model file is uploaded to the backend server when starting the segmentation. The weights must already reside on the backend server at the given location. The process folder is created on the backend server on demand given sufficient user rights.