Transcript A new tool for imaging at the IFR3 (IGH/CCIPE/CRIC): Image

Image restoration by deconvolution
Volker Bäcker
Montpellier Rio Imaging http://www.mri.cnrs.fr/
Pierre Travo
IFR3
Giacomo Cavalli
Frederic Bantignies
Patrice Mollard
Nicole Lautrédou-Audouy
Jean-Michel Poulin
[email protected]
Overview
➲
➲
Part 1
●
introduction
●
what is deconvolution ?
●
how does it work ?
●
when should it be used ?
Part 2
●
what are the parameters to know and care about
for image restoration by deconvolution?
fluorescence microscopy
specimen has to be
in focal distance
●
●
to image 3d specimen
move focal plane
through specimen
●
creating stack
of slides
●
●
fluorescence microscopy
specimen marked with dye that emists light of one wavelength
while being stimulated by light of another wavelength
●
Microscope types
●widefield
●confocal
●two photon
●
whole specimen bathed in light
image is constructed point by point to keep out out-of-focus light
two photons needed to stimulate emission, similar effect as confocal
Example: 2d widefield
Image from microscope
After deconvolution
(same levels)
Immunostaining on whole mount drosophila Embryo
Using an antibody against a nuclear protein
Example 3d confocal
Image from microscope
After deconvolution
Example: time series 2 photon
Image from microscope
After deconvolution
The aquired image is not the „real“
image
➲
Images are degraded due to the limited aperture of the
objective
➲
Deconvolution can be used to get an image nearer to the
real object
●
➲
by using knowledge of the imaging process
and the properties of the microscope
Deconvolution can be used for all kinds of fluorescence
microscope images:
●
2D, 3D, time series, widefield, confocal, 2 photon
Sources of image degradation
➲
Noise
➲
Blur
●
➲
Scatter
●
➲
Can be handled by image restoration
random distribution of light due to
heterogenous refrection index within specimen
Glare
●
random distribution of light that occurs
within the optical train of the microscope
Causes of image degradation
Noise
➲
Geben Sie eine Zusammenfassung der
momentanen Situation
Causes of image degradation
Noise
➲
Where does the noise come from ?
random fluctuations in the signal intensity
●
●
●
variation of the incident photon flux
interfering signals from electronic system of the
captor device
Causes of image degradation
Blur
Before restoration
After restoration
Causes of image degradation
Blur
➲
Where does the blur come from ?
●
contributions of out-of-focus light to the imaging plane
●
diffraction
●
a result of the interaction of
light with matter
●
diffraction is the bending
of light as it passes the
edge of an object
How does deconvolution work
➲
Image restoration
Get rid of noise
●
●
assume random noise
with Poisson distribution
●
remove it
Get rid of blur
●
●
Compute real image from sample
●
by applying a model of how the
microscope degraded the image
deconvolution
Point Spread Function
➲
Point spread function (psf)
●
Model of how one point
is imaged by microscope
●
Experimental
●
aquired by taking an image of
„point like objects“ - beads
●
Alternatevely, point like object
present in the acquired image
itself can be usedf.
Theoretical
●
●
computed from the microscope
and captor parameters
Convolution (Faltung)
➲
aquired image = real image convolved with psf
➲
Convolution is an integral
that expresses
●
amount of overlap of functions as g
is shifted over f.
i x
f x'
g x x ' dx'
i(x) : aquired image
f(x) : object function
g(x) : point spread function
●
N pixel => O(N*N) operations to compute it
Fourier Transform (FT)
F
f x e
i2
x
dx
Signal can be represented as sum of
sinoids
●
FT transforms from spacial to
frequency domain
●
Convolution theorem
i x
f x'
g x x ' dx' <=>
i(x) : aquired image
f(x) : object function
g(x) : point spread
function
I F G
I fourier transform of i
F fourier transform of f
G fourier transform of g
*
Object function
Fourier transform (FT)
FT can be computed in
O(n * log n)
psf
inverse FT
Object function
convolved with psf
FT
Deconvolution
i x
f x'
g x x ' dx' <=>
I F G
➲
Deconvolution:
find object function f for given image i and psf g
➲
Unfortunatly it is not practicable to compute
I
G
G has zeros outside certain regions
●
●
Setting F zero for these would create artefacts
In practice there is noise
●
●
➲
F
N/G would amplify noise
I
F G N
It's not possible to reconstruct the real object function
Deconvolution algorithms
➲
Solution
Find an algorithm that computes a function f' so that
●
●
f' estimates f as good as possible
●
works in the presence of noise
➲
Different deconvolution algorithms exist
➲
In general best for fluorescent microscopy:
●
(Classical) Maximum Liklihood Estimation - MLE
Maximum Likelihood Estimation
➲
Tries to optimise f' iteratively
➲
The basic principal is (but there's more to it)
➲
g(i|j) :
psf - the fraction of light from true location j
that is observed in pixel i
Fraction of light from pixel j
that hit other pixels
f new , j
f old , j
g i j
g j k f old , k
ii
i
k
Richardson and Lucy
R-L Iteration
Fraction of light from other pixels
that hit pixel j
0,3
0,2
1
A
B
C
D
0,2
0,1
6
psf
5
1
Numerator
2
1 C3 + 0.1 C4 + 0.2 B3
3
Denominator:
realign my light to me
get rid of foreign light that hit me
3
4
4
3
1 C3 + 0.3 C4 + 0.2 B3
4
image
5 * [5*1 / (5*1 + 0.3*4 + 0.2*6) + 0,1*4 /(5*1 + 0.3*4 + 0.2*6) + 0,2*6/(5*1 + 0.3*4 + 0.2*6)]
5 * [5 / 7.4 + 0.4/7.4 + 1.2/7.4]
5 * [(5 + 0.4 + 1.2)/7.4]
5 * [6.6 /7.4]
5 * 0.891891
4.459459
fraction of light lost
f new , j
f old , j
g i j
g j k f old , k
ii
i
k
New estimate
last estimate
aquired image
last estimate
fraction of light from
others
Summary and conclusions 1
✔
image from microscope is degraded
✔
it contains noise and blur
✔
blur can be described as a convolution of object function and psf
✔
✔
➔
➔
image nearer to the object function can be obtained by image
restoration yielding higher resolution and better contrast
MLE is a deconvolution algorithm approriate for fluorescent
microscope images
imaging process is not finished finished without
deconvolution
do it whenever high quality images are needed
Image restoration in practice
Many deconvolution software packages are commercially available
They use various types of deconvolution algorithms
In addition to these algorithms, they might incorporate other
imaging tools, such as filters of different kinds. Moreover, different
types of algorithms may introduce or not some « assumptions »
concerning the image sent to restoration.
In general, it is important to test the software. One basic « rule of
thumb » is also that the restoration should respect the acquired
image in terms of objects visible and of their relative intensity.
Objects « appearing », « disappearing » or changing relative
intensity with respect to neighboring structures are diagnostic of
problems. These problems might be due to the setting of relevant
parameters or, in the worst case, of poor quality of the software
Image restoration using the
huygens2 software from SVI
➲
➲
http://www.svi.nl/ - website of Scientific Volume Imaging (SVI)
It is the software used at the
Institute of Human Genetics
Relevant parameters in deconvolution
Setting Microscope parameters
➲
microscope type
●
widefield and multipoint confocal
● work with ccd camera
●
single point confocal and two photon
● work with photomultiplier
●
➲
different point spread functions
if you don't know
●
Ask your imaging facility and look at the specifications of
your microscope
Microscope parameters
➲
Numerical aperture
●
measure of ability to gather light and resolve fine specimen detail at a fixed object
distance
●
higher magnification doesn't yield higher resolution, higher NA does
➲
Maximal value written on objective
➲
Can't be larger than the the refractive index n of the medium
NA n sin
Sampling theorem
●
Imaging converts an anlog signal into a digital signal
When converting an analog signal into a digital signal
the sampling theorem applies
●
Nyquist-Shannon sampling theorem
“the sampling interval must not be greater than one-half the size of
the smallest resolvable feature of the optical image”
●
sampling at nyquist rate means using exactly this interval
●
sampling interval is the pixel size in our image
x
4 NA
Undersampling and oversampling
under sampling
loss of information
●aliasing artefacts
●
over sampling
higher computation times and
storage requirements
●longer acquisition times,
photobleaching.
●
under sampling
example. An object of
a given shape
(dashed line) can be
interpreted as a
different shape (thick
line) if too few points
are acquired along
any of the x,y,z axes
Changing the Numerical Aperture (NA) for
widefield / two photon
➲
huygens2 allows under/oversampling within a range
➲
at the borders of this range deconvolution can be
done but results are not good
➲
In this case better results when “lying” about NA
i
i
x
0.5
i
x
x
1.2
x
nyquist sample size
➲
if sampling size not in range change NA
x
4 NA
Microscope parameters
➲
Excitation and emission wavelength
➲
fluorescent dye absorbs light of one wavelength and
emits light of another wavelength
➲
filter cubes are used to ensure that only light of a wanted wavelength passes.
exitation and emission wavelengths depend on the cube used
●
●
GFP 473, 525
Microscope parameters
➲
The objective magnification used
●
determines the pixel size in the image
●
ccd camera
●
Pixel size = ccd element size / magnification
(eventually modified by other parameters)
photomultiplier
●
●
pixel size depends on resolution
and magnification
Microscope parameters
➲
➲
Refractive index n of the objective medium
●
oil
1,51500
●
water
1,33810
●
air
1,00000
Should match the refractive index of the sample
medium
Otherwise
●
●
Magnification error in axial direction
●
Spherical aberration (psf deteriorates with increasing depth)
Microscope parameters
➲
Cmount factor
●
adaptor that attaches the camera to the microscope
●
might contain additional optic that
●
●
changes the overall magnification
●
and therefore the pixel size
value is 1 if no additional optic
present
ps
ccd
obm cmf
Microscope parameters
➲
Tube factor
●
the tube might contain additional optics to change the tube
length
●
this changes the overall
magnification
●
and therefore the pixel size
ps
ccd
obm cmf tf
Microscope parameters
➲
sample medium
➲
refractive index n
➲
●
default (all media for example water)
1,33810
●
liquid Vectashield (not polymerized)
1,49000
●
90-10 (v:v) glycerol - PBS ph 7.4
1,49000
●
prolong antifade
limits the NA and
therefore the possible
resolution
1,4
NA nsin
Captor parameters
➲
size of the unitary ccd captor
➲
image sensor of the camera
➲
●
ccd – charge coupled device
●
diodes that convert light into electrical charge
property of the camera
●
Coolsnap
6450 nm
●
Micromax
6700 nm
For photomultiplier
● the pixel size is asked
● see table in help pages
Captor parameters
➲
Binning
➲
take nxn elements as one
➲
more light per pixel
➲
reduces noise
➲
higher signal to noise ratio
➲
lower resolution
ps
ccd bin
obm cmf tf
Captor parameters
➲
in case of XZY
●
➲
z step size
in case of time series
●
time interval
Captor parameters
➲
in case of confocal
●
pinhole radius
●
pinhole
●
keep out out of focus light
●
pinhole either fixed or
adjustable
●
Backprojected radius in nm
●
●
Size of pinhole as it appears in the specimen plane
rb
size should match airy disk (2d psf) size
6.66 for LSM510
r phys
m system mobj
task parameter
➲
Style of processing
step
●
process image slide by slide
●
●
converts stack into time series for processing
●
converts result back into stack
volume
●
●
use 3d information
step combined
●
●
do step processing
●
followed by volume processing
with fixed parameters
Full restoration parameters
➲
signal/noise ratio
●
the ratio of signal intensity to noise intensity
●
high noise case
can be measured in the image
●
Single photon hit intensity
●
●
find low intensity voxels from one photon hit
– add values – subtract background
Max voxel value
●
●
value of brightest voxel
low noise case
●
S
N
●
single photon hits can´t be seen
●
rough guess is sufficient
maxVoxelValue
singlePhotonHitIntensity
Full restoration parameters
➲
background offset
●
empty regions should be black
●
but contain some light in reality
●
subtract mean background to see object clearly
Full restoration parameters
➲
number of iterations
too low
●
f new , j
f old , j
g i j
g j k f old , k
ii
i
k
●
too high
●
●
optimal restoration not yet achieved
●
takes longer to compute
●
some signal may be removed
Usually between 30-70
Summary and conclusions 2
➲
deconvolution should be used to obtain high quality
images
for all kind of fluorescent microscope images
➲
parameters of the imaging system have to be entered to
create a model of the image degradation
End of presentation
[email protected]
[email protected]
Links
participants
➲
Montpellier RIO Imaging
http://www.mri.cnrs.fr/
➲
IFR3 / CCIPE
http://www.montp.inserm.fr/ifr3.htm
➲
➲
IGH
http://www.igh.cnrs.fr/
CRIC
http://www.iurc.montp.inserm.fr/cric/index.htm
literature
●
●
Introduction to Fluorescence Microscopy
http://www.microscopyu.com/articles/fluorescence/fluorescenceintro.html
How does a confocal microscope work?
http://www.physics.emory.edu/~weeks/confocal/
●
Two-Photon Fluorescence Microscopy
http://www.fz-juelich.de/ibi/ibi-1/Two-Photon_Microscopy/
●
Deconvolution in Optical Microscopy
http://micro.magnet.fsu.edu/primer/digitalimaging/deconvolution/deconintro.html
Links
literature
●
Diffraction of Light
http://micro.magnet.fsu.edu/primer/java/diffraction/basicdiffraction/
●
Image restoration: getting it right
http://www.svi.nl/support/talks/GettingItRight.pdf
●
Image Restoration in Fluorescence Microscopy
http://www.ph.tn.tudelft.nl/Publications/PHDTheses/GMPvanKempen/thesis_kempen.ht
ml
●
Image restoration in one- and two-photon microscopy
http://www.svi.nl/support/talks/Vancouver97.pdf
●
Introduction to Convolution
http://cnx.rice.edu/content/m11542/latest/
●
An Introduction to Fourier Theory
http://aurora.phys.utk.edu/~forrest/papers/fourier/
●
●
A Self Contained Introduction to Fourier Transforms
http://www.doc.eng.cmu.ac.th/course/cpe496/notes/fourier.pdf
Convolution theorem
http://www.fact-index.com/c/co/convolution_theorem.html
Links
literature
●
Three-Dimensional Imaging by Deconvolution Microscopy
Article ID meth.1999.0873, available online at http://www.idealibrary.com on IDEAL
●
Deconvolution of confocal images of dihydropyridine and ryanodine receptors
in developing cardiomyocytes
http://www.sfu.ca/~tibbits/research/JAP04.pdf
●
Maximum likelihood estimation via the ECM algorithm: A general framework
http://www.jbs.agrsci.dk/~lfo/talks/ECM_talk.pdf
●
The influence of the background estimation on the superresolution properties of
non-linear image restoration algorithms
http://www.ph.tn.tudelft.nl/People/lucas/publications/1999/SPIE99GKLV/SPIE99GKLV.pdf
●
Numerical Aperture and Resolution
http://micro.magnet.fsu.edu/primer/anatomy/numaperture.html
●
User guide for Huygens Professional and Deconvolution Recipes
●
http://www.svi.nl/download/
Digital Image Sampling Frequency
http://www.olympusmicro.com/primer/java/digitalimaging/processing/samplefrequency/index.html
●
●
Filter Cubes
http://www.olympusmicro.com/primer/techniques/fluorescence/filters.html
Filters for fluorescence microscopy
http://www.nikon-instruments.jp/eng/products/option/index1.aspx
Links
literature
●
Immersion Media
http://www.olympusmicro.com/primer/anatomy/immersion.html
●
How Digital Cameras Work
http://electronics.howstuffworks.com/digital-camera2.htm
●
Pixel Binning
●
http://micro.magnet.fsu.edu/primer/digitalimaging/concepts/binning.html
CCD Signal-To-Noise Ratio
http://www.microscopyu.com/tutorials/java/digitalimaging/signaltonoise/