3. Massaging and Manipulating Data Amplitude 0 0.2 0.4 0.6 0.8 -5000 -10000 -15000 Time (s) Sample Sample Prep Instrument Instrument Out put A signal can contain noise at a different frequency drift with time Distorts the “real” information about.
Download ReportTranscript 3. Massaging and Manipulating Data Amplitude 0 0.2 0.4 0.6 0.8 -5000 -10000 -15000 Time (s) Sample Sample Prep Instrument Instrument Out put A signal can contain noise at a different frequency drift with time Distorts the “real” information about.
3. Massaging and Manipulating Data 15000 10000 Amplitude 5000 0 0 0.2 0.4 0.6 0.8 -5000 -10000 -15000 Time (s) Sample Sample Prep Instrument Instrument Out put A signal can contain noise at a different frequency drift with time Distorts the “real” information about the sample Signal (Data) 1 1.2 Consider a system where our signal is a sin wave 15000 10000 Amplitude 5000 0 0 0.2 0.4 0.6 -5000 -10000 -15000 Time (s) It might have “noise” and it might “drift” with time 0.8 1 1.2 Consider a Sin wave as the signal Add some noise Composite signal contains noise A signal can contain noise at a different frequency drift with time 0.25 0.2 Signal 0.15 Amplitude 0.1 0.05 0 0 0.05 0.1 0.15 0.2 0.25 -0.05 -0.1 -0.15 -0.2 -0.25 Time Drift 0.3 0.35 0.4 0.45 0.5 The signal is superimposed on slowly changing background and Slowly decays 0.25 0.2 0.15 Amplitude 0.1 0.05 0 0 0.05 0.1 0.15 0.2 0.25 -0.05 -0.1 -0.15 -0.2 -0.25 Time 0.3 0.35 0.4 0.45 0.5 Enhancing the Signal of the desired Frequency Make use of: Summing “in phase” signals Statistics (std decreases as the sq rt of the Number of samples) Std = standard deviation, s Sq rt = square root What does “phase” mean? 180 degrees out of phase 90 degrees out of phase 15000 15000 10000 10000 Amplitude Amplitude 5000 5000 00 00 0.2 0.2 0.4 0.4 0.6 0.6 -5000 -5000 -10000 -10000 -15000 -15000 Time Time (s) (s) 0.8 0.8 11 1.2 1.2 Summing in phase sin waves leads to signal amplification 25000 25000 20000 20000 15000 15000 Amplitude Amplitude 10000 10000 5000 5000 00 00 0.1 0.1 0.2 0.2 0.3 0.3 0.4 0.4 0.5 0.5 -5000 -5000 -10000 -10000 -15000 -15000 -20000 -20000 -25000 -25000 time time (s) (s) Will make use of this concept to enhance Signal 0.6 0.6 0.7 0.7 0.8 0.8 0.9 0.9 11 This is the desired signal enhancement that comes from summing the entire waveform 25000 25000 If the summed waveforms are Slightly offset (out of phase) 20000 20000 The “enhanced” signal is distorted 15000 15000 Amplitude 10000 10000 5000 5000 00 00 0.2 0.2 0.4 0.4 0.6 0.6 0.8 0.8 11 1.2 1.2 -5000 -5000 -10000 -10000 -15000 -15000 -20000 -20000 -25000 -25000 tt This means that we design instruments to “trigger” the data signal acquisition timing to avoid distortion Sin wave Plus 180o out of phase wave = Destructive Summation 15000 15000 10000 10000 Amplitude 5000 5000 00 00 0.2 0.2 0.4 0.4 0.6 0.6 0.8 0.8 11 1.2 1.2 -5000 -5000 -10000 -10000 -15000 -15000 tt (s) (s) Will make use of this concept to reduce noise. In noise if it is random ½ of time it will Fluctuate positive and ½ time will fluctuate negative, thus is should sum “destructively) pp 12892 7690 N s~ 867 6 6 S x sample xblank x sample xblank N s pp 6 15000 12892 10000 7690 amplitude 5000 0 0 100 200 300 400 500 600 -5000 -10000 -15000 time or point S x sample xblank S 10000 10000 20000 S 20000 23 N 867 Sum of 2, 4, and 8 waveforms 100000 100000 91435 80000 60000 80000 40000 amplitude 73756 60000 20000 0 0 100 200 300 400 500 600 -20000 -40000 -60000 40000 -80000 -100000 amplitude time or point 20000 0 0 100 200 300 400 500 600 -20000 -40000 -60000 -80000 -100000 S 160000 160000 54.3 91435 73756 N 2946 6 time or point We went from S/N of 23 with 1 waveform to 54.2 with 8 summed waveforms # summed waveforms S/N Sqrt # summed waveforms 1 S 20000 23 N 867 1 4 S 46.4 N 2 8 S 160000 160000 54.3 91435 73756 N 2946 6 4/1 = 4 46.4/23=2.01 8/1=8 54.3/23 = 2.35 2.828 2/1=2 2.828/1=2.82 Observations? What implications does this have for designing an instrument? designing a method (quality control)? Build instruments with computers fast data acquisition time (micro to milli-seconds) Sum in phase signals over and over and over and over again Massage the Data after acquisition (Digitally) D time 4 0.1 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.2 0.21 0.22 0.23 0.24 "signal" -0.03185 -0.03503 -0.03821 -0.0414 -0.04458 -0.04776 -0.05094 -0.05412 -0.0573 -0.06048 -0.06366 -0.06684 -0.07002 -0.0732 -0.07637 noise 0.174321 -0.01688 -0.18818 -0.14037 0.070015 0.199355 0.099605 -0.11379 -0.19752 -0.05792 0.146468 0.187003 0.020172 -0.16849 -0.17265 sum 5 box 0.142473 -0.05191 -0.22639 -0.05843 -0.18177 -0.05661 0.025435 -0.03649 0.151594 -0.0248 0.048663 -0.03941 -0.16791 -0.06818 -0.25483 -0.08194 -0.11841 -0.06764 0.082805 -0.04402 0.120162 -0.04139 -0.04985 -0.06752 -0.24168 -0.09792 -0.24902 -0.10216 7 -0.01313 -0.05747 -0.08646 -0.07103 -0.03324 -0.0197 -0.04848 -0.08996 -0.10155 -0.07503 -0.04398 -0.04536 9 =AVERAGE(D4:D8) -0.05718 -0.08617 -0.0712 -0.0327 -0.01804 -0.04772 -0.09223 -0.10533 -0.07567 -0.03924 -0.03949 =AVERAGE(D5:D9) The “box” slides down= Sliding box car filter Raw signal 0.2 0.2 5 point 0.1 0.1 Sliding boxcar 00 Amplitdue Amplitdue 00 0.1 0.1 0.2 0.2 0.3 0.3 0.4 0.4 0.5 0.5 -0.1 -0.1 -0.2 -0.2 -0.3 -0.3 -0.4 -0.4 7 point Sliding box car -0.5 -0.5 time time (s) (s) 9 sliding box car gets much closer to the “real” signal. 0.6 0.6 0.7 0.7 0.8 0.8 Now that you have tried it I can tell you how to do it automatically Right click the data Add trendline Specify the number Of points in the Moving average Fourier Transform – eliminate high frequencies Consider a series of sin waves of Differing amplitudes and frequencies 1.5 Sin(f) Base frequency 1/3sin(3f) 1/5sin(5f) Multiples of the base frequency 1/7(sin(7f) 1 Voltage 0.5 0 0 0.2 0.4 0.6 -0.5 -1 -1.5 Time Sum the sin waves and what do you get? 0.8 1 1.2 What do you observe? 90% Sin(f) Sin(f) +1/3sin(3f) Sin(f) +1/3sin(3f) +1/5sin(5f) Sin(f) +1/3sin(3f) +1/5sin(5f)+ 1/7(sin(7f) 10% Rise time The fact that any waveform can be considered to be the sum of a large number of Frequencies allows us to do some nifty tricks to massage out noise 5000 4000 3000 Amplitude 2000 1000 0 0 0.2 0.4 0.6 0.8 1 1.2 -1000 -2000 -3000 -4000 -5000 Time Measurement A1 sin2f s,1t N1 sin215 f N ,1t Signal Noise Noise 160 Digitally filter the high frequency 140 120 5000 100 Amplitude 4000 3000 60 1000 0 0.2 0.4 0.6 0.8 1 40 FT 0 1.2 20 -1000 -2000 0 1 -3000 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Frequency -4000 -5000 Time subtract noise Signal wave 10000 4000 9000 3000 8000 2000 Anti-FT 0 0 0.2 0.4 0.6 -1000 0.8 1 1.2 relative amplitude 7000 1000 Amplitude Amplitude 2000 80 6000 5000 4000 3000 2000 -2000 1000 -3000 0 1 -4000 Time 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 frequency Running an FT in excel F f j f t exp 2jft dt 1 Euler’s theorem exp cos x j sin x jx F f f t cos2ft dt j f t sin2ft dt Discrete Fourier transform, k must be an integer power of 2: Allowed values are 2,4,8,16,32,64,128,256,512,1024) N 2k F f k 0 f t cos dt j k 0 f t sin2ft dt N N The digital FT performed by excel does not make reference to your measured time, but treats your data as a string of data from n to 2n. In cell a1 type a title: base frequency In cell b1 base frequency = 1 In cell a2 type a title: amplitude In cell b2 assign a value to the amplitude In cell c1 type a title: noise frequency In cell c2 type a title: noise amplitude In cell d1 assign a value to noise frequency In cell d2 assign a value to noise amplitude 2. Create a counter from 1 to 512 in cell A7 (label cell a5= time) 3. Create the actual normalized time in seconds so that your base frequency will just complete one full cycle in the 512 time points. =A7/512 4000 3000 2000 4. Create the sin wave starting in cell C7 =$B$2*SIN(2*PI()*$B$1*B7) Amplitude 1000 0 0 0.2 0.4 0.6 -1000 -2000 V o sin2ft -3000 -4000 Time 0.8 1 1.2 6. Create some noise to filter out In cell D7: =$D$2*sin(2*PI()*$D$1*B7) 7. Sum the signal (column C) and the noise (column D). Here I have it in column E. In cell 76: =C7+D7 5000 4000 3000 Amplitude 2000 1000 0 0 0.2 0.4 0.6 -1000 -2000 -3000 -4000 -5000 Time 0.8 1 1.2 Open the tools pack and go to Fourier transform. Specify the 512 data points associated with the signal plus noise in column E. Tell it to place the transformation in column F. signal The base frequency does not Always represent the signal Frequency, see next example Noise Lg amplitude At 15x . The data placed into column F will be complex number representation of the amplitude of the frequency. It consists of the base frequency and the noise frequency at 15x the base. The first point in column E will represent 0 frequency, or DC. Each sequential box will represent the amplitude of frequency 1 (1cycle/512 second), frequency 2 and so forth up to point 256 at which point the numbers display the amplitude of each decreasing frequency The data displayed is a complex number. To convert to something you can graph create a column J ( =imreal(F7) ) Create a histogram of column J. You do not need to use the Histogram package in Excel or to create bins here . Simply highlight the data in column J including only the 2nd through the ~ 30 data points (base frequency to frequency multiple 30). Using that single column insert a bar graph. The computer will automatically assign numbers 1, 2..... to your x axis. 160 140 120 Amplitude 100 80 60 40 20 0 1 2 3 4 5 6 7 8 9 10 11 Frequency 12 13 14 15 16 17 18 19 To filter the data create column H. Every number in column H should be zero except for the point representing the base frequency (point 8 in column F). Copy the complex number in column F over to column H. All other values should be zero. 10000 9000 8000 relative amplitude 7000 6000 5000 4000 3000 2000 1000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 frequency We made the amplitude Of the 15x frequency Go to zero The second half of the FFT output is A mirror image of the first half, so You have to make the 15x frequency Amplitude go to zero at the bottom too! Now re-create a time string of data by performing the inverse fft on column H. To do this go to tools, Fourier Transform. In the Fourier Transform command box you click inverse. Tell the computer to place the data in column I. This data needs to be converted from the complex notation. Type in column j =IMREAL(i7) Now plot the data in column M as an xy scatter plot. You should see a beautiful noise free sin wave identical to the original noise free sin wave you generated or acquired. 5000 4000 3000 Amplitude 2000 1000 0 0 0.2 0.4 0.6 -1000 -2000 -3000 -4000 Fft filtered -5000 Time 0.8 1 1.2 Another Example in which the Signal waveform is the sum of several sin waves 15000 10000 Amplitude (V) 5000 0 0 0.2 0.4 0.6 0.8 1 1.2 -5000 -10000 -15000 Measurement A1 sin2f s,1t As sin2f S ,Time A sin2f S ,n t 2 t .... (s) n N1 sin2f N ,1t N 2 sin2f N ,2 t ..... N n,1 sin2f N ,n t Signal Noise 50 Noise 45 Digitally filter the high frequency 40 35 Amplitude 15000 10000 30 25 20 15 0 0 0.2 0.4 0.6 0.8 1 FT 1.2 10 5 -5000 0 1 -10000 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103 109 115 121 85 91 97 103 109 115 121 frequency -15000 Time (s) subtract noise Square wave 2 1.5 1 3.5 0.5 3 0 2.5 0 0.2 0.4 0.6 -0.5 0.8 1 1.2 Anti-FT -1 -1.5 Amplitude Amplitude Amplitude (V) 5000 2 1.5 1 -2 Time (s) 0.5 0 1 7 13 19 25 31 37 43 49 55 61 67 Frequency 73 79 The next example is very complex time Varying signal. The example also shows what can happen if You cut too many high frequency components This part of the response Requires high frequency To define the response well 2048 Data points = 211 Total time = 18.5 s 0.12 0.1 0.08 1.4 0.12 Current, uA 0.06 0.04 0.02 0.7 1.2 0.1 0 0.8 0.9 1 1.1 1.2 1.3 -0.02 -0.04 0.08 -0.06 -0.08 V vs Ag/AgCl 0.06 0.04 0.8 0.02 0.6 0 -0.02 0.4 -0.04 0.2 -0.06 0 -0.08 0 2 4 6 8 10 12 14 Time (s) Signal A1 sin2f s,1t As sin2f S ,2 t .... An sin2f S ,n t N1 sin2f N ,1t N 2 sin2f N ,2 t ..... N n,1 sin2f N ,n t Signal Noise 16 18 20 Current (microAmps) V vs Ag/AgCl 1 1.4 Measurement A1 sin2f s,1t As sin2f S ,2 t .... An sin2f S ,n t 30 N1 sin2f N ,1t N 2 sin2f N ,2 t ..... N n,1 sin2f N ,n t 20 30 Amplitude 10 20 0 1 -10 2 3 4 5 6 7 8 9 10 11 12 DC -20 -30 10 -40 Amplitude Frequency 0 1 10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181 -10 -20 -30 cycles f s 1cycle 18.5s 1cycle f base 0.05405Hz 18.5s -40 Frequency Signal is composed of Frequency multiples of The base up to about 28x base frequency Base frequency is defined By the length of the data String subjected to FFT Measurement A1 sin2f s,1t As sin2f S ,2 t .... An sin2f S ,n t N1 sin2f N ,1t N 2 sin2f N ,2 t ..... N n,1 sin2f N ,n t 30 20 0 1 10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181 30 -10 20 -20 10 -30 -40 Amplitude Amplitude 10 0 1 10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181 -10 Frequency -20 0.12 Note distortion where there is Is a rapid change in the voltage signal Distortion is due to high frequency removal 0.1 0.08 Current, uA 0.06 0.04 0.02 0 0.7 0.8 0.9 1 1.1 1.2 1.3 -0.02 Remainder of signal Is well matched with Information content retained -0.04 -0.06 -0.08 V vs Ag/AgCl 1.4 Measurement A1 sin2f s,1t As sin2f S ,2 t .... An sin2f S ,n t N1 sin2f N ,1t N 2 sin2f N ,2 t ..... N n,1 sin2f N ,n t 30 20 0 1 10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181 -10 30 -20 20 Clipped all but 5xbase frequency -30 -40 10 Amplitude Amplitude 10 0 1 -10 10 19 28 Frequency 37 46 55 64 73 82 91 100 109 118 127 136 145 When we leave only 5x base frequency we have completely Distorted the underlying signal 0.12 0.1 0.08 Current, uA 0.06 0.04 0.02 0 0.7 0.8 0.9 1 1.1 -0.02 -0.04 -0.06 -0.08 V vs Ag/AgCl 1.2 1.3 1.4 Electronically Filter before digitization q C V I c V C2f cos2ft dq I dt o V IR dq d CV dV I C dt dt dt V V sin(2ft ) V IC X c o o V IC V o 2fC cos2ft Xc d sin(2ft ) dV o V dt dt 1 dV o V 2f cos2ft dt 2fC cos2ft Xc Amplitude only: 1 Xc 2fC V V1 IR1 Vout Voltage dropped from Here to ground V2 IR2 ground V2 IR2 R2 Vtotal IR1 IR2 R1 R2 V V1 IR1 Vout I Xc Xc Vtotal IR I X R1 X c 1 c R1 X c V2 IX c ground R12 X c 1 Xc 2fC Vo Vtotal 1 2fC 1 R 2fC 2 1 2 2 1 2fC V V1 IR1 V2 IX c ground Vo Vtotal Vo Vtotal Vo Vtotal 1 R 2fC 2 2 1 1 1 2fC R 2fC 2 1 1 2fCR 2 1 Vo Vtotal 2fC 2fC 1 2fCR 1 2 1 2 2 15000 15000 10000 10000 Amplitude Amplitude 5000 5000 After analog filtering 0 00 -5000 -5000 0.2 0 0.4 0.2 0.6 0.4 0.8 0.6 Vo -10000 -10000 -15000 -15000 Time (s) Time (s) 1 0.8 1.2 1 1.2 This is a “Bode Plot” Cut off frequency