Transcript Progress OpenEdge DBA Worst Practices

Hidden Gems:
UNIX Shared Libraries
Way better than PROBUILD.
A Few Words About The Speaker
• Tom Bascom, Roaming DBA & Progress User
since 1987
• President, DBAppraise, LLC
– Remote Database Management Service.
– Simplifying the job of Managing and Monitoring The
World’s Best Business Applications.
– [email protected]
• VP, White Star Software, LLC
– Expert Consulting Services related to all aspects of
Progress and OpenEdge.
– [email protected]
What is a UNIX Shared Library?
• A collection of executable code and data which is
loaded into memory separately from the launch of
the main program (_progres).
• Progress 4GL has supported Windows DLLs since
(at least) version 8.
• UNIX DLL support arrived with Progress version 9.
• I’ll be using Linux as an example but the principles
apply to all UNIX platforms.
More Info: http://www.dwheeler.com/program-library/Program-Library-HOWTO/t1.html
Why Would You Use One?
• To share code from 3rd party, non-Progress,
sources.
• To leverage proprietary internal libraries.
• To gain performance improvements from
routines written in faster languages.
• To access low-level OS features that are not
otherwise available.
/lib (and /lib64)
/lib> ls *.so*
ld-2.4.so
ld-linux.so.2
ld-lsb.so.2
ld-lsb.so.3
libacl.so.1
libacl.so.1.1.0
libanl-2.4.so
libanl.so.1
libattr.so.1
libattr.so.1.1.0
libaudit.so.0
libaudit.so.0.0.0
libblkid.so.1
libblkid.so.1.0
libBrokenLocale-2.4.so
libBrokenLocale.so.1
libbz2.so.1
libbz2.so.1.0.0
libc-2.4.so
libcap.so.1
libcap.so.1.92
libcidn-2.4.so
libcidn.so.1
libcom_err.so.2
libcom_err.so.2.1
libcrypt-2.4.so
libcrypt.so.1
libc.so.6
libdevmapper.so
libdevmapper.so.1.02
libdl-2.4.so
libdl.so.2
libe2p.so.2
libe2p.so.2.3
libext2fs.so.2
libext2fs.so.2.4
libgcc_s.so.1
libhistory.so.5
libhistory.so.5.1
libm-2.4.so
libmemusage.so
libm.so.6
libncurses.so.5
libncurses.so.5.5
libnscd.so.1
libnscd.so.1.0.0
libnsl-2.4.so
libnsl.so.1
libnss_compat-2.4.so
libnss_compat.so.2
libnss_dns-2.4.so
libnss_dns.so.2
libnss_files-2.4.so
libnss_files.so.2
libnss_hesiod-2.4.so
libnss_hesiod.so.2
libnss_nis-2.4.so
libnss_nisplus-2.4.so
libnss_nisplus.so.2
libnss_nis.so.2
libnss_winbind.so.2
libnss_wins.so.2
libpamc.so.0
libpamc.so.0.81.0
libpam_misc.so.0
libpam_misc.so.0.81.2
libpam.so.0
libpam.so.0.81.5
libpcprofile.so
libpthread-2.4.so
libpthread.so.0
libreadline.so.5
libreadline.so.5.1
libresmgr.so
libresmgr.so.0.9.8
libresmgr.so.1
libresmgr.so.1.0.0
libresolv-2.4.so
libresolv.so.2
librt-2.4.so
librt.so.1
libSegFault.so
libss.so.2
libss.so.2.0
libsysfs.so.1
libsysfs.so.1.0.3
libthread_db-1.0.so
libthread_db.so.1
libutil-2.4.so
libutil.so.1
libuuid.so.1
libuuid.so.1.2
libwrap.so.0
libwrap.so.0.7.6
libxcrypt.so.1
libxcrypt.so.1.2.4
libz.so.1
libz.so.1.2.3
Can You Narrow That Down?
• For backwards compatibility there are several
versions of many libraries.
• Many libraries are “niche” libraries and not
generally interesting.
• The most interesting (to me) are:
– libc, the standard “C” library
– libm, the math library
– libz, the compression library
– libcrypt, the encryption library
What Functions are Inside?
$ nm -DC /lib/libz.so.1.2.3 | grep -v " _" | grep " T "
00000000000017b0 T adler32
0000000000001a70 T adler32_combine
0000000000001c10 T compress
0000000000001b40 T compress2
0000000000001b20 T compressBound
0000000000001c70 T crc32
0000000000002330 T crc32_combine
0000000000004220 T deflate
00000000000058b0 T deflateBound
0000000000005f70 T deflateCopy
…
A Few Words of Warning
• The slides that you are about to see contain very
little in the way of:
–
–
–
–
Error checking.
Exception handling.
Garbage collection.
And other niceties.
• While this may be customary for sample code it is
very dangerous when working with real C code.
• You have been warned.
Calling Functions in
libc
Problem: No OS-PUTENV()
NAME
putenv - change or add an environment variable
SYNOPSIS
#include <stdlib.h>
int putenv(char *string);
DESCRIPTION
The putenv() function adds or changes the value of environment variables.
The argument string is of the form name=value. If name does not already
exist in the environment, then string is added to the environment. If name
does exist, then the value of name in the environment is changed to value.
The string pointed to by string becomes part of the environment, so altering
the string changes the environment.
RETURN VALUE
The putenv() function returns zero on success, or non-zero if an error occurs.
Calling putenv()
define variable x as integer no-undo.
procedure putenv external "/lib64/libc.so.6":
define input parameter env as character.
define return parameter x
as long.
end.
display os-getenv( "XYZZY" ).
pause.
run putenv( "XYZZY=pflugh", output x ).
display os-getenv( "XYZZY" ).
os-command value( 'echo "$XYZZY"' ).
return.
Calling putenv()
┌────────────────────┐
│?
│
└────────────────────┘
Press space bar to continue.
┌────────────────────┐
│?
pflugh
│
└────────────────────┘
Press space bar to continue.
pflugh
Press space bar to continue.
Problem: No Scientific Notation
NAME
sprintf - formatted string output
SYNOPSIS
int sprintf( char *str, const char *format, ... );
THE CONVERSION SPECIFIER
A character that specifies the type of conversion to be applied. The conversion
specifiers and their meanings are:
…
e,E The double argument is rounded and converted in the style [-]d.ddde±dd
where there is one digit before the decimal-point character and the number
of digits after it is equal to the precision; if the precision is missing, it is
taken as 6; if the precision is zero, no decimal-point character appears.
An E conversion uses the letter E (rather than e) to introduce the exponent.
The exponent always contains at least two digits; if the value is zero, the
exponent is 00.
Calling sprintf()
define variable x as integer no-undo.
define variable c as memptr no-undo.
define variable m as memptr no-undo.
procedure sprintf external "/lib64/libc.so.6":
define input-output parameter fStr as memptr.
define input parameter mask as memptr.
define input parameter arg as double.
define return parameter x
as long.
end.
set-size( c ) = 1024.
set-size( m ) = 1024.
put-string( m, 1 ) = "%1.4e".
run sprintf( input-output c, m, 0.0123, output x ).
display get-string( c, 1 ) format “x(20)”.
return.
Calling sprintf()
┌────────────────────┐
│1.2300e-02
│
└────────────────────┘
Procedure complete. Press space bar to continue.
Data Type Mapping
“C” Data Type
Procedure Parameter Definition
char
BYTE
short
SHORT
unsigned short
UNSIGNED SHORT
long
LONG
int
LONG
float
FLOAT
double
DOUBLE
char* (read-only)
CHARACTER or MEMPTR
char* (writeable)
MEMPTR
pointer to …
HANDLE TO …
Creating Your Own
Shared Library
Overview
•
•
•
•
You’ll need some “C” source code.
Compile to object code.
Run the Linker and create a Shared Library.
Test with Progress.
Problem: Sub-second PAUSE
define variable i as integer no-undo.
define variable s as integer no-undo.
s = time.
do i = 1 to 1000:
display i.
etime( yes ).
do while etime < 500: end.
end.
display string( time - s, “hh:mm:ss" ).
return.
What’s Wrong With That?
$ ps -ef | grep [_]progres
tom 21243 27534 1 17:16 pts/0 00:02:30 /usr/pro102a/bin/_progres -1
$ pro –p shortsleep.p
┌──────────────┐
│
i
│
│────── ───── │
│ 1,000 08:20 │
└──────────────┘
$ ps -ef | grep [_]progres
tom 21243 27534 8 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1
• 8 minutes and 20 seconds of CPU time!
Source Code
#include <time.h>
long shortsleep( unsigned long milliseconds )
{
struct timespec tmReq;
tmReq.tv_sec = (time_t)(milliseconds / 1000) ;
tmReq.tv_nsec = (time_t)(milliseconds % 1000) *
1000 * 1000 ;
nanosleep( &tmReq, NULL ) ;
}
Compile to Object Code
$ cc –c shortsleep.c -fPIC
$ ls
shortsleep.c
shortleep.o
$ file shortleep.o
ELF 64-bit LSB relocatable, AMD x86-64,
version 1 (SYSV), not stripped
Link to Create a Shared Lib
$ ld -shared shortsleep.o -o shortsleep.so
$ ls shortsleep*
shortsleep.c shortsleep.o
shortsleep.so
$ file shortsleep.so
shortsleep.so: ELF 64-bit LSB shared object, AMD x86-64,
version 1 (SYSV), not stripped
$ nm -DC shortsleep.so
shortsleep.so:0000000000100460
shortsleep.so:0000000000100460
shortsleep.so:0000000000100460
shortsleep.so:
shortsleep.so:0000000000000280
A
A
A
U
T
__bss_start
_edata
_end
nanosleep
shortsleep
Progress 4GL Test Harness
define variable i as integer no-undo.
define variable x as integer no-undo.
define variable s as integer no-undo.
procedure shortsleep external "./shortsleep.so":
define input parameter ms as long.
define return parameter x as long.
end.
s = time.
do i = 1 to 1000:
display i.
run shortsleep( 500, output x ).
end.
display string( time - s, “hh:mm:ss" ).
return.
Test With Progress
$ pro –p shortsleep.p
Could not open Dynamic Library: ./shortsleep.so (8013)
DLL Error : ./shortsleep.so: wrong ELF class:
ELFCLASS32 (8014)
Press space bar to continue.
$ file $DLC/bin/_progres
_progres: setuid ELF 64-bit LSB executable, AMD x86-64,
version 1 (SYSV), for GNU/Linux 2.4.0,
dynamically linked (uses shared libs),
for GNU/Linux 2.4.0, not stripped
Test Again!
$ ps -ef | grep [_]progres
tom 21243 27534 1 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1
$ pro –p shortsleep.p
┌──────────────┐
│
i
│
│────── ───── │
│ 1,000 08:20 │
└──────────────┘
$ ps -ef | grep [_]progres
tom 21243 27534 8 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1
• No measurable CPU time!
Problem: Regular Expressions
NAME
regcomp, regexec, regerror, regfree - POSIX regex functions
SYNOPSIS
#include <sys/types.h>
#include <regex.h>
int regcomp( regex_t *preg, const char *regex, int cflags );
int regexec( const regex_t *preg, const char *string, size_t nmatch,
regmatch_t pmatch[], int eflags );
size_t regerror( int errcode, const regex_t *preg, char *errbuf, size_t errbuf_size );
void regfree( regex_t *preg );
Source Code
#include <regex.h>
long regex_match( char *pattern, char *string ) {
regex_t
size_t
regmatch_t
int
xpattern;
matchDepth = 2;
pmatch[2];
offset;
if ( regcomp(&xpattern, pattern, REG_EXTENDED|REG_ICASE ) != 0 ) {
offset = -2;
} else {
if ( regexec(&xpattern, string, matchDepth, pmatch, 0 ) != 0 ) {
offset = -1; } else { offset = pmatch[1].rm_so; }
}
regfree( &xpattern );
return offset;
}
Compile and Link…
$ cc –c rx2.c -fPIC
$ ld –shared rx2.o –o rx2.so
$ file rx2.so
rx2.so: ELF 64-bit LSB shared object, AMD x86-64,
version 1 (SYSV), not stripped
$ nm -DC rx2.so
00000000001004e8 A __bss_start
00000000001004e8 A _edata
00000000001004e8 A _end
U regcomp
U regexec
0000000000000318 T regex_match
U regfree
Progress 4GL Test Harness
define variable pattern as character no-undo format "x(60)".
define variable string as character no-undo
view-as editor inner-lines 10 inner-chars 60.
define variable x as integer no-undo.
procedure regex_match external "./rx2.so":
define input parameter pattern as character.
define input parameter string as character.
define return parameter x
as long.
end.
update pattern skip string skip with side-labels.
run regex_match( pattern, string, output x ).
display x.
return.
Test!
$ pro –p rx.p
┌──────────────────────────────────────────────────────────┐
│pattern: .*(Fox).*
│
│string: The quick brown fox jumped over the lazy dog!
│
│
│
│
│
│
│
│
│
│x: 16
│
└──────────────────────────────────────────────────────────┘
Shared Library
Challenges
Which Library to Use?
procedure putenv external "/lib64/libc.so.6":
define input parameter env as character.
define return parameter x
as long.
end.
•
•
•
•
Use a variable? NOT! 
Use a pre-processor.
Use symbolic links.
Wait for 10.2B.
Variable Argument Lists
int sprintf( char *str, const char *format, ... );
• Isolate different cases in multiple .p files.
• Use wrapper libraries.
• Wait for 10.2B.
Call-Backs?
void qsort(
void *base,
size_t nmemb,
size_t size,
int(*compar)(const void *, const void *)
);
• Not supported .
Persistence
procedure putenv external "/lib64/libc.so.6" PERSISTENT:
define input parameter env as character.
define return parameter x
as long.
end.
• About 25% faster in crude testing.
• RELEASE EXTERNAL “/lib64/libc.so.6”.
MEMPTRs
define variable x as integer no-undo.
define variable c as memptr no-undo.
define variable m as memptr no-undo.
procedure sprintf external "/lib64/libc.so.6":
define input-output parameter fStr as memptr.
define input parameter mask as memptr.
define input parameter arg as double.
define return parameter x
as long.
end.
set-size( c ) = 1024.
set-size( m ) = 1024.
put-string( m, 1 ) = "%1.4e".
run sprintf( input-output c, m, 0.0123, output x ).
display get-string( c, 1 ) format “x(20)”.
set-size( c ) = 0.
set-size( m ) = 0.
return.
Dynamic
Shared Library
Call
(Special Bonus Slide!)
Dynamic Shared Lib Call
define variable LIBC as character no-undo initial "/lib64/libc.so.6".
function os-putenv returns integer ( input env as character ):
define variable hCall as handle no-undo.
define variable rVal as integer no-undo.
create call hCall.
assign
hCall:call-name = "putenv"
hCall:library = LIBC
hCall:call-type = DLL-CALL-TYPE
hCall:num-parameters = 1
hCall:return-value-dll-type = "LONG"
.
hCall:set-parameter( 1, "CHARACTER", "INPUT", env ).
hCall:invoke().
rVal = hCall:return-value.
delete object hCall.
return rVal.
end.
Conclusion
• Share code from 3rd party, non-Progress,
sources.
• Gain performance improvements from
routines written in faster languages.
• Access low-level OS features that are not
otherwise available.
• It beats the heck out of using PROBUILD.
Questions?
Thank-you!
dlopen() and Friends
NAME
dladdr, dlclose, dlerror, dlopen, dlsym, dlvsym - programming interface to
dynamic linking loader
SYNOPSIS
#include <dlfcn.h>
void *dlopen( const char *filename, int flag );
char *dlerror( void );
void *dlsym( void *handle, const char *symbol );
int dlclose( void *handle );
DESCRIPTION
The four functions dlopen(), dlsym(), dlclose(), dlerror() implement the interface
to the dynamic linking loader.