Transcript RETS Developer Seminar

RETS Software Development
RETS Software Development

RETS the standard

RETS transactions
– Identify issues, concerns and recommendations

Sample session workflow
File: 05_RETS_Software_Development.PPT
1
RETS Software Development
Topic Objectives
RETS Software Development

Improve understanding of the RETS
architecture

Build familiarity with the RETS document

Enable more productive RETS software
development
File: 05_RETS_Software_Development.PPT
2
RETS
Principles - 1
RETS Software Development

Stateless communications protocol

Carry state using a cookie:
– Basically a token that points to a persistent store on a
server somewhere
– RETS defines an optional cookie for this purpose

Recommendations:
– Return all cookies received
– Client applications using browsers
will need to accept cookies
File: 05_RETS_Software_Development.PPT
3
RETS
Principles - 2
RETS Software Development

Uses Http-based transport

Uses DMQL - an open query language

Supports XML and site-specific output data
formats

Supports site-specific metadata to describe
the local MLS system

Recommendation:
– Effort to support metadata produces interoperability
benefits
File: 05_RETS_Software_Development.PPT
4
RETS Document
RETS Software Development

Organization follows a typical real estate
industry workflow:
– Initiate single session
– Multiple data requests
– Terminate session

References standards produced by other
standards organizations
RETS document
can be challenging to read
File: 05_RETS_Software_Development.PPT
5
Simple Collaboration Diagram
LoginDisplay
1: Log in
RETS Software
Development
1.1: SendCredentials
LoginHandler
2: CapabilityUrls
2.2: GetMetadata
3: GetMetadata
2.1:
GetMetadataHandler
ActionUrlDisplay
MetadataCache
3.1: Metatdata Results
SearchResults
4:
SearchSelectorDisplay
4.1: PrepareSearch
SearchHandler
4.1.1.1: Resul ts
SearchDisplay
5: Search
4.1.1: DisplayResults
<<actor>>
Client
<<actor>>
RetsServer
File: 05_RETS_Software_Development.PPT
6
Login Transaction
Making the Initial Connection
RETS Software Development

Authentication
– Response with authorized resources

Types:
– Basic
– Digest using MD5 or SHA1
– SSL

Recommendations:
– Implement Digest using MD5
– SHA1 if you can
File: 05_RETS_Software_Development.PPT
7
Login Transaction
Authentication
RETS Software Development

Most common authentication type is digest
using MD5
– Somewhat tricky to implement algorithm
– Easiest to borrow the implementation from one of the
open source projects

When in doubt:
– Ask the RETS-DEV list
– Ask your software vendor
File: 05_RETS_Software_Development.PPT
8
Login Transaction Activity Diagram
RETS Client
RETS Server
RETS Software Development
>
Req uestLog in
ReceiveLog inRequest
Construct MD5
Valid header
Invalid header
ValidateMD 5
InternalActi vity
BuildSession
FetchCapabilityURL
ValidSession
<
ResponseLogin
File: 05_RETS_Software_Development.PPT
9
Login Transaction
Digest Authentication
RETS Software Development

Message Digest algorithms are one-way
functions

Easy to perform the calculation

Difficult to reverse the calculation

Both sides calculate the same message digest

Transmit only the digest

Compare your calculated digest with the
received digest
File: 05_RETS_Software_Development.PPT
10
Login Transaction
UserAgent
RETS Software Development

Application name accessing the MLS server
– All client requests MUST include this field
– This is a standard HTTP header field as defined in
RFC 2616

MLS server should validate application name
presented against list of allowable
applications maintained on a table

Application name and userid is a useful key
to a table that tracks MLS server usage
File: 05_RETS_Software_Development.PPT
11
Login Transaction
Gotchas
RETS Software Development

Using operating system libraries for MD5
can cause authentication failure

Proxies and tunnels can be a problem

Re-authenticate is part of the RFC but few
vendors enforce it

Case may matter
File: 05_RETS_Software_Development.PPT
12
Sample Login
Challenge Response Header 1
RETS Software Development
HTTP/1.1 401 Unauthorized xxx
Date: Mon, 02 Aug 2004 05:23:54 GMT
Server: WebLogic XMLX Module 8.1 SP1 Fri Jun
20 23:06:40 PDT 2003 271009 with
WWW-Authenticate: Digest
realm="[email protected]", nonce="313035",
opaque="6e"
File: 05_RETS_Software_Development.PPT
13
Sample Login
Challenge Response Header 1 cont.
RETS Software Development
Content-Length: 0
Content-Type: text/html
RETS-Version: RETS/1.0
Cache-Control: private
File: 05_RETS_Software_Development.PPT
14
Sample Login
Nonce
RETS Software Development
nonce=
"313035393731353435373630342058dd631265e3
360724c45d15f23aec7a"
File: 05_RETS_Software_Development.PPT
15
Sample Login
Opaque
RETS Software Development
opaque="6e6f742075736564"
File: 05_RETS_Software_Development.PPT
16
Sample Login
Challenge Response Header 2
RETS Software Development
HTTP/1.1 200 OK
Date: Mon, 02 Aug 2004 05:31:03 GMT
Server: WebLogic XMLX Module 8.1 SP1 Fri Jun
20 23:06:40 PDT 2003 271009 with
Content-Type: text/plain
File: 05_RETS_Software_Development.PPT
17
Sample Login
Challenge Response Header 2 cont.
RETS Software Development
Set-Cookie: RETS-Session-ID=20000439936
RETS-Version: RETS/1.0
Transfer-Encoding: Chunked
Cache-Control: private
0230
File: 05_RETS_Software_Development.PPT
18
Sample Login
Challenge Response Body 2
RETS Software Development
<RETS ReplyCode="0" ReplyText="Success">
MemberName=Mark Crawford
User=632344, NULL, NULL, 632344
Broker=National,1
MetadataVersion=1.2.0
MinMetadataVersion=1.1.7
OfficeList=National;1
TimeoutSeconds=900
File: 05_RETS_Software_Development.PPT
19
Sample Login
Challenge Response Body 2 cont.
RETS Software Development
Search=http://rets.ftc2.com:6103/search
GetObject=http://rets.ftc2.com:6103/get object
Login=http://rets.ftc2.com:6103/login
GetMetadata=http://rets.ftc2.com:6103/getmetadata
Logout=http://rets.ftc2.com:6103/logout
</RETS>
0000
File: 05_RETS_Software_Development.PPT
20
Login Transaction
Capability URL List
RETS Software Development

Response from the Login transaction includes a list
of known resources on the server
–
Each URL points to a RETS transaction type

Capability URL list may be different based on the user
and their authorization level

Required that the server return a list containing at
least Search

The URL references are not permanent locations
–
the standard allows the location to change
File: 05_RETS_Software_Development.PPT
21
Action-URL and Get Transaction
RETS Software Development

Included in the Capability URL List is an
Action-URL

Points to a human readable message
– Most useful if changed regularly; perhaps daily

Immediately following successful login, may
need to do a Get on the Action-URL
– Get Transaction is a file retrieval from the server
File: 05_RETS_Software_Development.PPT
22
Metadata
RETS Software Development

Description of the structure of the data on a
specific server

Unique to each site
– There will be common elements

Most powerful feature of the RETS
specification
Perhaps the most challenging part
of the RETS specification
File: 05_RETS_Software_Development.PPT
23
GetMetadata Transaction
Overview
RETS Software Development

Transaction to retrieve system metadata

Can select a specific level, the level and
descendants or a specific metadata ID

Formats supported are COMPACT and
Standard-XML

Metadata id value of 0
– Request is for all types at the same level – breadth

Metadata id value of *
– Request is for all types at the same level and all child
types - depth
File: 05_RETS_Software_Development.PPT
24
Metadata Format
ID
RETS Software Development

Hierarchy of types of metadata
– See Figure 11.1 on page 11-2
of the RETS specification

When referencing hierarchy as ID
– Parent: Child is the form
– Example:
Type: METADATA-UPDATE_TYPE
ID: Property: RES: Add
File: 05_RETS_Software_Development.PPT
25
Metadata Format
Versioning - 1
RETS Software Development

Hierarchy of metadata is versioned

Changes to lower level of the hierarchy
propagate upward
– Example: Changed METADATA-TABLE attribute will
propagate up through CLASS, RESOURCE and
SYSTEM
File: 05_RETS_Software_Development.PPT
26
Metadata Format
Versioning - 2
RETS Software Development

Caching of metadata is permitted

Use the login information to determine
metadata changes

Recommendation:
– If possible, get all the metadata on a change instead
of attempting to get a specific type
File: 05_RETS_Software_Development.PPT
27
Metadata Format
Version 1.5 changes
RETS Software Development

Added the Foreign Keys metadata
– Provides relationships between the offered resources
of the form parent:child

Clarified a number of issues with Metadata
Format

Recommendation:
– Use 1.5 as a reference even when implementing
1.0.1 compliant software
File: 05_RETS_Software_Development.PPT
28
Application Responses
to Metadata Changes
RETS Software Development

Ignore metadata
– Change breaks the application, intervention required
to fix

Static metadata
– Change is invisible, applications keep working,
intervention to required view change

Dynamic metadata
– Change is visible to applications without intervention
File: 05_RETS_Software_Development.PPT
29
Search Transaction
Overview
RETS Software Development

Most important RETS transaction

Can use either http GET method or http
POST method
– Example – GET method request
http://rets.server.com:6103/search?Class=RES

Issue: GET method is limited in the
character length of the request

Recommendation: POST method
File: 05_RETS_Software_Development.PPT
30
Search Transaction
Client Diagram
RETS Software Development
SearchSelectorDisplay
1: q uery selection
1.1: Http q uery request
MetadataC ache
<<actor>>
RetsServe
<<actor>>
Client
2.1: result display
2: Query response
SearchDisplay
File: 05_RETS_Software_Development.PPT
31
Search Transaction
Server Diagram
RETS Software Development
Server side view.
Implemented as MVC
<<controller>>
SearchHandler
1.1:
<<model>>
RepositoryModel
1:
<<actor>>
Client
2:
1.1.1:
<<view>>
SearchResults
File: 05_RETS_Software_Development.PPT
32
Sample Search
Search Request Header
RETS Software Development
HTTP/1.1 200 OK
Date: Fri, 31 Jul 2004 23:22:43 GMT
Server: WebLogic XMLX Module 8.1 SP1 Fri Jun
20 23:06:40 PDT 2003 271009 with
Content-Type: text/xml
File: 05_RETS_Software_Development.PPT
33
Sample Search
Search Request Header cont.
RETS Software Development
Set-Cookie: RETS-Session-ID=20000439934
Set-Cookie: RETS-Request-ID="42"
RETS-Version: RETS/1.0
Transfer-Encoding: Chunked
Cache-Control: private
File: 05_RETS_Software_Development.PPT
34
Sample Search
Search Response Body
RETS Software Development
0fe8
<?xml version="1.0" ?>
<!DOCTYPE RETS SYSTEM
"http://www.ftc2.com/rets/dtd/RETS20010812.dtd">
<RETS ReplyCode="0" ReplyText="Success">
<COUNT Records="2507" />
File: 05_RETS_Software_Development.PPT
35
Sample Search
Search Response Body cont.
RETS Software Development
<REData>
<Properties>
<ResidentialProperty>
<Building>
<BuildingDescription>
<Stories>1</Stories>
<PropertyCondition>Shows Well</PropertyCondition>
RETS Software
Development36
File: 05_RETS_Software_Development.PPT
Search Transaction
SearchType Required
RETS Software Development

SearchType: The resource id to search on
– Example – SearchType=Property
File: 05_RETS_Software_Development.PPT
37
Search Transaction
Class Required
RETS Software Development

The class of data within the resource to
search against
– Example – Class=RES

Issues:
– Some of the data may be in system specific Class
names rather than the Class name that represents an
XML Standard Name
File: 05_RETS_Software_Development.PPT
38
Search Transaction
QueryType Required
RETS Software Development

QueryType: The query language
– Only two types: DMQL and DMQL2
– Example – QueryType=DMQL2

Issues:
– Mainly stylistic difference, DMQL2 supports
quoted text
– Not suitable for highly complex searches
– Not possible to perform cross Class searches
Example – NOT Residential and Lot Land
File: 05_RETS_Software_Development.PPT
39
Search Transaction
Query Required
RETS Software Development

Query:
– Key – value pairs
– Uses field name based on metadata or on standard
names
– Example
Query=(ListPrice=200000-300000),(Status=|A)

Issues:
– Lookup type queries can execute for a long time
File: 05_RETS_Software_Development.PPT
40
Search Transaction
Optional Arguments
RETS Software Development

Optional argument are just that

A vendor MAY choose to implement none,
one or more

Software is RETS compliant without any of
the optional arguments implemented

Issue:
– Mismatch between client & server feature sets

Recommendation:
– Do NOT assume, verify the feature set
File: 05_RETS_Software_Development.PPT
41
Search Transaction
Count
RETS Software Development

Default is no count

Behavior outside of given three values not
defined
File: 05_RETS_Software_Development.PPT
42
Search Transaction
Format
RETS Software Development

Return records in XML or column
header/row values format
– XML is more verbose, but contains structure
information
– Some issues with site specific XML – clarification in
RETS needed

Default is Standard-XML

Issue:
– Site specific XML is not defined in the current
standard
File: 05_RETS_Software_Development.PPT
43
Search Transaction
Format Sidebar: XML - 1
RETS Software Development

XML is a meta-language that defines a
markup language

RETS is an example of an XML application

An application consists of a DTD and
documents that obey the rules embodied in
the DTD

The XML DTD is a document that describes
the structure and elements of an XML
document
File: 05_RETS_Software_Development.PPT
44
Search Transaction
Format Sidebar: XML - 2
RETS Software Development

The XML document takes data and applies
the DTD

RETS will eventually move from a DTD
based vocabulary to a Schema-based
vocabulary
File: 05_RETS_Software_Development.PPT
45
Search Transaction
Format Sidebar: Compact Formats
RETS Software Development

Provides a delimited column description

Provides delimited data in the same position
as the column description

Field delimiter is chosen by the server

Gotcha:
– Delimiter tag is a two character representation of the
ASCII character
– Delimiter value may collide with data values in earlier
implementations
File: 05_RETS_Software_Development.PPT
46
Search Transaction
Limit
RETS Software Development

Limits the number of records returned:
– The NONE well known name requests a suspension
of limit
– Behavior is implementation dependent
– Requesting a Limit should make the query faster
– May still search for all records but only transmit
records to a Limit
File: 05_RETS_Software_Development.PPT
47
Search Transaction
Offset
RETS Software Development

Return records starting from the specified
point in the result set:
– One method to handle very large data pulls
– Given that the underlying transport mechanism is not
guaranteed, very large results can fail with transport
errors
– Given that the data volumes can be very large, out of
memory errors can occur
File: 05_RETS_Software_Development.PPT
48
Search Transaction
StandardNames
RETS Software Development

Indicates that the query uses the standard
names or the system names:
– Servers are not required to implement standard
names to meet RETS compliance criteria
– Standard Names are intended to be common across
all RETS systems
Refining the list of Standard Names
is an ongoing process
File: 05_RETS_Software_Development.PPT
49
Search Transaction
Restricted Indicator
RETS Software Development

Certain fields may have restricted visibility:
– Client may request fields that are restricted be
replaced by a specific character string
– Example - compensation amount field
– Client may request that this amount be masked by
###
– Server default is to return a null value
File: 05_RETS_Software_Development.PPT
50
Search Transaction
Select
RETS Software Development

Provides the ability to create a subset of the
fields accessible to the client

Default is to return all accessible fields

Gotcha: Select field names are not specified
as either the system names or the standard
names
– Ambiguity in RETS standard
– Working assumption is that the server will understand
both
File: 05_RETS_Software_Development.PPT
51
Search Transaction
DMQL
RETS Software Development

Defines the search criteria for the search result set

Based on a BNF grammar to provide a neutral
representation

RETS Server parses the DMQL and creates a
server specific query

Trade off between domain specific query forms
and more complex query grammars like SQL-99

Issues around the lexical parsing of the grammar
File: 05_RETS_Software_Development.PPT
52
Update Transaction
Overview
RETS Software Development

Allows addition of records to the RETS
system

Supports Auto-population of fields

References only by System Name, not
Standard Name

Validation based on rules described in the
metadata

Single record per update – no bulk update
File: 05_RETS_Software_Development.PPT
53
Update Transaction
Validation
RETS Software Development

Basic validation on data types

Additional validation for single lookup,
multi-lookup and range

Additional validation for Boolean expression

Additional validation using external
validation

Certain validation steps may be server only
File: 05_RETS_Software_Development.PPT
54
Update Transaction
Error Block
RETS Software Development

Validation tests that fail result in an error
block that provides some information to
assist in correction

One error block per failed validation test
– Means a single field may trigger multiple errors
File: 05_RETS_Software_Development.PPT
55
Update Transaction
Gotchas and Recommendations
RETS Software Development

Gotcha:
– RETS 1.7d1 adds a third state to the validate-flag “2”
meaning “validate the record but do not save it”
– Be aware of this coming specification change

Recommendation:
– Client-side validation is fast fail – use it
File: 05_RETS_Software_Development.PPT
56
GetObject Transaction
Overview
RETS Software Development

Provides a way for users to get objects
defined in the metadata

Main use is to return pictures although there
are other media types

Returns the object in the appropriate mime
type format - this may be a binary format

Recommendation:
– Ordering of the object ID changed in 1.5D2
– Implement using the most recent RETS version
File: 05_RETS_Software_Development.PPT
57
ChangePassword Transaction
Overview
RETS Software Development

Allows a user to change their password

Some ambiguity in the 1.5 and earlier
standard

Cryptographically weak

Recommendations:
– Be very careful when implementing
– Use the 1.7 standard addition as described in RCP
041
File: 05_RETS_Software_Development.PPT
58
Logout Transaction
Overview
RETS Software Development

Courtesy transaction

Communicates that a user has finished the
session

A mechanism for the server to:
– dispose of any local resources
– communicate session and billing information

Recommendation:
– Clients should log out
– Servers should be aware that clients may never log out
File: 05_RETS_Software_Development.PPT
59
Extensions to the RETS Standard
RETS Software Development

RETS allows vendors and sites to extend the
standard to meet the specific needs of a site

Additional Capability List URLs have no
suggested form

Additional parameters to an existing
transaction and additional metadata have
should have prefix ‘X’
– Example: Add sort direction to Search: XSortDir=Asc

Almost everything else is covered by
metadata
File: 05_RETS_Software_Development.PPT
60
Role of Parser Generators
RETS Software Development

Used to produce parsers for:
– DMQL
– Validation Expressions

DMQL Java code will be used by RETS
server to generate SQL from DMQL

Validation expression Java code will be used
by RETS server to execute the validation
expression
File: 05_RETS_Software_Development.PPT
61
Benefits for using a
Parser Generator
RETS Software Development

Improves productivity for software developers
– Not easy to write a good parser
– Relatively easy to write the grammar input file for the
parser generator

Avoids having to develop a parser
– Difficult, demanding, time-consuming

Makes use of available parser generators
– Examples: javacc, ANTLR, lexx

Produces more robust Java source code
File: 05_RETS_Software_Development.PPT
62
Generating Java Source Code
from a BNF Grammar File
RETS Software Development
BNF Grammar File
(produced by
developer)
BNF grammar for DMQL
- specified in RETS specification
- see example on next two slides
Grammar file syntax
- specified in parser generator specification
- see example on next two slides
Javacc
Parser Generator
Java
Source code
File: 05_RETS_Software_Development.PPT
63
BNF Grammar File Fragment
DMQL Javacc Input File - 1
RETS Software Development
PARSER_BEGIN(DMQL)
// The DMQL Javacc file which is used to generate the the primary class
for parsing DMQL queries
public class DMQL
{<Your Java code here>}
PARSER_END(DMQL)
// Parse the DMQL statement.
void Input() :
{< Java attributes>}
{<Javacc expressions & your Java code>}
// Match the the DMQL Query at the highest level.
void Query() :
{< Java attributes>}
File: 05_RETS_Software_Development.PPT
64
BNF Grammar File Fragment
DMQL Javacc Input File - 2
RETS Software Development
{ Search() (<EOF> | <TERMINATOR>)}
... BNF and Javacode Productions and additional Java code for
processing tokens for parsing DMQL based on the RETS DMQL
Grammar...
// Terminals (Tokens)
TOKEN : /* The DMQL Tokens (See RETS Specification) */
{
< PLUS: "+" >
| < SINGLE_CHAR: "?" >
| < MULTI_CHAR: "*" >
.
| < OR: "|" >
}
File: 05_RETS_Software_Development.PPT
65
Application of
Parser Output
RETS Software Development
Input rules for update
(produced by
MLS)
Input DMQL
(produced by
client software)
Java Validation
Expression Code
to evaluate Rules
Java code to
process DMQL
SQL
Parser
Generator
Output
Validation
Outcome
File: 05_RETS_Software_Development.PPT
66
Resources for
Parser Generators
RETS Software Development

ANTLR, ANother Tool for Language Recognition
– http://www.antlr.org

Java Compiler Compilertm – JavaCCtm
– https://javacc.dev.java.net

Lexers and Parser Generators
– http://catalog.compilertools.net/lexparse.html
File: 05_RETS_Software_Development.PPT
67
Topic
Summary
RETS Software Development

RETS provides a rich set of features for
implementing a service or client

Careful design will result in a successful project

Work with the vendor or customer to ensure that
the correct features are built

Read the RETS document and watch for forward
and reverse references

Use RETS 1.5 as a
supplement to RETS 1.0.1
– it includes many clarifications
File: 05_RETS_Software_Development.PPT
68
Software
Quality
RETS Software Development
File: 05_RETS_Software_Development.PPT
69
Top 10 signs Technology
Has Taken Over Your Life
RETS Software Development
10. You need to fill out a form that must be typewritten, but you can't because
there isn't one typewriter in your house - only computers with laser printers.
9.
You would rather get more dots per inch than miles per gallon.
8.
You rotate your screen savers more frequently than your automobile tires.
7.
You think of the gadgets in your office as "friends," but you forget to send
your father a birthday card.
6.
You know Bill Gates' e-mail address, but you have to look up your own social
security number.
5.
On vacation, you are reading a computer manual and turning the pages faster
than everyone else who is reading John Grisham novels.
4.
The thought that a CD could refer to finance or music never enters your mind.
3.
Your letterhead lists a phone number, a fax number, e-mail addresses for two
online services, and your web site address but no street address.
2.
You have never sat through an entire movie without having at least one device
on your body beep or buzz.
1.
You sign Christmas cards by putting :-) next to your signature.
File: 05_RETS_Software_Development.PPT
70
Typical
Session Workflow
Login
RETS Software Development
Update
Metadata
?
Get Metadata
Data about
Offices,
Agents,
Listings
Search
Get
Objects
?
Data about
available
server
data
Get Object
Pictures,
Video
Documents
Logout
File: 05_RETS_Software_Development.PPT
71