A Java File Transfer Framework (and More) Using Quartz

Jared Lynem Amway Corporation [email protected]

Hello

    I’m Jared  We’ll talk about: File transfers: requirements, reasons, concepts    A framework for file transfers A way to schedule them Other quick victories Links to resources at the end of this document  Feel free to contact me [email protected]

Requirements

   The first business data “integrations”    Moving paper files from one office to another Ditto machines! Crazy.

Offices like on Mad Men Fast forward to early digital integrations  Moving data files from one server to another Bread and butter, reliable, easy

Requirements – What?

    What – select and move files based on location, file name When – based on a schedule or as a process step Where – multiple targets, multiple sources, archiving, PCI/PII considerations How – Protocols: FTP, SFTP, FTPS, CIFS, VPN; Encryption: PGP

Requirements – Why?

  Why indeed.

Not our problem, just do as the customer requests.

 Seriously though, fight for the best design.

Reasons

  Why should we use file transfers?

     To communicate securely with external entities To interact with legacy applications Because they’re simple and easy to support When you don’t need data transformation When someone need to review the data Beware of PCI/PII requirements!

Reasons

 Why develop a framework?

   Oracle SOA Suite “FTP Adapter” doesn’t do what you might think it does.

Oracle Data Integrator FTP/SFTP setup is clunky, not easily extensible Often times, many very similar integrations are required

Concepts - Java

      Object oriented programming Native language for Weblogic Application Server Relatively easy, JDeveloper is a great IDE Much, much faster than SOA tools Widely used, plenty of knowledgeable developers Oracle: “Resistance is futile”

Concepts – Transfer Protocols

  FTP – Old ‘n reliable    Credential based login, or anonymous Not encrypted Easy to implement, widely supported SFTP – “FTP” over SSH   Encrypted transfer using Secure Shell Allows for additional authentication measures

Concepts – Transfer Protocols

  FTPS – FTP with encryption over SSL   Support for multiple authentication methods Support for public key certificates (SSL) CIFS (aka SMB) – Used by Windows   Allows for Active Directory or Kerberos authentication Handy when FTP isn’t set up on Windows servers

Concepts – Security

  Transport Level Security (See above) Data Level Security – PGP (Pretty Good Privacy)  Offers encryption in a wide variety of algorithms and strengths   Offers data compression, integrity checking, and message authentication Unbreakable!

The Framework

 Most file transfers follow a similar design: 1.

2.

3.

4.

5.

Trigger condition (sometimes) Source(s) for files Encryption/decryption/renaming (sometimes) Target(s) for files Archiving (sometimes)

The Framework - Goal

  To make these common tasks configurable via XML elements To allow custom tasks to work seamlessly with common tasks to minimize development time

The Framework – JAXB

 The XML part is easy:     Create an XML schema with elements for each step’s configuration Use JAXB (built-in to JDeveloper) to create matching Java objects Use these objects as “properties” for your step implementations Create a mapping between XML element types and Java classes to execute

The Framework – Trigger Condition

    A trigger condition decides whether the transfer should occur Typically a “trigger file” is used, placed in a directory near where the actual file will be Useful for preventing accidental transfer during file write Configurations: FTP location, authentication, file name

The Framework – Source config

   FTP location, authentication, remote directory, file names or file patterns Java supports regular expressions, so use them Not all FTP servers support glob expressions

The Framework – local work

  Encryption config   Public key Whether to delete unencrypted file afterwards (yes) Decryption config   Private key Passphrase

The Framework – Target Config

 Multiple target elements, each with:   FTP location, authentication Remote directory

The Framwork - Archiving

 Considerations    Local vs. remote Append timestamps, other clerical tasks Archiving sensitive data

The Framework – Custom task

  Create your own Java class to do some complicated work (i.e., building a file from a database query) XML configuration:   Class name Name/Value properties used by the class

Putting it All Together – Task Interface  A “Task” needs:    A definition: settings, configurations A context: what happened before?

A result: something that can be interpreted and acted upon

Task Interface – Task Definition

  Say, one of the XML elements you just defined The Task knows what to do with it

Task Interface - Context

   A way to keep track of what happened in previous steps Most obvious example is, what files are we working with?

Doesn’t need to be too complex

Task Interface – Result

 Three values    Success/Failure Whether to continue Message for grateful support person

One More Interface - Plan

  Just a way to build a list of tasks from a given XML configuration Execute them in order, checking the result after each task  Bonus points: have “Plan” implement “Task”

phew

  Hard part is over Lunch soon!

I feel ya, bro

Quartz

      Well known (among Java nerds) library for scheduling jobs Free as in speech, beer Well documented, tons of features Runs in a clustered environment Free web-app UIs exist You can pay for support if you want

Quartz – Runs stuff

 Use Quartz if:     You want to run something every five minutes You want to run something every day You want to run something on demand You want to run something every 15 minutes from noon to 2pm on the fourth day of each month and every second Tuesday

An Alternative to Quartz

 Weblogic Job Scheduler      Built-in to Weblogic =) Not portable =( Not widely used =( More complicated setup =( Documented by Oracle =(

Quartz – Concepts

     Job – interface for Java classes, with only and “execute” method. Quartz can run any class that implements Job.

Trigger – Tells the schedule when to run Groups – Group Jobs for better control Listener – Your own optional class that gets notified when things run (for logging purposes) Job Data Map – Configurations or data passed in when the Job is run.

Putting the FTP Framework in Quartz    Create a Java class that implements Job (so Quartz can run it) Schedule the Job, put your XML file in the Job Data Map so it is accessible by your new class.

Have the class set up a Plan of Tasks based on the XML file, and execute it.

Bonus - Quartz/SOA Interaction

  Quartz -> SOA  Use the javax.xml.soap libraries to create a simple web service Job  Now you can schedule your SOA composites! SOA -> Quartz    Create a simple class in your scheduler project that triggers a job on-demand in Quartz based on an input job name.

In JDeveloper, right-click and select “Create Web Service” Now you can kick-off scheduler jobs from SOA!

You’ve been a great audience

 Takeaways:    Java is your best bet for file transfers if you’re using Oracle products for integration Abstract Source/Target/Encryption tasks since they’re all pretty much the same XML is a good format for configurations since it’s extensible and easy to import to Java

Thanks OWMUG

 Takeaways:   Quartz is a powerful, free scheduling tool used by many large organizations Quartz can interact with SOA via web services, too (or anything Java can interact with- ODI, AS400, SQL, etc.)

select * from QUESTIONS

Libraries and Links (all free!)

      Quartz http://quartz-scheduler.org/ Apache Java FTP/FTPS client library http://commons.apache.org/net/ JSCH Java SFTP client library – http://www.jcraft.com/jsch/ CIFS/SMB client library http://www.samba.org/ Bouncy Castle Java PGP library http://www.bouncycastle.org/ JWatch Quartz Manager UI http://code.google.com/p/jwatch/