Transcript Rolling the Process

Dyalog’09
Technical Road Map ’09
Morten Kromberg
Dyalog’09 – Princeton, NJ
Agenda
• Introducing Version 12.1
– Demonstration of some Key Features
• The Next Release Cycle
– Version 13.0
– Conga 2.1, Encryption & Encoding Library
• “APL#” - A new APL interpreter to complement Dyalog APL
• Cirrus or Cumulo-Nimbus?
• Eagle’s Eye View of Curves in the Road
– Recommendations on preparing for the future
Technical Road Map
Dyalog’09 - Princeton
3
Version 12.1 – Overview
• It was supposed to be a “Year of Consolidation”
• Let the new QA manager take charge
• Build, Install & QA on every platform, every night
• Find and Kill more bugs, Speed things up a bit
• ... But we managed to sneak a few new things into
v12.1 
Technical Road Map
Dyalog’09 - Princeton
4
Version 12.1 – Overview
• Faster on Single Core, Some Multi-Core Features
• Editor Enhancements
• Fast XML-to-Array Conversion Tool (⎕XML)
•
•
•
•
•
•
•
•
Microsoft.Net components embeddable in Dyalog GUI
User Command Processor for “Tool Delivery”
IPv6 Support with Conga v2.0
Journaling Component Files: Protection against O/S Crashes
New Developer Tools: SAWS, MildServer, LoadData, GRAPL
Applications deployable as single .exe file (interpreter+ws)
Improved Keyboard and Terminal Support
I-Beam (⌶) allows experimental & undocumented features
Technical Road Map
Dyalog’09 - Princeton
5
V12.1 Core Performance
v10.1 ms v12.1 ms
Factor
Boolean Scans and Reductions Highly Optimized
mat←1000 1000 rand 2
+⌿mat
vec←1000000 rand 2
≠\vec
5200
51000
680
830
7,6
61,4
“Set Functions” (⍳ ∊ ∪ ∩ ~) Optimized for “One Small Argument”
vec←1000000 rand 100
Transpose Rewritten
vec~5
33000
6500
(vec≠5)/vec
11000
7200
5,1
1,5
⍉mat
32000
3
23000
~0
1,4
11,3
4125
129
31,9
23000
59000
240
230
95,8
256,5
⍉vec
Indexing Uses Less Memory
i j←1000?¨1000 1000
mat[i;] ⍝ (kb used)
Indexing is also (often) Faster
data←mat[i;]
mat[j;]←data
Technical Road Map
Dyalog’09 - Princeton
6
V12.1 Core Performance
v10.1 ms
v12.1 ms
Factor
500000
180000
2,8
460000
480000
85000
34000
6300
16000
13,5
76,2
5,3
8
2
3,7
Grade Uses New (lexicographic) Algorithm
⍋vec
vec←1e6 rand 1e9
For “Small Range” Data, Even Faster
⍋vec
vec←1e6 rand 10000
“Sort” is now an idiom
{⍵[⍋⍵]}vec
mat←100000 4 rand 10000
{⍵[⍋⍵;]}mat
Basic arithmetic (+ - ×) has been fully rewritten in assembly:
Simple Arith uses “SSE” instructions where possible
a b←1000∘rand¨50 ⍝ 1-byte ints a+b
More Optimized Cross-Type Operations
a b←1 0.001×(1000 1000)∘rand¨50 1000000 ⍝ a: 1-byte ints, b:floats
a×b
16000
9000
1,8
16000
7300
2,2
Matrix iota idiom extended to numeric arrays
data←10000 10 rand 1E6
Technical Road Map
data{(↓⍺)⍳↓⍵}data
Dyalog’09 - Princeton
7
Multi-Core Support
• Most Scalar Dyadic Functions on Floating-Point
arguments will use multiple cores (configurable)
• But not + and -:
Function
2 cores
4 cores
+
12%
12%
-
12%
12%
×
12%
12%
÷
45-50%
70-75%
GCD/LCM
45-50%
70-75%
Technical Road Map
Dyalog’09 - Princeton
8
Parallel Each Operator
• A user-defined operator called PEACH is included with
v12.1. It uses Conga to orchestrate multiple cores.
• [Demo]
• Parallel Each on a single machine works well with
algorithms that have a good CPU:Memory ratio
• “Distributed Each” works for any kind of calculation if
transfer or arguments and results not too expensive
Technical Road Map
Dyalog’09 - Princeton
9
Performance Case Study
• Visited company in Bruxelles doing “actuarial
calculations”
• V12.1 gave a 20% speedup “out of the box”
• We found 20% more by doing “hot spot” analysis for a
couple of hours
• An 8 core machine (2xQuad) gave a 5x speedup
using “PEACH” on the inner loop of the application
• Summary: 7-8x faster. 1 Hour calculation takes <10
minutes. Customer is pleased .
Technical Road Map
Dyalog’09 - Princeton
10
User Command Processor
• Same idea as in APL*PLUS, but based on Unicode text files as
source for user commands
• A “User command” consists of a single Unicode Text file, all you
need to do is drop it in a particular folder (no need to restart APL)
• Makes it *VERY* easy to share development tools
]display 2 2↑3 3⍴'UCMDS' 'Rock!'
┌→────────────────┐
↓ ┌→────┐ ┌→────┐ │
│ │UCMDS│ │Rock!│ │
│ └─────┘ └─────┘ │
│ ┌→────┐ ┌→────┐ │
│ │Rock!│ │UCMDS│ │
│ └─────┘ └─────┘ │
└∊────────────────┘
Technical Road Map
Dyalog’09 - Princeton
11
IDE Enhancements
• The Editor has been completely rewritten to better
support editing large classes and functions
• Outlining, Collapsible Regions, “Refactoring”
• John Daintree will show you all the new features after
lunch!
Technical Road Map
Dyalog’09 - Princeton
12
Fast XML Transformations
• Converts XML text to (&from) an array:
[;1] Level of nesting
[;2] Element Name
[;3] Element Data
[;4] Attributes
[;5] Node type (0=Empty Tag, 1=Text only, 3=Has
Children, etc…)
• [;4] not required when generating XML, [;5] ignored
Technical Road Map
Dyalog’09 - Princeton
13
Integrated MS.Net GUI
Technical Road Map
Dyalog APL
14
Version 12.1 – Overview
• Faster on Single Core, Some Multi-Core Features
• Editor Enhancements
• Fast XML-to-Array Conversion Tool (⎕XML)
•
•
•
•
•
•
•
•
Microsoft.Net components embeddable in Dyalog GUI
User Command Processor for “Tool Delivery”
IPv6 Support with Conga v2.0
Journaling Component Files: Protection against O/S Crashes
New Developer Tools: SAWS, MildServer, LoadData, GRAPL
Applications deployable as single .exe file (interpreter+ws)
Improved Keyboard and Terminal Support
I-Beam (⌶) allows experimental & undocumented features
Technical Road Map
Dyalog’09 - Princeton
15
v12.1 Timeframe
• Complete except for packaging and a rapidly
diminishing number of bugs in the new editor
• Target for General Availability on Windows+AIX is
August/September 2009 (was “before” Dyalog’09)
• Linux and PDA before year end
• It isn’t too late to join the Beta-Test programme: Write to
[email protected] and explain what you would like to test…
• Monitor http://www.dyalog.com or twitter.com/dyalogapl for
notifications
Technical Road Map
Dyalog’09 - Princeton
16
Status – Spring’09
Dyalog APL
17
The Next Release Cycle
• Conga 2.1
• Version 13.0
Technical Road Map
Dyalog’09 - Princeton
18
Conga 2.1
• Support for More Certificate Types:
– We’ve had help from Peter-Michael Hager
– Encrypted Certificates
– The Windows Certificate Store (allows use
of SmartCards and other “Tokens” that
interface with this)
• General available early in 2010
Technical Road Map
Dyalog’09 - Princeton
19
Likely Features for ”v13”
• More Performance and QA work
– Continued focus on multi-core, cache optimization and other
aspects of modern chip architectures
– “Industrial Strength” local and distributed PEACH
• Remove “2 billion element” limit from 64-bit version
• New Data Types:
– 128-bit Decimal Floating Point (IEEE-754-2008) for banking
applications
– Complex Numbers to attract educational users
– Rational Numbers / Unlimited Precision Integers can wait
– We have decided we do NOT NEED 64-bit Integers
Technical Road Map
Dyalog’09 - Princeton
20
Likely Features for ”v13”
• Improve and rationalise internal developer tools:
• ⎕DMX: Diagnostic Message – eXtended
– Provide more detailed information about errors
• ⎕MONITOR: Measure CPU and Elapsed Time with
higher precision, easier to use, handle recursion
• ⎕STACK, ⎕NSI to return “refs”
• ⎕NC, ⎕NL… Rationalise “Metadata” or “Reflection”
tools:
Technical Road Map
Dyalog’09 - Princeton
21
More Likely v13 Features
• Support for Unicode source files built-in to interpreter
– Load, Save and direct Edit of Unicode Text files
• General-purpose Encryption Library (Hashing,
Symmetrical and Asymmetrical encryption)
• Target availability for v13 in “roughly 12 months”
Technical Road Map
Dyalog’09 - Princeton
22
Other Potential Projects
• SQAPL Server: Allow an APL application to appear to
be a Relational Database Engine (with ODBC, ADO,
ADO.NET and JDBC drivers)
– (This will not be a free product due to 3rd party licensing)
• Remote IDE for DyalogAPL on all platforms
• “Mono” Bridge – use .Net Framework under Linux
• Provide More Developer Tools – as part of the APL
Library. Cultivate an “Open Source” code repository,
Technical Road Map
Dyalog’09 - Princeton
23
”APL #”
• We have started a research project, to produce a
completely new, ”Fully Managed” APL Interpreter
• We will publish a specification and discuss it, first with
major customers and then ”in public”
• Will be fully integrated with Microsoft.Net – but the
core language syntax will be independent of .Net
Technical Road Map
Dyalog’09 - Princeton
24
APL # - Benefits
• Applications written in APL# will be deployable in a
”secure” fashion over the internet, and in other
scenarios where ”managed code” is a requirement or
provides a significant advantage
• For example, you will be able to write web applications
with APL code executing in the browser to provide a
”rich client” experience
Technical Road Map
Dyalog’09 - Princeton L
25
APL # - ”Likely Characteristics”
• Full set of APL primitives, operators and other core
language features
• Designed as a tool for APL thinkers – but comfortable
for software engineers
• ONE BIG difference: No ”semiglobals” (a function
cannot see the locals of its calling environment)
– There WILL be true globals
– And the equivalent of namespaces will provide a ”reasonable”
migration path
Technical Road Map
Dyalog’09 - Princeton
26
APL # - ”Likely Characteristics”
• WILL PROBABLY NOT HAVE:
– Auxiliary Processors, ⎕ML, ⎕OR, ⎕SM, ⎕RTL, ⎕SD,
⎕SR, ⎕NTIE, ⎕ARBIN, ⎕W*, ⎕AI, ⎕SVO, ⎕DQ,
⎕D..
– (Library functions for which a perfectly good equivalent exists
in the .Net framework)
– Saved workspaces will not be able to have an )SI stack
• First versions may not be able )SAVE workspaces at all
• WILL PROBABLY HAVE:
– ⎕CT, ⎕IO (jury is still out), and a small set of carefully
selected library functions, Component Files, ⎕XML (etc)
which provide unique value to APL programmers
Technical Road Map
Dyalog’09 - Princeton
27
”Dyalog” vs ”APL #”
• APL# is a complimentary product which allows the use
of APL in places where it can not be deployed today
• It is useful to organizations who need managed code,
or use Visual Studio / C# or are otherwise heavily
invested in Microsoft.Net
• The rate of development of the core product will
continue to accelerate
• The two products will inter-operate seamlessly
Technical Road Map
Dyalog’09 - Princeton
28
Dyalog → APL # ”Interop”
• APL# assemblies can be called by Dyalog APL (as
any other .NET assembly)
• Dyalog and APL# will (eventually) be able to share
Component Files and communicate (many) APL
arrays via TCP/IP
– But not saved workspaces
• Selected New Language Features in APL# will be
ported to Dyalog APL
– Will allow the writing of code which can be used in both
environments
Technical Road Map
Dyalog’09 - Princeton
29
Timelines
• APL# is still being designed, but see
C:\aplsharp\silverlight\aplsharp.html
• We will start reviewing the design externally this winter
(come to APL2010 to heckle)
• A prototype may be available at APL’2010 – earliest
”General Availability” in 2011.
• Pressure (and funding ) from customers could make
us more ambitious about the features, but probably
not the timing, of the first release
Technical Road Map
Dyalog’09 - Princeton
30
Cirrus or Cumulo-Nimbus?
• A good home for ”Vapourware”?
• Dave Thomas is about to teach us how
to recognize and use different forms of
clouds…
Technical Road Map
Dyalog’09 - Princeton
31
Amazon Web Services (AWS)
• A ”mature” provider of ”web and computing
infrastructure” (since 2006)
• You provide a virtual machine image and Amazon
”scales it infinitely” at reasonable cost
• We are investigating providing tools to support EC2
(Elastic Compute Cloud), S3 (Simple Storage Service)
• Goal is to provide recipes, or pre-configured ”Amazon
Machine Images” to build upon
• We have a lot to learn, but hope to wrap our new
knowledge up nicely for Dyalog users
• No timeline... Talk to us if you have a potential app.
Technical Road Map
Dyalog’09 - Princeton
32
Thoughts about
the Road Ahead
Technical Road Map
Dyalog’09 - Princeton
33
Looking Back Very Briefly
Version 11.0 (Autumn 2006)
• Object Orientation – Classes
– New Tool of Thought
– Significantly Improved Integration with .Net
– Source code management using standard tools
• 64-bit Version - LARGE Workspaces
• Core Language Extensions
– A number of new primitives: ⍣ ⌷ ∧ ∨
Technical Road Map
Dyalog’09 - Princeton
34
Looking Back: v12.0
First Release (Win32) in February 2008
• Main Features:
– Unicode Support
– Journaling Component Files
– Simplified, Secure Communications Toolkit
(“Conga”)
– Built-In Charting and Document Creation
– Language Bar and On-Screen Keyboard
Technical Road Map
Dyalog’09 - Princeton
35
Looking ”Back”: v12.1
October 2009, Key Features
• Faster, more Robust
• Multi-Core Support
• Enhanced Editor
• XML, Web Services
• Tutorials and Tools – and more to come
Technical Road Map
Dyalog’09 - Princeton
36
To Come ...
•
•
•
•
•
New “Commercial” and “Technical” Data Types
More Speed – Especially On Multiple Cores
Encryption and Other Web Enabling Tools
Managed Code Interpreter
More Ways to Run APL as a Service across Multiple
Platforms (ODBC Data Source etc)
Technical Road Map
Dyalog’09 - Princeton
37
Recommendations
• 64-bit is Becoming Common, even on Windows
– Good news: Porting to 64-bit requires few code changes (once
all the other components you need are available in 64-bit)
• Unicode
– Please investigate what it will take to migrate
– No hurry – but eventually, we will start adding new
features in Unicode only (we’ll need more symbols )
• Multi (at least 2)-core machines are everywhere
– Look at PEACH and other forms of ”multicore” splits
Technical Road Map
Dyalog’09 - Princeton
38
Recommendations
• To prepare for the Web or Cloud - think about
addding:
– Web Front Ends using ASP.NET, MildServer etc
– Web Services using ASP.NET or SAWS
– An ODBC or ADO frontend using the SQAPL Server?
• Take a look at Source Code in Unicode Text Files
– Easier sharing and management
– Access to many excellent (often free) tools!
Technical Road Map
Dyalog’09 - Princeton
39
Other Talks by Devt. Team...
Today (Monday)
14:15
John Daintree: The Version 12.1 IDE
16:15
Richard Smith: ⎕XML, Journaled Files Phase II
16:15
Roger Hui: Complex Numbers Implementation Stories
19:30
John Scholes: Complex Numbers
Tomorrow (Tuesday)
09:00
Morten Kromberg: An ODBC Driver for APL
09:00
John Scholes: Session Whizbangs
Wednesday (Courses)
14:00
Dan Baronet: Source Code Mgt using SALT and SubVersion
14:00
John Daintree: Using the Microsoft.Net framework
14:00
Morten Kromberg: Multicore / Application Tuning Workshop
Technical Road Map
Dyalog’09 - Princeton
40
Conclusion
• We’ve been keeping busy!
• We’re starting to enjoy the benefits of being a
significantly larger team.
• We hope that you will enjoy some of the new
Features - and Products - that are heading your
way!
• Thank You For Listening!
Technical Road Map
Dyalog’09 - Princeton
41
Other Talks by Devt. Team...
Today (Monday)
14:15
John Daintree: The Version 12.1 IDE
16:15
Richard Smith: ⎕XML, Journaled Files Phase II
16:15
Roger Hui: Complex Numbers Implementation Stories
19:30
John Scholes: Complex Numbers
Tomorrow (Tuesday)
09:00
Morten Kromberg: An ODBC Driver for APL
09:00
John Scholes: Session Whizbangs
Wednesday (Courses)
14:00
Dan Baronet: Source Code Mgt using SALT and SubVersion
14:00
John Daintree: Using the Microsoft.Net framework
14:00
Morten Kromberg: Multicore / Application Tuning Workshop
Technical Road Map
Dyalog’09 - Princeton
42