ProjectWise 101 – Chapter 9 Document Indexing

Gary Cochrane – Technical Director Geospatial Sales – North America

Introduction

• ProjectWise Document Indexing – Really means three things • Full Text Indexing, in support of full text searching • Thumbnail Extraction • Document Property Extraction – We won’t cover this one in PW101 – See Bentley Institute PW Admin course guide for this

Full Text Indexing

• We did not write the engine for this – But elected to use the one Microsoft provides – – • Included with every copy of Windows That engine is called the MS Indexing Service • And it was installed in the VM as an optional Windows component Microsoft indexes the following file formats • MSWord, Excel, PPT, HTML, XML, TXT

Pre-installed in VM

      ProjectWise Integration Server ProjectWise Orchestration Framework MicroStation V8i-SS1 Supported Database Engine Microsoft Message Queuing Service Microsoft Indexing Service Microsoft .NET Framework 2.0

Windows Server 2003 with SP2

Extending the MS Index Service

• Microsoft provides an SDK for third parties to extend the Indexing service – So the Indexing service will know how to “filter” files from that vendor • For instance, Adobe provides an “iFilter” that teaches the MS Index Service how to extract text from a PDF file • The Adobe PDF iFilter is installed with Acrobat Reader V9x

Indexing Overview

• Within PW, Indexing consists of: – Scheduling – • A process that wakes up, checks for new, (or modified files), adds them to the Copy-out queue, and goes back to sleep Copy-out • Copy the file from the Storage Area, to the machine running the Indexing Service. Then add file to the extraction queue.

• Remember, files may be stored on multiple servers • Also, in large installations, a machine may be dedicated to indexing

Indexing Overview – Part II

• Overview – continued – Extraction – • This process gets the text from the file and adds it to the MS Index catalog. Then adds the file to the Update queue Update • This process sets the flag on the file (in the PW database) that says it is “done” • New files are added with the flag set to “undone” • Check-out/in causes the flag to be set to “undone”

A note on “done”

• Done does not necessarily mean it was successful – It means the file has been processed – • In other words, what happens if an unknown file (Ex: an Autocad file) is sent to the Indexing Service?

The file is attempted… – • And the indexing service says, “I don’t know how to extract text from this file” There would be no point in trying the file again • So it is marked as “done”, even when unsuccessful

MicroStation and AutoCAD

• ProjectWise provides a mechanism to index the text from these file types – Instead of writing an iFilter, Bentley elected to: – • Copy-out the file • Run MicroStation in the background, extract all the text, and write it to an XML file • Send the XML file to the Indexing Engine Since MicroStation can parse DWG as well… • Then this method saved us from having to write two iFilters

Summary

• So within ProjectWise, we index: – Word, PPT, Excel, XML, HTML, TXT – – Adobe PDF DGN, & DWG • More good news – iFilters can be found for many file formats • Some free, and some for purchase

PW Orchestration Framework

• Remember when we installed this?

– PWOF is responsible for managing batch processes for ProjectWise – • This includes all those processes discussed on the previous slides For Full Text Indexing, that means • Scheduler process, Copy-out process, Extraction process, Updater process, and the MicroStation instance running in the background

Lab 1a

• PW Orchestration Framework – Start the Windows Task Manager – – – • Hint: Right-click on empty part of Taskbar Examine memory usage • On the Performance tab Switch to Processes tab • Sort by Mem Usage column (descending) • Look for ustation.exe

• Look for DmsAfpEngine(s) Lots of memory consumed here…

Lab 1b

• Now open Services dialog – Remember “gears” icon on Quick-Launch – – – • Locate PW Orchestration Framework service Select the PW OF service, and choose> Stop • Watch memory usage in Task Manager For remainder of exercise, we need PWOF running • So start it back up now • Note PWOF is configured for automatic startup – It will run each time machine is booted Close Services and Task Manager

Lab 2a

• Open PW Administrator – Log in as> adminpw – – Drill down to: • Document Processors> Full Text Indexing Right-click, choose> Properties

Lab 2b - Full Text Indexing

Accept defaut, unless Indexing is to be run on another machine Turn on adminpw adminpw Set to 60

Lab 2c - Full Text Indexing

Enable all times in the schedule Set to 2

Lab 2d

• Switch to File Type Associations tab – Press> Add – • In the Extension field, enter> DWG • In the bottom field, enter> DGN – So that DWG files are processed as if they were DGN Press> OK

Lab 2e

Lab 2f

• Still on the File Type Associations tab – Again, press> Add – • In the Extension field, enter> itiff • In the bottom, enable> Do not process these documents – You can’t extract text from a raster so this prevents wasted file transfers Press> OK • Press OK again – To close the Full Text Indexing Properties

Lab 2g

• Open Task Manager again – Switch to Performance tab – • Within 2 minutes, you should see heavy CPU usage • Memory usage will also go up Up to 60 documents will be indexed in the first pass • If there are more than 60 documents to be done, then they will be queued in the next pass – 2 minutes from now

Analysis

• All documents will eventually be processed – When done, the index will be ready for fast full text searches • Once the indexer has caught up, future load will be lighter due to only processing incremental documents

Lab 3a

• When done, close Task Manager, open PW Explorer – Log in as user1 • From the main tool box, select> Find Documents – Binocular icon • Change to Full Text tab – Enter Look For> detail • Press OK to start search – Then Close the Search dialog • Your results should include: DGN’s, DWG’s, and PDF’s

Lab 3b

• Browse to: – User1/Document Indexing/MS-SHT • These files were not successful because they have an unknown extension • But they were attempted, and flagged as done • Return to PW Administrator – Select datasource name (pwdemo) – • Right-click, choose> Properties • Change to Statistics tab • Choose Refresh • Review Full Text Statistics Close dialog

Lab 3c

• While still in PW Administrator – Open Full Text Indexing Properties again – – • Switch to the File Type Associations tab Press Add • In the Extension field, enter> SHT • In the bottom Extension field, enter> DGN – So that SHT files will be processed as if they were DGN files • Press OK to complete the Extension mapping Press OK again to close the Properties dialog

Lab 3d

• Once new file type has been added… – Now a small problem • These files were flagged as done, and the Indexer won’t try them again unless they are checked out/in • And even that won’t work unless you actually makes changes… • PW compares files to version on server, and doesn’t transfer back if there are no changes

Lab 3e

• Rather than check them all out, and back in – From PW Administrator – – – • Right-click Full Text Indexing Choose> • Mark folder Documents for Reprocessing Browse “…” to • USer1/Document Indexing/MS-SHT Press OK • Press OK again

Analysis

• Within 2 minutes, these documents will be re processed – If you run the search again (in a few minutes), you should also get SHT files in your results – Re-visit Datasource statistics to see if it Full Text categories have changed

Summary

• Once the index is created, – You can stop the PW Orchestration Framework service – – • It is used to create the index, but not to search the index This will save memory, and CPU cycles • So in a demo, your machine will run faster • BUT, new, (or modified) files will not be re-indexed Up until now, the PWOF was not being used at all • Full Text Indexing is the first time we’ve needed PWOF, even though it has been running since installation

PW Thumbnails

• PW Thumbnails is not “indexing” in the proper sense, but it is similar in nature to Full Text – PW Thumbnails extracts a thumbnail from the document, and stores a copy in the PW database – • This allows one to browse PW Explorer, and see thumbnails in the Preview Pane Not all file types support thumbnails • Among those that do, some don’t do it per the industry standard

Thumbnails – Part II

• Important to remember – ProjectWise does not create thumbnails – • It only extracts what might be in the file A good test is to check to see if Windows Explorer displays a thumbnail for the file • If it does, then PW should as well

Lab 4a

• Open Windows Explorer – Browse to: – • C:\PW-101 Class Files\Document Indexing\MS-V8 Change to Thumbnail display • MicroStation V8 files have thumbnails

Lab 4b

• Browse through remaining Document Indexing folders – Note which include thumbnails – Additional notes • PDF files take a long time because you are really looking at a small view of the whole file, not a thumbnail • AutoCAD doesn’t adhere to the Industry standard – These files only display correctly because MicroStation is installed, and is responsible for displaying a thumbnail – Autodesk may have fixed this in later versions?

Lab 5a

• Open PW Administrator – Log in as> adminpw – – Drill down to: • Document Processors> Thumbnail Extraction Right-click, choose> Properties • Similar to Full Text Indexing – But actually less involved

Lab 5b

Turn on adminpw adminpw Set to 60

Lab 5c

Enable all times in the schedule Set to 2

Lab 5d

• No changed required on the File Type Associations tab – Press OK to complete the configuration and close the dialog • Within a few minutes, thumbnails should show up in the preview pane

Analysis

• Thumbnails are extracted and stored in the PW database – Because document storage may not be local – • Thus “touching” the document to see thumbnail in real-time is not practical Thumbnail notes • Requires less processing than full text – MicroStation not running in this process – Requires PWOF to extract, but not to display

Review

• Topics covered in this Chapter – Full text Indexing – Configuration – – – – Full Text Searches ProjectWise Orchestration Framework Thumbnail Extraction Microsoft Indexing Service • And iFilters to extend default supported file types • (I have a free Visio, and MSG iFilter from Microsoft)