Jeff Fried Senior Product Manager Microsoft OFS210 Big Improvements in Search In areas with common customer requests Scale – content and query scale becomes a.
Download ReportTranscript Jeff Fried Senior Product Manager Microsoft OFS210 Big Improvements in Search In areas with common customer requests Scale – content and query scale becomes a.
Jeff Fried Senior Product Manager Microsoft OFS210 Big Improvements in Search In areas with common customer requests Scale – content and query scale becomes a non-issue User experience – guide users with rich, visual navigation Social search – search unlocks the value of the people and expertise Relevance –improving matching and ranking and language support Platform –tools to customize, extend, or build new applications Management & reliability – reliable crawling, scriptable admin, improved monitoring Rich Content Processing– extract metadata from my enterprise content Microsoft Enterprise Search- Productivity Products for Every Customer Need Complete intranet search High end search delivered through SharePoint Common across the product line • Common UI Framework • Social search features and integration • SharePoint platform integration • End user and site administrator enablement • Common Connector Framework (BDC) • APIs and developer Experience • Admin & deployment capabilities • Operations advantages (SCOM, scripting) Search Center - UI for users to issue queries and interact with results OpenSearch Source Query Object Model Query Servers - Accept query requests from users and returnresults Query Servers Query Federation - Returnresults from non-SharePoint Indexes Index Partition Indexing - Extract information from items to enable efficient matching Indexer Index Partition - Subset of the overall index Crawling - Traverse URL space to record items in searchcatalog Crawler Connectors - Know how to processdifferent content sources Content Content Content Topics End-user Enhancements Search is Social Scale-out Architecture Extensibility Enhancements Deployment and Manageability End-User UI Out-of-box refinement Refine over key results properties Easy to extend over custom properties One-stop Search Center Scopes, web parts, best bets, top answers , advanced search Query federation brings together results from all over native support for OpenSearch Core search experience Metadata, taxonomy and social tags based results refinement Improved did you mean suggestions New pre-query and post related query suggestions “View in browser” link (for most office docs) Improved query syntax End-User UI (continued) Improved relevance ranking New ingredients: URL fuzzy matching, social tags, results click through, implicit phrase matching, extracted metadata, etc. Improved low-noise snippets in summaries Enhanced multi-lingual support Automatic detection of language of many document types and part of documents Compound word handling - e.g., Innovationszyklen” and ”innovation“, “zyklen” Improved ranking of documents in multilingual collections New form factors Mobile search from Smartphone browsers Desktop search integration in Windows 7 New Query Syntax Support for Boolean operators for FreeText queries and property queries (“SharePoint Search” OR “Bing Search”) AND (title:“keyword syntax” OR title:”query syntax”) Prefix matching support for keywords and properties Micro* author:bill* Improved operator support for property restrictions =, >, <, <=, >= Can create range refinements End-user UX Jeff Fried Enterprise Search Group Topics End-user Enhancements Search is Social Scale-out Architecture Extensibility Enhancements Deployment and Manageability Search is Social People finding experience Better expertise & interest search Email mining to bootstrap profiles with interests and colleagues “Address book style” search Phonetic name matching Nickname matching Relevance tuned for people search Metadata refinement, better hit highlighting, recently authored content Search is Social (continued) Social behavior drives search quality Search click through behavior drives relevance ranking Query suggestions mined from search logs Social tagging influences relevance ranking Self search - to drive people to participate content Social definitions extracted from indexed content People Search Jeff Fried Enterprise Search Group Topics End-user Enhancements Search is Social Scale-out Architecture Extensibility Enhancements Deployment and Manageability Scale-out Architecture 2010 core engine tenets Sub-second query latencies at large scale Fresher indexes Better resiliency/higher availability Basic philosophy Componentize the system Remove system bottlenecks through scale-out MOSS 2007 Search Scale-out “The whole index” Query “Bottleneck” Query Indexer + Query “Bottleneck” “Single point of failure” SharePoint Server 2010 Search Scale-outMultiple Indexers Stateless Crawlers Crawl Distribution Admin Database Admin Component Query Query Query Mirroring Query Partitioning Multiple Property DBs “The whole index” Admin Database + Admin Component Query Query “Bottleneck” “Single point of failure” Indexer Indexer Crawler Crawler Crawler “Bottleneck” Crawler Content Distribution Crawl Distribution In-built load balancer distributes hosts to crawl databases Crawlers crawl content that is covered by crawl database Default configuration can be overwritten using host distribution rules E.g. purchasing a new connector Query Distribution Low query latency if all index partitions equal in size Distribution by hash of documentId Crawlers partition indexed data and propagate to query servers Industrial Strength Resiliency 2007 style mirroring for index partitions Redundant components provide failover Ability to add multiple crawl components to minimize crawl downtime Machine down doesn’t result in crawl downtime Native support for SQL mirroring Scale-out from an art to a recipe To improve this… Take these actions Index freshness/crawl times If Indexer machine is saturated , add more Indexer machines and/or crawl components If Crawl Database is I/O bound on SQL Server, add additional crawl database on same SQL Server If SQL server is Memory/CPU bound, add additional SQL Server with additional crawl databases Query Latency/Throughput If time spent in resolving query in full-text index is high, partition the index into smaller index partitions. Each index partition can contain ~10M items. If query throughput is low, add query components with mirror index partitions. If Property Database is I/O bound on SQL Server, add additional crawl database on same SQL Server If SQL server is Memory/CPU bound, add additional SQL Server with additional crawl databases Query Availability Deploy redundant query servers, redundant index partitions and components, and use clustered or mirrored database servers to host property databases. Crawl/Indexing Availability Use multiple crawler components on redundant index servers, and add crawl databases. Crawl components associated with a given crawl database can be distributed across index servers for availability and load distribution. Other Engine Enhancements Support for regular expressions in Crawl Rules Native support for crawling case sensitive repositories Ability to prioritize Content Sources so as to distribute crawler resources New ‘Crawl Policy’ to define how crawler treats error conditions Low indexing downtime Search Backups Topics End-user Enhancements Search is Social Scale-out Architecture Extensibility Enhancements Deployment and Manageability Do More With Search Across a Spectrum of Customization and Development Configure Extend Create SharePoint Search Platform Architecture Search Extensibility: End-User UI Change web part properties – no code Modify XSLT Modify config XML Refinement panel – control metadata available for refinement Advanced search – control metadata available for advanced search queries Extend OOB web parts programmatically All OOB web parts are public Extend web parts to change default behavior E.g. Extend core results web part to show result previews Extend Search Center with vertical experiences All web parts communicate through public interface (SharedQueryManager object) Search Extensibility - Federation Query Federation Bring external search results into the SharePoint experience Based on Open Search standard Exposed through UI + public federation object model Connector Framework New features Support for attachments Item level security Crawl through entity associations Inline caching + batching for better citizenship Richer crawl options Regular full crawl (like 2007) Time stamp based incremental crawl Change log crawl + deleted count Change log + delete log crawl * Protocol handler API still supported Connector Architecture Search user queries Search page points to Profile Page or custom page to show search result Search results page – results based on metadata Search Engine/Index Search BDC Runtime Protocol Handlers BDC Profile Pages or Virtual Lists Fetch data from repository Execution call routing through BDC Shim using model file WCF Databases .NET Shim Custom shim How to write connectors Use OOB shims (Database/WCF/.NET) Create/Deploy model file using SPD and use search UI to configure crawls Create/Deploy .NET classes using Visual Studio and use search UI to configure crawls Recommended if backend structure is static Write Custom Shim + Model file Useful for dynamic back-end structures. E.g. exchange public folders Topics End-user Enhancements Search is Social Scale-out Architecture Extensibility Enhancements Deployment and Manageability Deployment simplicity Prerequisite Tool SharePoint Server Setup SharePoint Server post setup config Farm Config Wizard Search Dashboard Search Installation and Topology extension Jeff Fried Enterprise Search Group Richer Manageability Consolidated administration UI dashboard Automated service password management through “managed accounts” PowerShell support for scripted administration Built-in system health monitoring, support for SCOM monitoring and alerting Built-in and extensible search analytics reporting Search Administration Jeff Fried Enterprise Search Group SharePoint Server 2010 Search: Best Infrastructure Search Major improvements over MOSS 2007 Area Relevance New features and benefits Provide great search results • New ingredients, more experience -> better OOB search • Better wordbreakers in more languages • Social Distance, tagging, click-through influence relevance Refinement Find information faster • Metadata-based refiners provide drill-down • Related query suggestions show popular queries Social Connect with people and expertise • Address-book style name lookup with phonetic matching • Improved expertise matching - mined e-mail inbox data • Social behavior improves search; search improves mySites Platform Industrial Strength Search at High Scale • Index partitioning -> sub-second latency at 100M items • Full failover support • Connector and BDC enhancements and tooling support Extensibility Create powerful applications based on search • Web parts, query OMs are public -> easy to modify • Enhanced query syntax TCO Deploy and Operate with less effort • Virtualization and native 64-bit simplify deployment • Powershell, Admin web parts, SCOM provide easier admin Microsoft SharePoint 2010 Business Collaboration Platform Resources www.microsoft.com/teched www.microsoft.com/learning Sessions On-Demand & Community Microsoft Certification & Training Resources http://microsoft.com/technet http://microsoft.com/msdn Resources for IT Professionals Resources for Developers Related Content CLI317 Windows 7 and its role in Enterprise Search 11/10/2009 15:45-17:00 London 3 - Hall 7-1b CLI07-IS Search, Organize, and Browse Discussion 11/11/2009 17:30-18:45 Interactive Theatre 3 – Blue OFS05-IS Microsoft FAST Search Server 2010 for SharePoint 11/13/2009 09:00-10:15 Interactive Theatre 1 - Red Complete an evaluation on CommNet and enter to win an Xbox 360 Elite! © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.