Offline SharePoint

This is my sixth and last post in a guest series I’m doing here on “The Myths & Truths of Email Management with SharePoint.”. My last post was on SharePoint list scalability.

SharePoint is a great platform for managing email and attachments and has several advantages in the right scenarios. In addition to providing the capability to store, organize, and search for content, SharePoint enables email to become part of the content that is shared throughout the organization. This improves collaboration and content re-use. There are a number of alternatives for moving emails to SharePoint, including out-of-the box methods such as email-enabled lists, managed folders, and third party applications such as Colligo Contributor Add-In for Outlook. The key to success is building an architecture that is scalable, while making it easy for information workers to use.

PST Files – While some of the capabilities of Exchange and SharePoint overlap, there are distinct differences and advantages of one platform over the other for different scenarios. When important files are stored on individual PST files throughout the organization, the information becomes inaccessible islands. Worse still, PST files pose security and data loss concerns, and often lead to the loss of enterprise content. In reality, PSTs are a thing of the past. Storage and sharing of email and attachments in SharePoint offers distinct advantages for collaboration, document management, and discoverability through a rich search interface.

Exchange Public Folders – While I don’t recommend deploying Public Folders for new applications, you needn’t worry if you already have them. They will continue to be supported by Microsoft in the next major version of Exchange server after Exchange 2007, and for at least 10 years thereafter. For new Exchange deployments, SharePoint is “strongly encouraged” for application development.

If deploying both Exchange 2007 and SharePoint, it doesn’t make sense to build up a huge Public Folder deployment. It’s confusing to users and additional overhead for the team managing Exchange. Microsoft is making major investments in SharePoint and has indicated that it’s the preferred application platform for collaboration and document management going forward.

SharePoint Advantages – As discussed, SharePoint offers a number of advantages over PST files and Public Exchange folders for email management:

• It supports the sharing of email across the enterprise
• User-based email and attachment selection ensures that “important” content is stored
• SharePoint centralizes critical enterprise content on secure company servers
• Email becomes “structured” enterprise content when custom metadata is applied
• The email body and attachments become findable and reusable
• Information workers can take advantages of the version control and history features
• Content types support retention policies and regulatory compliance requirements
• SharePoint improves collaboration and enterprise content management

“Out of the box” there are two ways (highlighted in this series of posts) that Microsoft products support the integration of Exchange and SharePoint for email management: (1) “send to” email-enabled SharePoint lists and (2) “auto copy” Managed Folders to SharePoint.

Email-Enabled Lists – Email-enabled self service lists can easily get out of control and become a significant drain on IT resources. Microsoft IT decided against using email-enabled lists internally. Most organizations don’t need the email-enabled functionality and the oversight it requires, and users prefer to access archived emails in .MSG format (rather than the .EML format created in SharePoint lists). With proper planning and design, email-enabled lists can be deployed successfully. However, Managed Folders are a better alternative.

Managed Folders – Introduced in Exchange 2007, Managed Folders provide administrators with an easy way for users to archive email. Managed Folders can be configured to “auto copy” emails added to them to SharePoint libraries. This incredibly insightful feature can significantly reduce mailbox sizes, while capturing the intended emails (in .MSG format) and attachments. Still, abuse must be avoided to ensure that SharePoint does not become a dumping ground. Managed Folders require a solid information architecture design, and trained administrators who understand scalability and how to manage it.

SharePoint Scalability – Storing emails and attachments in SharePoint can have great advantages for project and knowledge management in corporations, if they are secured in the proper context. However, SharePoint lists and libraries can quickly be bogged down with too much content if not carefully managed. The scalability of SharePoint lists is a major consideration and requires oversight, especially when using email-enabled lists. The default “All items” view can be particularly problematic.

Training and guidance may be required to change corporate culture so users understand where and how content should be stored. Training users to tag content with metadata will increase the usefulness and “findability” of documents. Folders, custom views, indexed columns, and a query-based design are best for large lists to ensure fast performance.

Third-Party Solutions – When deploying SharePoint for email management, it’s important to consider what you get out of the box with Microsoft, and what you get with a partner. Most gaps you find in SharePoint are filled by a very solid partner ecosystem. For example, Gold Certified Microsoft Partner Colligo Networks offers an Add-In for Outlook that makes it easy for users to drag-and-drop emails into SharePoint, while reducing the burden on IT. I would encourage you to try their .Net client-only solution to see if it can meet your Outlook and SharePoint integration requirements.

Well, that’s it for this series of posts on the OfflinesharePoint blog. I hope you found them interesting and informative. If you’d like to keep up with my latest posts, please visit my blog “SharePoint Joel’s SharePoint Land”.

- Joel Oleson.

This is the fifth in a guest series I’m doing here on “The Myths & Truths of Email Management with SharePoint.”. My last post was on Managed Folders.

Storing all emails and attachments in a single document library is a common practice and popular method for personal storage, however this is not a recommended best practice for knowledge repositories. In SharePoint, document libraries require special information architecture because of performance degradation associated with lists that contain a large number of items.

To gain a better understanding of the issues, there are a few documents that I would recommend developers and architects review, as outlined below.

• The whitepaper called “Working with large lists in Office SharePoint® Server 2007” has charts and metrics based on query time with performance considerations being a key indicator. The recommended limit of 2000 items per folder popularized in WSS 2.0 is not a bad one for those that don’t have the time to invest in designing a query based interface, though there is no performance “cliff” at 2000 items in a list. You may want to recommend to your users that they keep their lists under 1000 or use 3000 items as an upper limit at which point IT gets involved in assisting teams and groups to more efficiently scale their lists.
• Microsoft IT recently published a whitepaper titled “SharePoint Performance Optimization” that discusses ways of optimizing render time of sites and reducing database blocking. At Microsoft, the IT team has set-up a scan to detect large lists over 3000 items to ensure proper usage and divide up content where appropriate.
• Another useful article is the one on capacity boundaries on TechNet. It’s a good resource for seeing guidance on “limits” within the product along with charts on list performance.

If you currently have lists with a large number of items, or cannot avoid them in your application, below are a few quick ways to improve their performance.

1. Views can have a big impact on performance. The default “All Items” view should be avoided because it can take tens of seconds to render as a list grows into the tens of thousands. Views with too many items can can cause content database locking during query time. Changing the default view to something with a smaller number of items can often be a quick fix.
2. Filters are another way of limiting the number of items displayed in SharePoint. Note, however, these queries can be inefficient in large lists.
3. For document management deployments, use folders and sub-folders for better scale and retrieval. Keeping each folder with less than 2000 will support better performance. Even better, limit folders to one hundred or so to prevent unnecessary scrolling.
4. Using Indexed columns is another way to manage list scalability and increase index query optimization.
5. The most efficient method of retrieval is search queries.

Truth: SharePoint lists exhibit performance problems as list size grows, so it’s best to limit the number of items in lists and use views, folders, and filters to improve query performance.

Happy New Year! This is my first post of 2009 on the Offline SharePoint blog and the fifth in a guest series I’m doing here on “The Myths & Truths of Email Management with SharePoint.”. My last post was on SharePoint email-enabled lists.

The subject of this post is Managed Folders. Managed Folders were introduced in Exchange 2007 to provide administrators with an easy way for users to archive email. Any Managed Folder can be configured such that all emails sent to it are routed to SharePoint. It’s an incredibly insightful feature and when implemented properly can reduce mailbox sizes, while capturing the intended emails and attachments. When not implemented properly, Managed Folders can be abused, causing SharePoint to become a dumping ground.

One poor example of managed folder design could be a managed folder called “Keep.” If everyone is told to put their email in that one folder and it’s archived to SharePoint, it sounds like a perfect solution for email archiving. However, there are some serious drawbacks. First, scale is an issue. Putting all that junk into one list can overwhelm SharePoint since it doesn’t scale well to support millions of items in one list, especially if there is a single view. Second, what about the security of that list? Managed Folders require extensive administrative set-up. It’s an IT option in Exchange, not a feature that is end user configurable.

So what does a good Managed Folder design look like? An example might be a folder titled “Legal Hold,” which is used to archive items under legal hold because of an investigation or other circumstance. On the SharePoint side, a specific document library is set-up and secured, then a legal site administrator is responsible for any tagging and for managing the views for the LCA team. A special search view might be set-up with specific indexed columns to support a quick and easy search. To avoid performance problems, avoid the “All items” view. I’ll discuss performance issues related to SharePoint list scalability next.

Truth: Managed Folders can work well, but require a solid information architecture design & trained administrators who understand how to manage scalability.

This is the fourth post in a guest series I’m doing here on Email Management in SharePoint. The third post was Myth #2.

Emailing a post to a blog … very cool or archiving an Exchange Discussion List to a SharePoint list … super cool … but be careful. Email-enabled self service lists can easily get out of control. Microsoft IT, which loves to use nearly every feature of SharePoint, decided against using email-enabled lists.

Email-enabled lists can be a significant IT resource drain. Without the proper planning and management, AD objects will be created with archiving and no lifecycle. Contact account naming standards are another reason. IT doesn’t want to see random contacts in AD.

Everyone wants to have the document library called “docs” and everyone wants to have the discussion list called “discussion”. If you have a process or even a workflow to get requests and manage these requests, you can better manage who needs them, when they are needed and for how long. So, my recommendation here is to know what you’re doing. Otherwise, it’s very easy to end up with a mess.

Like Public Folders, email-enabled lists can also pose security risks if not managed properly. Fortunately, they are more often secured to the context of the team so they are not as much of an exposure.

List scalability can pose problems with email-enabled lists. You don’t want to send all data from all users to one list. Put content in context in different site collections, sites, folders, as it relates to the context of the group, team, or project.

Most SharePoint environments don’t need the email-enabled functionality and the oversight it requires. Those that decide to use it should plan to set-up specific content objects and point the lists at them. Steve Smith’s document called “How to configure Incoming Email Enabled Libraries in MOSS2007 RTM using Exchange 2003 in an Active Directory Domain” explains in detail how to set-up email enabled lists. Don’t be surprised if it’s more complex to set-up than initially thought. I recommend setting it up in a preproduction environment first and learning how it works, then exercising administrative tasks and troubleshooting tasks around maintenance of the list, inbound SMTP, and AD contact objects.

In summary, I’d suggest you get answers to the following questions before implementing email-enabled SharePoint lists:

• Are those contact objects in a separate OU?
• How do you know if the inbound SMTP stops working?
• Can anyone send to the list?
• Does the item show up the way you expect it to, or are just attachments showing up?
• Does the metadata look like you expect it to?

The answers will help you better understand the nature of these special lists and help you better take advantage of this functionality, if you decide to use it.

Truth: Use with caution: To be successful, email-enabled lists require management & oversight to scale past a few thousand items.

This is the third post in a guest series I’m doing here on Email Management in SharePoint. The second post was Myth #1.

In early 2006, the Exchange Team at Microsoft outlined their thoughts about the future of Public Folders in a blog post titled “Exchange 12 and Public Folders.” It was intended to let customers know that Microsoft was de-emphasizing Public Folders for certain applications, but many misunderstood this to mean that Public Folders were dead.

Microsoft’s position was clarified in March 2008 in a post titled “Updated Exchange Public Folder Guidance,” which encourages use of SharePoint for new deployments and migration where overlap exists. This blog confirmed that Microsoft will continue to support Public Folders in the next release of Exchange and for an additional 10 years from its release.

This post also provides the first scenario-based examples where Public Folders and SharePoint are compared. I know both the SharePoint team and the Exchange team were involved since I was on the SharePoint team when it was designed and played a role in getting it cross posted on the SharePoint team blog.

Click to Enlarge

Above is my version of the table in the blog post. I’ve changed some of the wording to make it easier to understand, with a bit of commentary emphasis on really investigating SharePoint functionality when both Public Folders and SharePoint have overlap across a scenario. SharePoint is “strongly encouraged” for most application development, though in some instances there still may be some advantages to Public Folders. For example, the capabilities of team calendars in Exchange vs SharePoint should be compared against the specific requirements of the application. Also, if you are looking at email-enabled lists in SharePoint, you may find that public folders are a better match for your requirements.

While de-emphasizing Public Folders, Microsoft didn’t stop evolving them altogether. For example, in Exchange 2007 there were new ways of managing the Public Folders with Powershell cmdlets. Please note that the functionality and features require investments on the part of the customer. However, the current guidance is that Public Folders are now in maintenance mode.

It’s illustrative to look at what Microsoft IT is doing internally with Public Folders. They decided to impose constraints both on time and size to limit the growth of the stores. Microsoft IT also turned off replication to reduce the unnecessary growth of storage. New folders were locked down to exception-based requests, with new provisioning requests pointed to SharePoint (with the common exception of distribution list archiving).

So why is Microsoft IT moving more content to SharePoint? In contrast to Public Folders, SharePoint was designed as an application platform and it’s important to note that this is the continuing direction of Microsoft. Both Windows SharePoint Services and Office SharePoint Server are major investments for them and they will continue to augment their functionality to support the common scenarios of customers.

While Exchange Public Folders were previously the endorsed solution by Microsoft for sharing email, SharePoint is now the recommended solution. If you are still considering Public Folders, it’s important to understand it has several well known limitations and challenges, described below.

1. Public Folders and custom applications built on them are the bane of the life of an exchange administrator. Control, performance, scale, and replicating chaos are a number of common concerns. Public Folders require heavy upfront and ongoing management.
2. Security is also often a concern due to the public nature of email-enabled Public Folders. Organizations are exposed to large risks if the random content flying around in emails is not controlled, so Public Folders need to be carefully managed. Security and liability are definitely something to be considered in all cases where email content is stored. It’s important to understand who has rights to store email on the server.
3. Public Folders can also be an HR nightmare. These “public” social dumping grounds often become a place where pictures and music are exchanged. As well, DLs (Distribution Lists) can get buried in group and team email-enabled security groups.

Bottom line: if you are deploying both Exchange 2007 and SharePoint technologies (WSS or SharePoint Server), it doesn’t make sense to build up a huge Public Folder deployment. It’s confusing to users and additional overhead for the team managing Exchange.

But should you use Public Folders when you are doing a Groupwise, or Lotus Notes migration? For nearly all your Notes applications you should seriously look at SharePoint. For email it’s a no brainer… it’s Exchange. For discussion groups and archiving, you’ve got a decision to make. When migrating to SharePoint, you should do it with an understanding of what you get out of the box, and what you get with a partner. Most gaps you find in SharePoint are filled by a very solid partner ecosystem.

One last point. If you plan to customize SharePoint sites and site templates, have a look at this blog post on Features and Solutions. Features allow reusable pieces of functionality to be created and deployed to other sites, without modifying site templates. Solutions allow you to package Features in a cabinet (.cab) file and define important metadata about those Features.

Truth: The Exchange Team will include public folders in the next version and at a minimum support them for 10 years after the release. However, they encourage you to seriously consider SharePoint for application development.

This is the second post in a guest series I’m doing here on Email Management in SharePoint. The first post was an introduction.

It’s very common for people to think, “I need access to an important document so I’ll email it to myself and then I’ll have it offline in my PST folder.” While this may provide value to a single user, it conflicts with the goals of enterprises that are trying to make content reusable. When important files are stored on individual user desktops or laptops in personal PST files, the information becomes an island that no one else can access.

Worse still, PST files are not generally backed up, which means critical information is lost when files become corrupted through a hard disk crash. Of greater concern are the millions in corporate assets that disappear each year through hardware theft or loss; information on laptops and mobile devices is even more of a liability than the hardware.

In addition to the convenience of mailbox folders, email feels like a safe place to store information. However, since email is usually the first thing exposed to the firewall, there is a risk that critical corporate information can be compromised or stolen by hackers.

In reality, PSTs are a thing of the past. As email evolves, PSTs are being phased out in many corporations. Quotas force mail purging, and the liability of some confidential conversation from years ago can simply get expired.

So what’s the alternative to storing email in mailbox files and PSTs? Why SharePoint, of course. If users can easily save their important emails and attachments in lists and libraries that relate to their projects, teams, and groups, that content becomes searchable and reusable by others in the corporation. Since users can then take advantage of version control and history, emails can evolve to published content in knowledge repositories. Like other documents in SharePoint, emails now inherit a history, and the collaboration around them is tracked and relevant.

The use of SharePoint to manage email at organizations has been growing. Take Microsoft for example. A quick scan of the SharePoint environment returned .MSG/.EML files among the top 10 file types. It followed the ones you’d expect (the common office file types), but ranked higher than Visio and Project files. When stored in SharePoint document libraries, .MSG files can be quite rich and look just the way they appeared in Outlook. They even get the familiar envelope icon and, most importantly, the body and attachments are searchable.

There are a few things to consider when developing email management solutions in SharePoint. If the application is for archival purposes to comply with regulations, content types can be useful to specify an expiration policy so that emails sent to a records center are automatically deleted after a specified hold period. If the application is more about collaboration and content management, there needs to be a mechanism to ensure that important conversations and documents that need to evolve end up in SharePoint, while unimportant information is not stored.

This cannot be accomplished with an automatic push mechanism that just sends everything up to the server. In addition to creating a storage nightmare, that would pose some big security risks. For those reasons, I recommend that emails and attachments saved in SharePoint are selected by proactive information workers who know the relevancy of the content and who want to do the right thing for their team and the company.

If the solution enables users to apply metadata at the time content is moved to SharePoint, documents will have the context that will make them searchable and discoverable in the knowledge repository, document management system, project site, team site, or MySite. This enables the content to live on and reach its potential. We need to learn to share; it’s one of those things we learned in Kindergarten, and it’s a lesson we can apply to our adult life. If we keep email and attachments in our mailbox and don’t share them, the information may well die there.

Truth #1: SharePoint is a better place to store email and attachments for archival and collaboration applications.

-Joel

As Barry mentioned, I’ve agreed to do a series of posts on Email Management in SharePoint right here on offlinesharepoint.com over the next couple of weeks. I encourage your feedback here or on my blog, sharepointjoel. Feel free to comment or ask questions on the material I’m going to post.

So let’s get started.

With the growing demands for compliance, doing more with less, and information reuse, CIOs, Information Architects, and System Administrators have an overwhelming number of technology choices and strategies to consider when designing a solution to manage and share their information assets. I’m sure we would all agree that emails and attachments form a large part of those assets. Today, users must sort through thousands of emails to find the critical ones while being pressed to make decisions immediately. It’s difficult to find, categorize, and sort content to stay efficient, so data gets lost, and more and more time is spent searching for information.

In this series of posts, I’ll discuss five things you MUST know if you are developing an email management solution. The focus is on high level issues that can affect the overall architectural approach. I’ll dispel some of the myths surrounding out-of-the-box functionality available on Microsoft platforms, and provide guidance on how to manage some of its limitations. Where possible, I will describe some of the choices that Microsoft IT made internally to deal with some of these limitations.

Along the way, Colligo Networks will provide tips on how their Colligo Contributor Add-In for Outlook, can improve your email management solution in SharePoint by making users more efficient, while reducing the burden on IT.

Whether you are architecting an email archiving solution, or consulting for a customer, this series of posts should interest you.