Everyone loves data, I am sure of it…  The key difference though, is most people in their right mind, simply wouldn’t admit it.  Data is useful, particularly in this day and age, in what is now commonly referred to as the Information Age.

Now, you might be thinking I am crazy – and that is probably true – but data is the life-blood of modern-day organisations.  So what happens when can’t query, transform or extract your data because you hit Salesforce API Limits?

Intro to APIs….

Ok, ok – I hear you.  What the hell is an API limit?  For that matter, what the f*** is an API?  And why should you care?

Starting at the beginning (feel free to skip this section if you already know this!)…

An API, or Application Programming Interface, is at its most basic a way programs or systems talk to each other.  I am way over-simplifying things here.  But an API ultimately allows one piece of software to communicate with another piece of software in an agreed way.

Software, like Salesforce, is built to be a relatively open platform.  And they do this by using a number of APIs to allow other bits of software to communicate and link to it.  These links allow developers to do things like query, extract, create or update records within Salesforce with relative ease.

Salesforce supports SOAP API, REST API, BULK API, Streaming API, Metadata API, etc….  So there is a number of ways to access what you need.

Tools you may use as an Admin everyday, use a variety of these APIs.  For example, DataLoader uses the SOAP/REST/BULK APIs depending on its setup.  Likewise Workbench supports APIs too, as it can connect and manipulate data, query the fields, test API connections within Salesforce.

Even tools like Salesforce for Outlook use APIs to query records and insert attachments.  And as an admin this is why you should care.  These tools, along with many others, use APIs to connect to Salesforce.

But with great power, comes great responsibility…

So with all these connections into and out of Salesforce globally, surely that can slow down the platform?

That is what brings us to the Salesforce API limits.

Salesforce is a multi-tenanted platform, meaning no-one company has an exclusive server/computer/machine setup for them, everyone shares the infrastructure which drives the platform.  So being cloud-based, Salesforce needs to ensure the performance of the platform works for everyone.

To do that, Salesforce impose limits.  Governor Limits, Storage Limits, Per User Limits….  All based on the version you have bought from them.

These limits are designed primarily to cap usage of the platform, ensuring stability of the platform and still give a fair amount of flexibility.  All the while, also providing a potential upsell opportunity for extra Salesforce services (eg Data Storage limit).

Salesforce API Limits

Among these limits though are API Limits.  And even though you normally shouldn’t notice there are limits in place, when you do hit the limits – especially when you hit them unexpectedly – it can be a little jarring…

System.Web.Services.Protocols.SoapException: REQUEST_LIMIT_EXCEEDED: TotalRequests Limit exceeded.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)

This is the error which stops all work via APIs… If you rely on APIs to connect to your systems – you really don’t want to see this message!

In this example, a connection using the SOAP API to connect into Salesforce has reported the Request Limit has been exceeded…

Diagnosing API usage…

As an Admin, to diagnose the issue the first port of call is to analyse and find out how you hit the limit.

Every org has quick look view at the API limits.  So go ahead – check for yourself.  Go to Setup | Quick Find | System Overview to see what you have within your org (if you have access to the Setup/Config of your org that is…).

Firstly, all this really confirms is what has happened.  Like a car not starting, you have confirmed there is a problem – but you haven’t diagnosed the issue.

When is 24hrs, not 24hrs?

And secondly, when is the 24hrs?  Is it a rolling 24hr period, is it the start of each day local time or even the start of the day in San Francisco?

Well, the good news is this limit is a rolling 24hr period.  Meaning it shows you the last 24hrs, from the time you are checking the limit.  This is important to know, if you have a lot of batch processes which might be scheduled at specific times of the day.

Is that all the information you need?

Does that really help you though?  Overall it doesn’t really give you much insight as to what has really happened within your org.

So, what about Company Information?  There is an API Request Limit there too…. (Go to Setup | Quick Find | Company Information)

Nope… still not much help!  In this example (yes, two different orgs), we can see the API Limits is 98.9% at the limit for the day.  So you will likely start to see problems with API connections…

API Usage Last 7 Days

On the Help page, Salesforce referenced an ‘API Usage Last 7 Days’ report, but didn’t mention where I could find the actual report!

The help article advises that you need: ‘View Setup and Configuration’ permission on your to see this report.  I definitely have this permission within the org…. But I still couldn’t find the report: 

Then I was reminded about the fact Salesforce comes with some generic but useful standard reports.  And guess what is included in the folder!?  Yep, an API Usage Report 🙂

But you need to go via the Administrative Reports folder in Classic to view it…  Go to Reports | Folders | Administrative Reports:

If that doesn’t work for you, you can also use this URL hack (thanks to Joshua) to jump to the report within your org:

https://na1.salesforce.com/00O?rt=104&retURL=%2F00O&c=UN&c=FULL_NAME&c=EM&c=CID&c=TS&c=CC&duel0=FULL_NAME%2CUN%2CEM&scope=organization&details=yes
Replace the na1 with your instance.  Below is the same url without the na1.salesforce.com
	
	/00O?rt=104&retURL=%2F00O&c=UN&c=FULL_NAME&c=EM&c=CID&c=TS&c=CC&duel0=FULL_NAME%2CUN%2CEM&scope=organization&details=yes

Diagnosis, complete…

Well… Almost….

The report gives you a great breakdown to see what/who is using up your Salesforce API Limits over the last 7 days.  But it doesn’t fix the overall issue.  The fix itself can be as complex as the software/tool that is causing the API Limit to be used up!

But at least you can now diagnose the issue and decide if you can fix the issue, alternatively if it is something you aren’t likely to be able to fix/reduce you can sometimes buy additional API Limits from Salesforce by speaking with your AE…

Maintaining a clean Salesforce org, doesn’t need to be a battle. Recently, I was watching a great webinar by Kelly & Leanne entitled ‘Cut that Clutter‘. And it got me thinking about how the problems faced by a cluttered Salesforce, and how it can easily consume an Admin’s time and effort. And it not only affects us as Admins, but also our end users! So in this post I am going to recap some of the awesome tips shared in this webinar, and also see offer some additional FREE tools to tackle the problem that so many of us face!

The War Against Clutter

Ok, ok. I admit that this header is full of hyperbole.

But a cluttered Salesforce creates a lot of frustration and anxiety for me. And I assume most it does for most of you reading this too! My personal vendetta against clutter drives me to ensure I am always improving the org for my end-users…

I have previously written about how we started to tame the Technical Debt beast haunting our primary orgs and removed over 2 million records from an org (hint: very manually).

But there is still so much to do…

It is a seemingly never-ending fight. But as Admin’s we are always looking for tools and resources to help us in our day to day Admin Superhero duties. And to help us in maintaining a clean Salesforce org…

Cut that Clutter! – The Recap

Now before we get much further. If you have a spare 30mins I strongly recommend that you watch the webinar as I am only going to briefly summarise it here…

Cut that Clutter! Maintain a Clean Org

If you have hundreds of custom fields on an object, 20+ installed packages and more page layouts than you know what to do with, it’s time to clean your org. ...

YouTube

The session covers the Three-S’s. These are the primary areas to focus on, to ensure your CRM is kept in tip-top shape.

• Security – making sure you know who can see what in your CRM
• Structure – does the setup of Salesforce ensure data security and meet any data governance requirements
• Strategy – how to plan and scale while ensuring you don’t have to keep doing ‘big clean-ups’ each year

If you want to hear more, then please check out the video.

Cut that Clutter: Resources mentioned

Next up, the ladies mentioned some great tools to help you in maintaining your Salesforce org.

From Salesforce:

• Salesforce Optimizer (aka Optimiser in non-US/Canada countries 🙂 ) – I am in LOVE with Optimizer reports. It is such an amazing tool to help you analyse and understand where the Technical Debt is likely to be hiding within your org. This should be your first port of call, in maintaining a clean Salesforce. That is how much I love it!
• Security Health Check – helps you understand any vulnerabilities you may have within your Salesforce. This covers areas like Password Policies, Critical Updates, etc.

From AppExchange:

• Field Trip – this tool is one I install in every org I have managed now for a number of years!! It is a great tool to help analyse and understand just which fields are being populated and used by your end users. It is worth noting, that if you have a field that is always updated automatically by a trigger/workflow… Then it will obviously show as being used, even if that trigger/workflow update isn’t actually required. But overall it will help you understand your org in very tangible way.
• The Permissioner – can help you when mass assigning/removing Permission Sets from your users.

From Trailhead:

Additionally the ladies have set up an Admin Trailmix.

This covers a number of modules covering: Salesforce Profiles/Permission Sets, User Authentication, Data Quality, Data Management and finally Reporting & Dashboards.

Extra, extra! Two more tools to add to your Salesforce Cleaning toolkit…

Now for the bonus round.

There are always so many tools and ideas out there helping admins when maintaining a clean Salesforce org. And I am only skimming the surface with these next two tools…

Compare Permission Sets & Profiles

When watching the webinar, albeit not live, I started shouting at my screen.

During the Security section, there was a point around Profiles/Permission Sets. As an admin it is a mammoth task to compare all profiles/permission sets and what they might grant access to within your org. This can be kryptonite to Salesforce Superadmins…

There was a recommendation to switch off Enhanced Profile View, and then compare the permissions… But why do that? Especially, when there is a secret weapon at your disposal?

Perm Comparator by John Brock is that secret weapon!

Seriously… More people need to know about this tool! And I am not even on commission! 🙂

Stop duplicates in their tracks…

Salesforce hasn’t always been an admin’s best friend when it came to cleaning an org…And without getting all ‘back in my day’-ish…

But there was a time Optimizer, Security Health Check and those tools simply didn’t exist.

There was also a time Salesforce didn’t have an easy way to prevent duplicates… Admins had to either buy other tools to identify and manage duplicates, or create complex formulas and validation rules to try and prevent exact match duplicates.

But when planning your strategy for maintaining a clean Salesforce, you should investigate the in-built duplicate management tools from Salesforce.

After all, what good is cleaning up your security (profiles, access policies, passwords) and clearing out fields you don’t use any more – if your end users are still swimming in duplicate records?!

The in-built feature will take care of the basics, but depending on your use case, there may still be a reason to buy a tool like Cloudingo or DemandTools (just to name a few).

What is in your toolkit?

As I mentioned I only skimmed the surface here… And this is a topic I can (and will likely) write more about in the future. I have rambled more than enough for now…

So to wrap up the post, feel free to add any other suggestions or recommendations for your ‘Cleaning Salesforce Toolkit’ into the comments section below.

Simple put, GDPR is inescapable for those dealing with individuals within the EU.  In the last post we took a look at what GDPR is and how to start preparing for it.  Additionally, within the next month we also have Spring ’18 release going live.  So the good news is there is a number of GDPR features included to assist Admins prepare…

Recap: What is GDPR?

General Data Protection Regulation, otherwise known as GDPR, is the new privacy regulation coming into force in 2018.  It will replace the EU’s previous Data Protection Directive, and align privacy laws and regulations across all EU member states.

Part of the GDPR is the retention and removal processes re: personal data after a period of time.  Companies should only retain personal data for as long as needed.  So for example, if you have contact details of a person which is linked to contract data you may need to retain it for 7 years.  Versus simply having a contact on a mailing list, retaining the data for this long may not be justifiable.

For more details, take a look at my last post for the what/when/how of GDPR.

GDPR: Be Prepared

GDPR is ultimately an update to a number of data/privacy regulations across the EU.  But the easiest place to start for most companies is to map out and understand the following: 

• where your data comes from. (eg web-to-lead, email-to-case, data.com, users entering in the data),
• how it gets used/stored. (eg are there integrations which also use the data, is it stored outside of Salesforce, are records stored in standard and custom objects within Salesforce),
• what business processes which clean up or remove data. (ie how long does your company need hold onto personal data for (and for what purpose)? Is there already a process to remove personal data?)

Once you understand there where, how and what of your data, you will then understand where the following features and changes may help and support you.

Spring ’18 & GDPR Features

Data Privacy Records (aka Individuals)

Spring ’18 release brings with it a new setting under Company Profile within Setup.  The setting to enable Data Protection and Privacy, will expose the new object ‘Individuals’ within your org.

This new object connects to either a contact or lead record within Salesforce to store data privacy settings for that person. So for example you can store the customer’s preference re: soliciting products and services.

A key note here is that in the preview instances of Spring ’18, Individuals it isn’t treated like other objects and doesn’t have a standard tab available to access the object.

But a gentle word to note before we go through how to set this up.  As with any changes to a production system, it will require planning before setting this up to ensure it works for your processes.  A number of these changes below may also require assistance from a developer or changes to your marketing platform via API.  This is not entirely a point and click setup…

1. Activate the feature within Setup

Lightning: Setup -> Company Settings -> Data Protection and Privacy
Classic: Setup -> Company Profile -> Data Protection and Privacy

This will expose the new Individual object, with the standard fields within your org.  Most of these fields are created for the purpose of tracking opt-out requests and personal preferences.

Some examples include:

If a contact opts-out of geo-location tracking, you can store that preference here.

You can even store the D.O.B of the person and indicate if they are considered a minor or not.  (You might need to do this as minors carry different standards for elements like data retention under GDPR.)

And good news is you can also add custom fields to this object if you wish to.

2. Update Page Layouts

Add field to relevant page layouts on contact & lead objects.  Salesforce also suggests to rename the field label from ‘Individual’ to something more meaningful to your users.

3. Existing Contacts/Lead Records

There is a code-based solution to create data privacy records for existing contacts & leads within your Salesforce, which also includes code for handling person accounts.

I won’t go through the code here.  But if you are interested, you can take a sample of the code here.  Remember you will need to execute this somehow (ie start the code running), so you may need help from a developer friend(s).

4. New / Changes to Preferences

Finally, you will also need to plan how new records get managed within Salesforce.

Will an Individual record be created when a contact or lead is created?  What about field updates / etc?  Do you do this via a trigger in Salesforce or via API from your email platform?

And what do you get at the end of this?  A new record of course … 🙂

Data Quality & Duplicate Jobs

GDPR features in Spring also come in the form of expanded functionality.  Spring ’18 release provides an update for its out-of-the-box duplicate management to allow Admins to run a duplicate job.

Previously matching rules would run when a record was being added or updated by a user.  This would then either alert the user or block the user, depending on how the Admin had set it up.

But in Spring ’18 release, us Admins can now run a “Duplicate Job“.  This allows you to use (or create) a matching rule and then run a search within Salesforce to find duplicates.  Once a Duplicate Job completes, you will be able to take action by comparing and merging records, right from the summary page.

Changes to Web to Case and Web to Lead

Spring ’18 release also brings another GDPR feature.  This time there is a slight update to how failed Web-to-Case and Web-to-Lead emails are handled.

When a Web-to-Case and Web-to-Lead fails an email is generated automatically to advise.  This email will no longer include any personal data inserted by a third-party application.

Data.com for UK/Ireland Contacts

This only impacts users who are using Data.com Prospector or Clean.

Data relating to contacts stored within Data.com Connect (only UK & Ireland) will now be removed from the Connect database.

Also the Clean Status field on a Contact/Lead records will be changed to ‘Not Found’.

Wrap Up

As you can see there is a lot to cover, and Salesforce is enabling businesses to become GDPR compliant with all the GDPR features in Spring ’18.  There is still more to read in the Release Notes for Spring ’18 if you are using other tools like Pardot, Inbox or a developer (specifically around Event Logs).

There is support at hand, and if you haven’t already I would strongly recommend completing the Trailhead modules setup for GDPR Basics (and there is also a trail for US Privacy Basics too).

Additionally Salesforce has a new section in the Help for Data Protection & Privacy.

And finally there is a basic scenario guide, which looks at a number of common requests and suggested actions/things to consider to be compliant under the various privacy laws (including GPDR).

There has been a lot of attention surrounding GDPR.  But what is it?  Why does it matter?  How will GDPR affect your Salesforce?  What responsibilities do you have as a Salesforce Administrator?  And what can you do to prepare?  So many questions!  In this article we take a quick look at these questions and some additional resources available to you, when preparing for this change.

What is GDPR?

First up, let’s take a look into what exactly GDPR is.

General Data Protection Regulation, otherwise known as GDPR, is the new privacy regulation coming into force in 2018.  It will replace the EU’s previous Data Protection Directive, and align privacy laws and regulations across all EU member states.

It is primarily aimed at giving individuals more control over how their data can be collected, used, and stored.  While also extending the rights of individuals to have their data removed/deleted.

One key point to note.  Even though this is obviously applicable for all EU countries, it may also apply to any business which deals with personal data of an individual within these countries.  So if you are in USA, India or even Fiji – but dealing with personal data from an individual living within the EU – GDPR will still apply.

When will GDPR take affect?

The legislation will take affect from the 25th May 2018.

What is ‘personal data’?

Another thing to consider is how GDPR will redefine what is considered to be ‘personal data’.  Overall the definition has been clarified and it will include ways modern technology allow an individual to be identified.

Personal data, relates to any information which can identify an individual.  This could be either directly (for example a person’s name) or indirectly (a unique online identifier/ID or geolocation data).  The purpose of GDPR here is to clarify and provide stronger governance towards what is personal data.  Especially when compared to the preceding legislation.

What can you do to prepare?

At this stage it is worth pointing out this is something that potentially needs company-wide buy-in and support especially from your organisation’s leadership team.

Educate

The first step is to be aware of the changes, we have taken some of the first steps here.

Salesforce is trying to support its users through the pending change. The responsibility though relies on end-users to ensure compliance.  And to that end they have published a Trailhead module to support their users.

The Spring ’18 release also includes a number of changes to support users with GDPR.

Kick-off

If your company hasn’t started the process to confirm compliance, then it is worth having a discussion with your manager or with your companies designated Data Protection Officer.

GPDR is an enhancement from previous legislation so the changes required might be small, but it is important to understand how those changes could impact to your business.

For example, say your business relies on email marketing.  Your contact lists may not contain details on how the individual’s consent has been attained.  With GDPR, it may be worth investigating your requirements for that data and plan how to reconfirm or get consent.

Which leads us nicely onto the next step…

Map your data processes

A key to being able to understand any risks relating to GDPR and your Salesforce is to map out how data enters your Salesforce org.

Is it a web-to-lead form?  Or perhaps a landing page describing your products that tracks user behaviour via Marketing Cloud?  Or do your sales team import contacts via Data.com?

What happens to it once it is in Salesforce?  Is it stored as leads, contacts, person accounts?  Do you have custom fields on objects containing personal data?

This is a chance to get all these flows down on paper.  And remember to include if there are any prompts for user opt-in and what the purpose of the data is.

Are there any other resources to help me?

So as we wrap up here, it is worth mentioning that we have only just started to scratch the GDPR surface.

The changes come into affect in May, so there is still time to prepare.  And if you were compliant with previous legislation like the Data Protection Act here in the UK, then GDPR may be an extension of your existing processes.  But don’t take this as legal advice!

There are a number of great resources available to help companies with this transition.

For Salesforce users:

• as previously mentioned, I would start with Trailhead.
• Salesforce has now also published a Data Protection & Privacy guide in the Help section, and
• check the Spring ’18 release notes, as there are a number of features changing (or added to the platform) due to GDPR

Additionally, I found a number of site relating to GDPR when preparing this post:

• Data Protection People has a post called, ‘The perfect CRM system for GDPR compliance’
• Next port of call, the ICO has published a 12 step e-book relating to GDPR.
• SalesforceBen also has an e-book detailing GDPR.
• And finally (and if you are really interested) here is an 88 page document from the EU relating to GDPR

Data is such a big focus for anyone working with a CRM, Salesforce is no exception.  Previously we looked at migrating data into Salesforce.  But what happens when you need to remove data?  Cleaning a Salesforce org can present a few challenges.

Say you need to clean your Salesforce org and delete/archive, because…  you might have gone over your Salesforce data allowance?

This is what happened to us recently.  One of the orgs my team manages went over its allocated Data Space, and we started getting emails / calls from Salesforce to remind us that we have used up our allowance.

We had a choice, buy more storage (at Salesforce’s very inflated data prices).  I mean c’mon it is 2017, 1TB with Dropbox/Google/Onedrive/etc is only around $100/year…

As you can probably guess, this wasn’t our “go to” option, so we had to find out what we had in Salesforce vs what we needed, and then make a decision…

Legacy org & technical debt

The org in question was what I deem a “legacy org”, and has had presented a few challenges over the years.  It has been an active Salesforce org since 2003. And without constant love and attention has built up so much technical debt because it was never actively managed/improved/developed…

Even relatively simple things like enabling and rolling out Chatter were never done – even years after Chatter launched.

As the company has grown, obviously so has the data stored within the org.

Add into the mix that new apps installed in the org which have driven a sudden increase in the volumes of records being created (eg telephony integration and training people to log calls).  And hey-presto data storage & how we are using it is suddenly a priority.

Where is the data?

For an org which has grown from around 50-100 people in 2003, and a very simple business processes.  The question was how could we be using up out data storage suddenly?

Why now?  How do we get to the bottom of what is happening?

Admittedly it wasn’t something we had kept an eye on.  So the first port of call was the setup menu.

As you might know Salesforce offer a section in the Setup menu called ‘Storage Usage’, which is quite basic but gives you a snapshot of where your data and file storage is used.  To use it, go to Setup -> in Quick Find, and search for ‘Storage Usage’.

Boom, there it was.

The org with over 23,000,000 records…

The org had ballooned to over 23,000,000 records.  Shocking as our number of accounts are a fraction of that overall volume.

What was even more shocking for us was that it was two objects consuming almost 70% of the total storage!

The two objects in question: Tasks & Email Messages…

<sarcasm> Oh joy! </sarcasm>

Not all objects are equal

Why was I so *not* excited that it was Tasks & Email Messages using up our storage?

Salesforce Tasks & Archiving

Activities in Salesforce have a unique feature which means they get archived by Salesforce after a set number of days.  This is typically around 365 days of being closed (but there are a few caveats to that), and can also be extended if you request it from Salesforce.

This is an issue, as once archived you can no longer use the standard Salesforce reporting to analyse.

And due to the sheer volume of records (over 12,500,000), it was even crashing Dataloader/Workbench/Developer Console when trying to export the data.

When I did manage to get the export, by trying to filter by created year, the file was still too big to view in Excel.  Also we were limited to just Excel, which meant we hit a brick wall.

Email to Case & Email Messages

This org heavily relies on Email-to-Case.  And when received, the email is stored in the EmailMessage object.

Additionally all auto-response emails are also saved against the case, in addition to any replies from the customer.

Great for keeping track of all communication.  But once again creates some difficulties when trying to report and analyse.

Also as an email gets saved against a case, it also creates a task.  So we end up with a sort of duplication, with a task and an email message created and linked to the same case.

Getting the data out of Salesforce

In the end we contacted Salesforce Support, as we couldn’t use normal methods to export and clean the data.

The only suggestion that Salesforce could provide was to schedule a data export of the objects we wanted to export and analyse.  Simple enough…

Anyone who has used this feature will know the output of this is zip files, which contain CSV files inside.  Great it was going to be small enough to work with!

Nope… Each CSV was still over 1,000,000 rows.

Excel still was too unusable to analyse the data.  At this point I really was wishing for something like Access/MySQL to load the files into.

Enter CSVSplitter, a really simple tool that allowed us to split the CSV files down into smaller more manageable chunks.

Once they were broken into the smaller files, then we were able to start analysing the tasks.

Analysing the data

The road to cleaning a Salesforce org is paved with lots of data to analyse!

You need to understand what you have, before you can understand what you need.

So we started analysing the data.  And we dissected the data in many different ways to understand what was driving the volume we were seeing.

We looked at tasks created by month and year.  Were there specific users who created more than others?  Were there common subject lines – which might point to auto-generated tasks.

Our Salesforce, had never ever been cleaned.  And we have used tasks in the past to drive system automation within the business.  So where relevant, these records could go!

Archiving the data

So we had analysed the data, now to archive it somewhere in case we actually needed to reverse the process. (FYI – while researching this post, I found this useful guide to creating a Archiving Policy for your company).

Though storing the records outside of Salesforce and then trying to restore in the case of a profile would be painful if we had to.  But at least we had a fall back plan, and if needed the business could still use in reports.

In addition a lot of care was taken in being conservative with what we are removing and working with various stakeholders to ensure the different departments were on board with our plan.

Now we could start the actual cleanup!

Cleaning a Salesforce Org: 2,600,000 records deleted (so far)…

I have to admit, this part ended up actually feeling strangely cathartic.

Being able to delete over 2.6 million records from an org was also a first for me 🙂

We essentially identified the records to be deleted, creating a CSV file of the IDs of the records we wanted to delete, and then use Dataloader to remove the record.

Once we started with the tasks, we were able to then also move on to other records and start an overall clean out of Salesforce.  Opportunities, Accounts, Cases… All are now in scope and we have created a data clean-up roadmap and are making great headway.

We have a long way to go still, but at least we can start to let go of the legacy data past.

And most importantly, make Salesforce a focused CRM for the sales and customer service teams, so it is easier for them to use.

Got your own ‘lesson learned’?  Share your tips…

I have worked in org’s where hitting the data storage limit was expected and almost required.  As we deployed tools like FinancialForce which create a lot of records (and they need to).  So we simply bought more data.  But it really depends on your scenario, as every orgs needs are a little different.

Have you been in a similar situation?  How did you decide what to archive?  Did you use any specific tools to help you?

As part of this issue, we were able to make a business case for getting tools like DemandTools (paid app).

And I am currently investigating Passage Tech’s ‘Storage Helper‘ and ‘Rollup Helper‘ (both have a limited free version) to see if they can help profile accounts to then identify what records can still be removed and archived from Salesforce.  But I will save the details for another post later 😉

One struggle every Salesforce admin faces, is how to get users inputting the right data. Free form text fields with no format or structure to them. And then trying to report on these fields becomes a nightmare! So how do you then make sure your users are entering in the correct values? That is where the wondrous (yet often overlooked) field type called a ‘picklist’ comes in…

And picklists within Salesforce can help you structure data while guiding your users to complete fields as needed. While recent releases have also added to their versatility. In turn, this can help you as an Awesome Admin setup your Salesforce data more effectively.

So let’s go back to basics and dive in to Salesforce Picklists!

You keep saying picklist!!!!!

Let’s take a quick step back to make sure everyone has the same understanding of what a picklist is.
A picklist is ultimately a way to present several values to the user, and restrict what then select as a value in the field.
As an example, a status field when selected could show ‘To do’, ‘In Progress’ or ‘Done’ in the picklist.

Two types to rule them all.

Now we start getting into the fun stuff. And yes, I know I am a bit of a geek! In Salesforce there are two basic types of picklists available for an Admin to create, the regular ‘Picklist’ and ‘Multi Select Picklist’.

The primary difference is illustrated below, where a ‘Picklist’ only allows one value to be entered at a time, while Multi-Select picklists allow a user to select more than one value.

Flexibility with global control

Picklists and world domination… Not coming to a cinema near you!

Seriously though, as you can hopefully see picklists give you a fair amount of flexibility. But what if you want the same picklist options available across different objects in Salesforce?

As of Winter ’17 release, you can now define a common or global set of values for any picklist in your org to use. To do this for a new picklist you can go to:

In Classic: Setup -> Create -> Picklist Value Sets
Or in Lightning: Setup (via the little gear icon) -> Object and Fields -> Picklist Value Sets

And as added extra, the Summer ’17 release allows you to promote any existing picklist values and turn it into a global value set. Simply go to the picklist you want to promote, and you will see a button on the top of the page.

Picklist ‘gotchas’

Picklists are very handy in ensuring your users add the right value into the field. But because of the way they are structured there are a few things to remember when setting up and using picklists.

1) Record types:

Have you ever wondered why one user will see an option in the drop down, but someone else can’t see that option? The combination of record type and the profile of the user defines what she/he will see on-screen! (Don’t worry though, this one gets even the most seasoned of admins.)

2) Dependant picklists:

Alternatively, it could be a dependant picklist. This is where the value of one field, controls what options are available in the second field. An example of this could be on an Opportunity – where you have two picklists – the ‘Stage’ field and ‘Lost Reasons’.

You might want to display a list of ‘Lost Reasons’ if the user selects the ‘Stage’ of ‘Closed Lost’. Thereby allowing you to once again guide the user in what the valid options are.

3) Using Dataloader and picklists

Picklists are ultimately just text when Salesforce looks at it at the database level, so be mindful that when loading in values via Dataloader.

Any new values in the file being loaded into Salesforce may appear unless you tick the ‘restrict picklist values’ option on the field (or setup a validation rule).

4) Picklists in formulas

Picklist values are technically stored as a text value in Saleforce, however when using them in any advanced formula within Salesforce, like a validation rule, remember to use the function ‘ISPICKVAL()’.

For example, say we have a picklist field with the name ‘Industry’ and you wanted to see if a picklist value was set to ‘Financial Services’, you would use the formula like so: ISPICKVAL(Industry, “Financial Services’)

So there we have it. In a few short minutes, we have covered the basics of picklists! Exciting right 🙂 Now go forth and structure your data! Any business that values data quality will thank you for it!

So we are almost ready to go.  In this data migration series, we have already looked at how to migrate data and some tools to help with the migration.  But now for the fun part, I want to share with you some data migration tips to help make your migration into Salesforce as smooth as possible.

Let’s jump right on in, and please share any of your own tips in the comments section below.

Data Migration Tip #1: External IDs & Upserts

Let’s walk through a common scenario in a data migration…

The Data Migration nightmare:

You are about to migrate 12 different objects into Salesforce, everything from Accounts, Contacts and Tasks through to Opportunities, Cases and a number of custom objects you have created in your org.

How do you create these records in Salesforce, and still have all related records end up linked together?

Do you insert the first object (let’s say Accounts).  Because you need to link the Contacts to those Accounts, in the second file do you use Excel to create the link?  Do you take the import file from Dataloader, run a VLOOKUP in the Contacts file and then pull in the related Account IDs?  YUCK!  Now imagine having to repeat that process for all 12 objects…

I have been there myself.  And when it is a very large migration it can become extremely difficult to track and manage all the spreadsheets.  Especially when you are working with live data, that is always changing and is never static.

This scenario can be the nightmare of any data migration but it doesn’t need to be.  #NoMoreVLOOKUPS

Salesforce External IDs:

As you know, each Salesforce record is given a unique 15/18 character ID.  That is great, if you are only dealing with data in a single system or Salesforce org.

But what happens when you are need to store the ID of a record, that is actually sourced from a system outside of your Salesforce?  For example an ID from an invoicing system like SAP or Oracle.

This is where External IDs come into the picture.

By configuring a field to be an External ID you can then use it to store the original reference/ID from the other system.  Additionally, you can also use it to prevent duplication within Salesforce by setting the External ID field to unique.

It is this ‘uniqueness’ that helps us out when migrating bulk data into Salesforce.  As Salesforce will automatically index the External IDs, allowing you to use this as a reference when doing an Upsert within Dataloader.

#NoMoreVLOOKUPS: Upserts with External IDs:

Now let’s take a quick look at Upsert within Dataloader.  Upsert combines the ability to either Insert/Update records while using a single file.  However, using Upsert in Dataloader also gives you access to a few extra features.

After you click Upsert and start going through the Import Wizard, you will get the ability to select either the Salesforce ID or an External ID (if available on the object).

Additionally, for any relationship you have on the object, Dataloader will also prompt you to select the applicable ID fields which is also referenced in our import file.

This is how you avoid using VLOOKUPs!  

By populating an External ID, on each record you load into Salesforce you can then use this to get Salesforce to do the lookup for you.  Even if you don’t actually have an External ID from another system, you can create your own.  Just make sure it is unique for each record.

I know this is a little confronting to start with.  It took me a little while to wrap my head around just how exactly to do this when I ran my first data migration.  However Upsert & External IDs can save you so much time, I strongly recommend you practice and learn how to use this ‘double act’ to your advantage 🙂

There is of course more resources available on this topic.  Salesforce Help has an article which gives you a step by step process you can follow.  I also found a video series on YouTube by Doug Ayers, which really breaks this down over an informative video tutorial.

Data Migration Tip #2: Update System Audit Fields (Salesforce’s ‘hidden’ trick)

In Salesforce, every record has a number of system date fields which typically can’t be edited.  Fields like ‘Created Date’, ‘Created By’, ‘Last Modified Date’ and ‘Last Modified By’ are locked down (and for good reason!)

These are your standard audit fields and are very important when tracking a record’s history.  While planning a data migration, you might come to realise that you need to retain this information from the previous data source.  And that is where the permission ‘Update System Audit Fields’ comes into the story.

Imagine a sales person’s activity report, which is displays only activities created in this quarter.  As such after a data migration is complete, you still want your users to run these reports.  So what can you do?!

In the past, being able to edit these fields was part of what I called the ‘hidden Salesforce toolkit’.  You know the list of things that Salesforce can do, but they don’t really publicise? Like increasing the cheeky workflow limit here or there…  But all part of the Salesforce Support toolkit, but you had to ask for it to be enabled in your org first.  The problem was, you had to know them in the first place you could even ask, so that you could get it requested to be turned on!

Winter ’16 Release & Update System Audit Fields

In Winter ’16 Salesforce made a number of these ‘hidden’ features publicly available.  And with a few clicks you can now enable it easily for your data migration.

There are a few considerations to look at before you enable the ability to update the System Audit Fields, so check them out here.

And then head over to here, for instructions on how to setup and enable this feature in your org!

The biggest point I will highlight about accessing these fields, is that they are only available on record creation via Dataloader.  Meaning you can’t insert a record and then try to update the created date on it separately.  If you did try to edit/update these fields after a record has been created you will get an error.

Data Migration Tip #3: Automation Anchors

Automation with Salesforce is fantastic.  Admins and Developers have access to a wide array of automation tools within most Salesforce orgs, including Process Builder, Triggers, Workflows, Sharing Rules.  But when you are migrating data, they can quickly become an anchor.

What do I mean by that?  Do you know what automation will fire off based on the data work you are doing?  For example, is there a workflow that will send an email alert to another team, or even the customer directly?  Is this what you expect to happen, if so that is OK.  But you do need to make sure you know what any data you load or update into the system will do to the existing automated processes in your org.

In addition, automation in Salesforce can end up adding extra time to any data insert, update or upserts.  Remember even deletion of records can fire off a trigger!  When you are moving or manipulating a bulk volume of records, this time can easily add up.  Complex sharing rules might really slow down an insert/update of records.  And depending on the volume of data you are working with, you of may end up hitting some of Salesforce processing limits.

So before doing any data migration, test and make sure the automation you expect to work does.  Otherwise see if you can deactivate the workflow while you are doing the migration.

Share your tips

This is by no means a definitive list, and there is plenty more to come.  But in the meantime, please share your own experiences or tips in the comments section below!

Happy migrating!

Everyone loves data!  In my last post I shared with you was some thoughts on how to make sure a data migration goes smoothly.  There is a treasure trove of tools out there that can help you with any Salesforce Data Migration (more than I can put in this single post!).

In this post, I am going to share some of the most popular tools to help you out with any Salesforce Data Migration, and best of all they are all free!

Tools to help you create & edit records in bulk

Salesforce has a good data import wizard which can guide you through basic record inserts, like when you need to create a bunch of new accounts/contacts.  But what happens when you need more power?

First thing is first, the tools below will respect any object/admin permissions you have within Salesforce org.  Meaning if you don’t have access to create records on a certain object, you won’t be able to create records via one of these tools.

1) Dataloader

This is a tool developed by Salesforce themselves.  Once installed on your PC, you have a powerful tool to help you to migrate or update data in Salesforce.  The interface itself is a bit basic, but it is relatively straightforward to use and allows you to either Insert, Update, Upsert, Delete and Export records from any object in Salesforce that is available via the API.

Most of the features of Dataloader are available through a simple user interface, but if you want to unleash even more power there is also an advanced option allowing you to use a command line interface.

It is worth mentioning that Apex Dataloader only supports .CSV files.  So if you have Excel/Google Docs, make sure you save the file as a .CSV file before starting your Salesforce data migration!

Final thing to point out, this tool uses SOAP API, which means you need to be on Enterprise, Unlimited or Performance Editions of Salesforce.

To get it, once logged into Salesforce, go to Setup -> Data Management -> Data Loader.   Once there you have options to download the installation file for Windows or Mac.

To read more about Dataloader, click here.

2) Jitterbit Data Loader

This is a tool is similar to Apex Dataloader in most regards.  The basics are relatively similar – you can still Insert, Update, Upsert, Delete records though has a few extra tricks up its sleeve!

First up Jitterbit’s tool supports more versions of Salesforce, with Group Edition the minimum requirement!  Meaning many more people can use this tool versus Apex Dataloader.

Secondly, this tool supports more than just CSV files, it works with any flat file or even a database connection for the more advanced users out there.

You can also perform basic transformations with your data on the fly, and link your configuration settings to the Jitterbit Cloud.

It also has a relatively simple point and click interface with a helpful wizard to guide you through the steps.  Where to get it?  Head on over to Jitterbit’s website to register, download and install.

3) Field Trip

In my last post, I made reference to making sure you understand the data you are migrating.  Question, analyse, question and repeat!

Field Trip is a free tool available on the AppExchange, which helps you understand the data completion rates already in Salesforce.

Why is this helpful? You can view which fields actually contain data, represented as a percentage across the entire data set within an object.  This is very handy if you are looking at removing technical debt within your org, or trying to understand what fields are most used.

Imagine you are reporting on the Account table.  A field that has 100% completion rate means every single record has a value in that field.  This enables you to then analyse the fields that are most used and ignore the fields which aren’t.

Go forth and migrate!

We are only just skimming the surface here, and there are plenty of other tools you can use for a Salesforce data migration.  As an example, Salesforce also have Workbench which is an web-based version of Dataloader, with a few differences.

Everyone has their own favourites, so please feel free to share yours in the comments below!

In my next few posts, we will cover some popular tips to use when migrating data.

There comes a time in every administrator’s life, where you might have to migrate data into Salesforce.  Why is that important?  Data is the core of any CRM platform and has the power to become a blocker for your customers and/or internal users!

A request to migrate data could be as simple as someone asking you to import data into Salesforce from a spreadsheet.  Or it could be due to the retirement of another system.  Even a merger/acquisition!  But when the time comes, you will need to be able to really get under the hood to understand the data and create a data migration plan.  Because without a plan, migrating data can become a massive risk to any project.

Over the next few weeks, I am going to dive into a number of topics relating to all things data migration.  YAY! I hear you screaming! Also please feel free to share any of your experiences in the comments below, as with all things data migration there is always plenty to learn!

So let’s get started and dive into the key things to plan for when you are attempting to migrate data into Salesforce.

Everyone loves data

Data is the lifeblood of any business.  And your data in your CRM is no different.  Good data can empower any business.  While on the flip side, data can also become a blocker for your teams when not managed well.  But the good news is this shows your users have something at stake in making it better.  These are the same people who can become your data champions!

To further the point I am trying to make around identifying your stakeholders.  Your end users play a big part in advising you on what the data actually means (for them).  And it is here you will also need to identify other potential stakeholders.  Does the data have an impact on sales reporting?  How about on your company’s financial reporting?  Or is it used to help serve or fulfil your customers.

I have a question for you ma’am!

As part of any data migration, you will need to get up to speed on understanding the data.  Understand and interrogate the old data.  Analyse it.  Test out any assumptions you might have.  And also think through how you want the data to look once the migration has been completed.

This is a time you are going to need to ask a lot of questions and ensure you (and your stakeholders) fully understand the definitions being used.  If there is something that doesn’t make sense, ask questions!  Questions are your friend!  Really drill down and confirm your understanding.  Leverage the experience of your stakeholders, bring them along the journey too.  And make a note of these questions are you get answers.

One lesson I have learned over many data migrations is no-one has all the answers unfortunately.  Most times you have to really dive into the data and test what you have been told.  This is one area that is a big risk to any project and can end up causing a lot of pain no matter how well planned!

I have been on the tail end of a few really complex data migrations.  Even with the best planning and stakeholders, there has always been an inconsistency somewhere in the data which has you screaming ‘WHY!!!!’   Especially if you are dealing with an older CRM!  (but that is a story for another time!).

The point I am trying to make is it has been down to the project team to really untangle and interrogate the data to make sure it makes sense.

Prepare, verify, test

One you understand the data, you can plan to migrate the data!  There can be a lot to think about at this stage.

In your preparation, you need to map out where you want the data to go.  Are there new fields created to support the migration or are you mapping to existing fields in Salesforce?  Does the data-type match (eg does a number field map to another number field)?  Do you have any validation rules or mandatory fields that might get in the way?  If so what is your plan to mitigate that?  Do you update the data before you move it?

Some of the biggest problems I have encountered in a data migration have all stemmed from a lack of full testing in a sandbox.

If you have access to a sandbox, plan out your testing.   Even if you have Professional Edition, you have access to at least a Developer Sandbox (check here).  Part of your testing needs to include verifying the actual data.  Does to match what you had in the data source?  Do any lookups to other records link up correctly?  Are the number of records the same?  All those questions you wrote down while understanding the data, can be very useful in the testing phase.

I have thrown a lot of questions around there, but hopefully you get an idea of some of the things to think about in this stage.  Testing is another great time to re-engage with your stakeholders.  Heck, even get them to login and run a few test scenarios on the migrated data.

Once you have tested and verified your data, you can start planning the go live and move into production!  Your almost there!

Step back in time

With the actual data migration into a production / live system, you should plan in advance on how you intend to roll back any changes you make.  Even after testing in a sandbox, there could be something which causes you to halt the data migration.  And if that happens, you need to be able to restore the system to how it was before you started.

Take a backup, run reports, hold onto the import files from Dataloader.  Whatever it is, just know how you can go back without causing further issues.

This is doesn’t need to be overly complex, but it will depend on the size of the overall size of the data migration.  Sometime it can just be a simple plan to say ‘if this happens, this is how we move forward’.

Press the button & migrate data!

This is the fun part, and all your planning comes together.  This is where you start the migration and hopefully completed the migration as planned.

Once you have migrated the data into Salesforce, test and verify again.  Repeat the steps you did after you loaded the data into the sandbox.  Test and verify, test and verify!

Are you finished yet?  Good.  But did you disable any workflows/validation rule/mandatory fields?  Did you remember to turn them all back on?

And finally, and most importantly.  Have you communicated the successful data migration out to your stakeholders!? 🙂

Do you have any lessons learned?

As mentioned at the start of the post, I feel there is always soomething new to be learned everytime you migrate data.  One of the projects I am working on at the moment is pushing me to learn something new every day! (More on that in another post though).

Sharing is caring 🙂 Do you have any tips or things you like to plan for?  Share them in the comments section below!

Data security, sharing rules, org-wide defaults, role hierarchies – are all parts of a bigger puzzle, which can feel like a maze sometimes.  It isn’t a sexy topic by any stretch of the imagination, yet defining a robust data security policy is fundamental to all Salesforce orgs.  Let’s dive into what data security is, and how to set it up within Salesforce.

Data security in Salesforce

Data security within Salesforce has a number of facets to it.  You have varying levels of security controls, which can seem quiet complex to start off with.  Though it is this flexibility which actually gives you a lot of control, allowing you to show a specific object/record/field to one group of users while hiding it from another.

The most common form of data security is controlling access to an object or field through the user’s assigned Profile/Permission Set.  For example, you set a user’s access to an object by defining if their profile can either Create, Read, Edit or Delete.  This is often referred to as CRED permissions (or CRUD where the U = update).

Object access is the first level of security a user will need to pass.  If a user doesn’t have access to an object, they will never see any records.  This gives you a sledge hammer approach to control access, but what happens if you need to finesse this?

Say you want to control access to a record based on a specific field value and then share that with certain groups of users?  Controlling a user’s access to an object doesn’t really help you out, does it?

This is where your Salesforce instance’s Org-wide Defaults (OWD) & the sharing rules come into play.

Org-wide Defaults (OWD)

Org-wide Defaults or OWDs for short, is the next level of security and will allow you to define what the default level of record security is per object.   If you were to try to visualise it, most people draw it out as a funnel.  This is because you first start by restricting the Org-wide Defaults of an object, and then extend access from that default.

What do I mean by first restricting access?  In Salesforce you have three levels of default record visibility per object:

• Public Read/Write – all records are publicly available to read & edit for all users who can access the object.
• Public Read Only – all records are visible for all users who can access the object, but not edit.
• Private – no records are visible/editable for users who can access the object.

Now the above rules have two key exceptions. The first is when the object in question is part of a Master-Detail Relationship.  If it is and it is the child object, it will inherit the OWD from the parent object(s).

The second ‘gotcha’ is when the user is an owner of the record in question.  Ownership of the record will grant access to the specific user who owns the record, providing their profile grants them access to the object.

Role Hierarchy

Role Hierarchies within Salesforce play an important part in record visibility.  As they allow you to build in general record visibility for those higher up the hierarchies branch.  The most common example would be when you want a manager to see what their team is doing within the system.

This is important to note as when you set your OWD to either Public Read Only/Private, the role hierarchy needs to be taken into account to ensure that relevant hierarchy above the owner of the record still have the relevant access.  When setting the OWD, you do this by ensuring ‘Grant Access Using Hierarchies’ is enabled.

(For more information on Role Hierarchies, Salesforce Help & Training article)

Sharing Rules

Now we have built the foundations for who can see what, we need to extend access as required to the various types of users.  This is where Sharing Rules comes into the picture, they help you extend the default settings from above.  One example could be where a team of sales users needs access to each other’s records.  If the OWD is Private, you would need to extend the sharing of records within that team.

Sharing Rules within Salesforce give you two types of control, based on either the:

• Record owner (eg user ‘Sharon Williams’ owns the record), or
• Criteria based sharing (eg if field ‘Record Type’ equals ‘North America’)

This allows you to define how a record gets shared.  And this can be to a group, specific role or a role and all of its subordinates in the hierarchy.  You then pick how to the level of access to finish up (read only or read/write).

There are some nuance’s with sharing rules, and limits are enforced on the number of rules you can have by Salesforce.  For example you can have up to 300 sharing rules on an object, with 50 criteria based sharing rules.  And only specific field types can be used for Criteria based sharing rules.  To view the considerations, go to the Help & Training article for more.

Manual sharing

Manual sharing is an additional control which allows you or the user to manually share a record.  This is done by clicking the ‘Sharing’ button on a record (if it is on the page layout).  This is where you can also see the sharing detail, which will display who has access to the record, what that access is and why they have that access.

Final words

Hopefully that helps you control your org’s data security, as you can see there is a lot of detail and thought that goes into ensuring the right people see the right data.  As an additional topic, there are also Apex Sharing rules which can give you even more control over who can see records, but I won’t cover these in this post.

There are also some great resources are available from Salesforce if you want to dive into this topic further.  I found this video walk through provided which is ~5min video.  It shows you how to setup the rules we discussed here in Salesforce Classic. There is also an awesome Trailhead on the topic of Data Security, which you can find here.