data migration

  • A Tale of Salesforce API Limits

    Everyone loves data, I am sure of it…  The key difference though, is most people in their right mind, simply wouldn’t admit it.  Data is useful, particularly in this day and age, in what is now commonly referred to as the Information Age.

    Now, you might be thinking I am crazy – and that is probably true – but data is the life-blood of modern-day organisations.  So what happens when can’t query, transform or extract your data because you hit Salesforce API Limits?

    gasp in horror, hitting API limits during a data migration

    Intro to APIs….

    Ok, ok – I hear you.  What the hell is an API limit?  For that matter, what the f*** is an API?  And why should you care?

    Starting at the beginning (feel free to skip this section if you already know this!)…

    An API, or Application Programming Interface, is at its most basic a way programs or systems talk to each other.  I am way over-simplifying things here.  But an API ultimately allows one piece of software to communicate with another piece of software in an agreed way.

    data apis, allow computers to talk to each other
    APIs allow computers to talk to each other

    Software, like Salesforce, is built to be a relatively open platform.  And they do this by using a number of APIs to allow other bits of software to communicate and link to it.  These links allow developers to do things like query, extract, create or update records within Salesforce with relative ease.

    Salesforce supports SOAP API, REST API, BULK API, Streaming API, Metadata API, etc….  So there is a number of ways to access what you need.

    Tools you may use as an Admin everyday, use a variety of these APIs.  For example, DataLoader uses the SOAP/REST/BULK APIs depending on its setup.  Likewise Workbench supports APIs too, as it can connect and manipulate data, query the fields, test API connections within Salesforce.

    Even tools like Salesforce for Outlook use APIs to query records and insert attachments.  And as an admin this is why you should care.  These tools, along with many others, use APIs to connect to Salesforce.

    But with great power, comes great responsibility…

    So with all these connections into and out of Salesforce globally, surely that can slow down the platform?

    That is what brings us to the Salesforce API limits.

    Salesforce is a multi-tenanted platform, meaning no-one company has an exclusive server/computer/machine setup for them, everyone shares the infrastructure which drives the platform.  So being cloud-based, Salesforce needs to ensure the performance of the platform works for everyone.

    To do that, Salesforce impose limits.  Governor Limits, Storage Limits, Per User Limits….  All based on the version you have bought from them.

    These limits are designed primarily to cap usage of the platform, ensuring stability of the platform and still give a fair amount of flexibility.  All the while, also providing a potential upsell opportunity for extra Salesforce services (eg Data Storage limit).

    Salesforce API Limits

    Among these limits though are API Limits.  And even though you normally shouldn’t notice there are limits in place, when you do hit the limits – especially when you hit them unexpectedly – it can be a little jarring…

    System.Web.Services.Protocols.SoapException: REQUEST_LIMIT_EXCEEDED: TotalRequests Limit exceeded.
       at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)

    This is the error which stops all work via APIs… If you rely on APIs to connect to your systems – you really don’t want to see this message!

    In this example, a connection using the SOAP API to connect into Salesforce has reported the Request Limit has been exceeded…

    Did I do that? API Limit in Salesforce hit...

    Diagnosing API usage…

    As an Admin, to diagnose the issue the first port of call is to analyse and find out how you hit the limit.

    Every org has quick look view at the API limits.  So go ahead – check for yourself.  Go to Setup | Quick Find | System Overview to see what you have within your org (if you have access to the Setup/Config of your org that is…).

    Salesforce API Usage within System Overview
    Not quite there yet… But closing in on the daily API limit!

    Firstly, all this really confirms is what has happened.  Like a car not starting, you have confirmed there is a problem – but you haven’t diagnosed the issue.

    When is 24hrs, not 24hrs?

    And secondly, when is the 24hrs?  Is it a rolling 24hr period, is it the start of each day local time or even the start of the day in San Francisco?

    Well, the good news is this limit is a rolling 24hr period.  Meaning it shows you the last 24hrs, from the time you are checking the limit.  This is important to know, if you have a lot of batch processes which might be scheduled at specific times of the day.

    Is that all the information you need?

    Does that really help you though?  Overall it doesn’t really give you much insight as to what has really happened within your org.

    So, what about Company Information?  There is an API Request Limit there too…. (Go to Setup | Quick Find | Company Information)

    API Limits at 99%
    Company Information in Setup

    Nope… still not much help!  In this example (yes, two different orgs), we can see the API Limits is 98.9% at the limit for the day.  So you will likely start to see problems with API connections…

    API Usage Last 7 Days

    On the Help page, Salesforce referenced an ‘API Usage Last 7 Days’ report, but didn’t mention where I could find the actual report!

    The help article advises that you need: ‘View Setup and Configuration’ permission on your to see this report.  I definitely have this permission within the org…. But I still couldn’t find the report: 

    Then I was reminded about the fact Salesforce comes with some generic but useful standard reports.  And guess what is included in the folder!?  Yep, an API Usage Report 🙂

    But you need to go via the Administrative Reports folder in Classic to view it…  Go to Reports | Folders | Administrative Reports:

    API Usage Report in Salesforce
    Salesforce API Usage in Last 7 Days Report

    If that doesn’t work for you, you can also use this URL hack (thanks to Joshua) to jump to the report within your org:

    https://na1.salesforce.com/00O?rt=104&retURL=%2F00O&c=UN&c=FULL_NAME&c=EM&c=CID&c=TS&c=CC&duel0=FULL_NAME%2CUN%2CEM&scope=organization&details=yes
    Replace the na1 with your instance.  Below is the same url without the na1.salesforce.com
    	
    	/00O?rt=104&retURL=%2F00O&c=UN&c=FULL_NAME&c=EM&c=CID&c=TS&c=CC&duel0=FULL_NAME%2CUN%2CEM&scope=organization&details=yes

    Diagnosis, complete…

    Well… Almost….

    The report gives you a great breakdown to see what/who is using up your Salesforce API Limits over the last 7 days.  But it doesn’t fix the overall issue.  The fix itself can be as complex as the software/tool that is causing the API Limit to be used up!

    But at least you can now diagnose the issue and decide if you can fix the issue, alternatively if it is something you aren’t likely to be able to fix/reduce you can sometimes buy additional API Limits from Salesforce by speaking with your AE…

  • How to migrate data into Salesforce

    There comes a time in every administrator’s life, where you might have to migrate data into Salesforce.  Why is that important?  Data is the core of any CRM platform and has the power to become a blocker for your customers and/or internal users!

    A request to migrate data could be as simple as someone asking you to import data into Salesforce from a spreadsheet.  Or it could be due to the retirement of another system.  Even a merger/acquisition!  But when the time comes, you will need to be able to really get under the hood to understand the data and create a data migration plan.  Because without a plan, migrating data can become a massive risk to any project.

    Over the next few weeks, I am going to dive into a number of topics relating to all things data migration.  YAY! I hear you screaming! Also please feel free to share any of your experiences in the comments below, as with all things data migration there is always plenty to learn!

    So let’s get started and dive into the key things to plan for when you are attempting to migrate data into Salesforce.

    Everyone loves data

    Data is the lifeblood of any business.  And your data in your CRM is no different.  Good data can empower any business.  While on the flip side, data can also become a blocker for your teams when not managed well.  But the good news is this shows your users have something at stake in making it better.  These are the same people who can become your data champions!

    To further the point I am trying to make around identifying your stakeholders.  Your end users play a big part in advising you on what the data actually means (for them).  And it is here you will also need to identify other potential stakeholders.  Does the data have an impact on sales reporting?  How about on your company’s financial reporting?  Or is it used to help serve or fulfil your customers.

    Everyone loves data! Source: dilbert.com

    I have a question for you ma’am!

    As part of any data migration, you will need to get up to speed on understanding the data.  Understand and interrogate the old data.  Analyse it.  Test out any assumptions you might have.  And also think through how you want the data to look once the migration has been completed.

    This is a time you are going to need to ask a lot of questions and ensure you (and your stakeholders) fully understand the definitions being used.  If there is something that doesn’t make sense, ask questions!  Questions are your friend!  Really drill down and confirm your understanding.  Leverage the experience of your stakeholders, bring them along the journey too.  And make a note of these questions are you get answers.

    One lesson I have learned over many data migrations is no-one has all the answers unfortunately.  Most times you have to really dive into the data and test what you have been told.  This is one area that is a big risk to any project and can end up causing a lot of pain no matter how well planned!

    I have been on the tail end of a few really complex data migrations.  Even with the best planning and stakeholders, there has always been an inconsistency somewhere in the data which has you screaming ‘WHY!!!!’   Especially if you are dealing with an older CRM!  (but that is a story for another time!).

    The point I am trying to make is it has been down to the project team to really untangle and interrogate the data to make sure it makes sense.

    Prepare, verify, test

    One you understand the data, you can plan to migrate the data!  There can be a lot to think about at this stage.

    In your preparation, you need to map out where you want the data to go.  Are there new fields created to support the migration or are you mapping to existing fields in Salesforce?  Does the data-type match (eg does a number field map to another number field)?  Do you have any validation rules or mandatory fields that might get in the way?  If so what is your plan to mitigate that?  Do you update the data before you move it?

    Some of the biggest problems I have encountered in a data migration have all stemmed from a lack of full testing in a sandbox.

    If you have access to a sandbox, plan out your testing.   Even if you have Professional Edition, you have access to at least a Developer Sandbox (check here).  Part of your testing needs to include verifying the actual data.  Does to match what you had in the data source?  Do any lookups to other records link up correctly?  Are the number of records the same?  All those questions you wrote down while understanding the data, can be very useful in the testing phase.

    I have thrown a lot of questions around there, but hopefully you get an idea of some of the things to think about in this stage.  Testing is another great time to re-engage with your stakeholders.  Heck, even get them to login and run a few test scenarios on the migrated data.

    Once you have tested and verified your data, you can start planning the go live and move into production!  Your almost there!

    Step back in time

    migrate data, but only with a backup plan
    Back to the Future

    With the actual data migration into a production / live system, you should plan in advance on how you intend to roll back any changes you make.  Even after testing in a sandbox, there could be something which causes you to halt the data migration.  And if that happens, you need to be able to restore the system to how it was before you started.

    Take a backup, run reports, hold onto the import files from Dataloader.  Whatever it is, just know how you can go back without causing further issues.

    This is doesn’t need to be overly complex, but it will depend on the size of the overall size of the data migration.  Sometime it can just be a simple plan to say ‘if this happens, this is how we move forward’.

    Press the button & migrate data!

    This is the fun part, and all your planning comes together.  This is where you start the migration and hopefully completed the migration as planned.

    Once you have migrated the data into Salesforce, test and verify again.  Repeat the steps you did after you loaded the data into the sandbox.  Test and verify, test and verify!

    Are you finished yet?  Good.  But did you disable any workflows/validation rule/mandatory fields?  Did you remember to turn them all back on?

    And finally, and most importantly.  Have you communicated the successful data migration out to your stakeholders!? 🙂

    Do you have any lessons learned?

    As mentioned at the start of the post, I feel there is always soomething new to be learned everytime you migrate data.  One of the projects I am working on at the moment is pushing me to learn something new every day! (More on that in another post though).

    Sharing is caring 🙂 Do you have any tips or things you like to plan for?  Share them in the comments section below!

Back to top button