Because copy-pasting SOQL into Data Loader for the fifth time this sprint shouldn’t be a viable workflow.
If you’ve followed along on this blog, you know I have a fondness for working with Salesforce data locally. Back in 2024, I wrote about using SQLite3 as a staging area for Salesforce data work, and earlier this year I covered Jupyter notebooks as a discovery tool for new org engagements. The through-line in both posts was the same idea: pull data out of Salesforce, work on it locally, and push the results back.
The Python approach works well. simple_salesforce is a solid library, and pandas makes data manipulation pleasant. But there’s always been some boilerplate to write: authenticate, describe the object to get field names, build the SOQL query, call the Bulk API, juggle the CSVs, load them into SQLite, remember what you did and in what order. Do it for one object and it’s fine. Do it for a dozen objects as part of a repeatable data migration? It starts to feel like homework.
So I built bulkyard, a Salesforce CLI plugin that handles the extract/load cycle between Salesforce and a local SQLite3 database. It’s open-source, installable in seconds, and driven by a YAML config file that makes multi-object operations repeatable by design.
If you have the Salesforce CLI installed, adding bulkyard is one line:
sf plugins install bulkyard
That’s it. No virtual environments, no pip install, no requirements.txt to manage.
The heart of bulkyard is a YAML configuration file that describes what you want to extract or load, and from/to where. Here’s a simple example:
database: bulkyard.db
objects:
- sobject: Account
query: "SELECT Id, Name, Industry, Website FROM Account WHERE CreatedDate = LAST_N_DAYS:90"
- sobject: Contact
query: "SELECT Id, AccountId, FirstName, LastName, Email FROM Contact WHERE Account.CreatedDate = LAST_N_DAYS:90"
This tells bulkyard to extract two objects into a local SQLite file called bulkyard.db. The tables are named after the SObject by default, though you can override that too.
The value of a config file over a one-off script makes it so that, six weeks from now, you (or a colleague) can open the YAML and immediately understand exactly what was extracted and why. If you can’t run it again cleanly, it’s not a workflow.
With the config in place, extracting is straightforward:
sf bulkyard extract --config-file config.yaml --target-org my-sandbox
bulkyard uses the Bulk API 2.0 under the hood. It describes each object to build proper type mappings, creates typed SQLite tables, and inserts records in transactional batches of 1,000 rows. You end up with a local .db file that you can query with any SQLite tool (DBeaver, the sqlite3 CLI, or from a Jupyter notebook if that’s your jam).
If you want to skip the config file for a quick one-off extraction, you can pass the object and query directly:
sf bulkyard extract \
--sobject Opportunity \
--query "SELECT Id, Name, StageName, Amount FROM Opportunity WHERE IsClosed = FALSE" \
--database opps.db \
--target-org my-sandbox
That kind of flexibility matters during discovery work. Sometimes you want a full config-driven pipeline. Sometimes you just want to grab a table and start poking at it.
Once the data is in SQLite, you’re in familiar territory. Everything from the 2024 data work post applies here: SQL JOINs for relationship work, Pandas for anything more complex, or even just running queries in a GUI to sanity-check what you’ve got.
The key difference is that bulkyard eliminated the extraction boilerplate. By the time you’re writing your first transformation query, the data is already there.
For example, after extracting Account and Contact, you might produce a cleaned Contact table for a migration scenario:
CREATE TABLE Contact_Clean AS
SELECT
c.Id,
c.AccountId,
c.FirstName,
c.LastName,
c.Email
FROM Contact c
INNER JOIN Account a ON c.AccountId = a.Id
WHERE c.Email IS NOT NULL
AND a.Industry IS NOT NULL;
When you’re ready to push data back into Salesforce, bulkyard load handles it:
sf bulkyard load \
--sobject Contact \
--operation upsert \
--external-id-field Id \
--database opps.db \
--target-org my-production-org
The --operation flag supports insert, update, upsert, and delete, so the full CRUD surface is covered. For upsert operations, you specify the external ID field. Typically, that’s Id for updates within an org, or a custom external ID field for cross-org migrations.
For multi-object loads, this is where the config file can pay off:
database: bulkyard.db
objects:
- sobject: Account
operation: upsert
external_id_field: External_Id__c
- sobject: Contact
operation: upsert
external_id_field: External_Id__c
table: Contact_Clean
The table override on Contact is how you tell bulkyard to load from Contact_Clean instead of the raw Contact table. Run it with:
sf bulkyard load --config-file config.yaml --target-org my-production-org
And that’s your full pipeline: extract, transform locally, load back. All config-driven, all repeatable.
I want to share something that bit me during development, because it’ll probably bite you too if you reach for the same tooling.
When I started working on bulkyard, the obvious path for Bulk API 2.0 operations in a TypeScript SF CLI plugin is conn.bulk2.query() from @salesforce/core’s Connection class (which wraps jsforce under the hood). The API looks clean:
// The first approach — looks fine, isn't
const queryResult = await conn.bulk2.query(soqlQuery);
const records = await queryResult.toArray();
This works. Right up until you point it at an object with a meaningful number of records, at which point it times out. The fix was to drop the abstraction entirely and implement the job lifecycle directly against the REST endpoints: POST a query job, poll until it completes, then walk the paginated CSV results page by page until complete.
// The actual implementation
const jobInfo = await conn.requestPost<QueryJobInfo>(jobsUrl, {
operation: 'query',
query: soqlQuery,
});
await waitForJobCompletion(conn, jobInfo.id);
let locator: string | null = null;
do {
const url = locator ? `${resultsUrl}?locator=${encodeURIComponent(locator)}` : resultsUrl;
const response = await fetch(url, { headers: authHeaders });
locator = response.headers.get('sforce-locator');
// parse CSV page, batch insert...
} while (locator && locator !== 'null');
This is not unusual territory if you’ve worked with the Bulk API 2.0 REST spec directly, but it’s a meaningful gap in the jsforce abstraction that I didn’t know. If you’re building anything on top of the Bulk API 2.0 in TypeScript and care about completeness for large datasets, go to the REST endpoints.
What bulkyard adds is a tighter integration with the Salesforce CLI, which most developers already have installed, and a configuration model that makes the extract/load cycle feel effortless.
If you’re already using the SF CLI day-to-day, bulkyard should slot into your workflow without friction. And if you’ve been building Python scripts to wrangle data between Salesforce and SQLite, consider whether you want to keep maintaining that boilerplate or just offload the plumbing.
bulkyard is open-source. You can find the source on GitHub and install it from npm. Issues and PRs are welcome.
sf plugins install bulkyard
Happy data wrangling!
]]>You could click around Setup for a few hours. You could ask the client a bunch of questions (and get a bunch of “I think maybe someone set that up a few years ago?” answers). Or, you could do what I’ve been doing lately: fire up a Jupyter notebook and let the org tell you its own story.
I’ve been reaching for Jupyter notebooks more and more during client discovery, and at this point I’m not sure how I ever lived without them. The appeal boils down to a few things.
First, notebooks are iterative. You can run a query, look at the results, decide what to dig into next, and keep going without losing context. It’s the closest thing to having a conversation with a Salesforce org.
Second, they’re self-documenting. When you’re done poking around, you don’t just have a pile of SOQL queries in a text file. You have a narrative (think markdown headings, inline results, and charts) that you can hand to a colleague or even share with the client. It’s a living artifact of your discovery process.
Third, and this is the one that sold me, they’re reusable. I’ve built up a standard org analysis notebook that I bring to every new engagement. I connect it to the org, hit “Run All,” and within a few minutes I have a high-level picture of what I’m working with. It’s not a replacement for real discovery, but it’s a phenomenal starting point.
The tooling is straightforward. I use simple_salesforce as a Python client for the Salesforce REST API, pandas for wrangling data into DataFrames, and matplotlib (with seaborn for aesthetics because I have some self-respect) for visualization.
import pandas as pd
import matplotlib.pyplot as plt
import seaborn
from simple_salesforce import Salesforce
seaborn.set()
The notebook is designed to run either locally or in Google Colab, which is nice when you’re pairing with someone who doesn’t have a local Python environment set up.
This is the part where I get on a small soapbox. Never hard-code your Salesforce credentials. Not in the notebook, not in a config file you commit to Git, not anywhere that isn’t explicitly designed for secrets management.
When running in Google Colab, I pull credentials from Colab’s userdata secrets store:
from google.colab import userdata
sf = Salesforce(
username=userdata.get('SF_USERNAME'),
password=userdata.get('SF_PASSWORD'),
security_token=userdata.get('SF_SECURITY_TOKEN'),
consumer_key=userdata.get('SF_CLIENT_ID'),
consumer_secret=userdata.get('SF_CLIENT_SECRET'),
domain=userdata.get('SF_DOMAIN')
)
When running locally, I load from an environment file that is .gitignored into oblivion:
import json
filename = f'./envs/{env_name}.json'
with open(filename, 'r') as f:
creds = json.load(f)
sf = Salesforce(
username=creds['username'],
password=creds['password'],
security_token=creds['security_token'],
consumer_key=creds['client_id'],
consumer_secret=creds['client_secret'],
domain=creds['host']
)
The environment file itself is just a simple JSON blob with your connected app credentials and login info. The key point is that it stays local. It never touches version control. If you’re using dotenv or OS-level environment variables instead, even better. The principle is the same: secrets stay secret.
(Stepping off soapbox now.)
Here’s the rough structure of my standard discovery notebook. Think of it as a checklist of the things I want to know about any org before I start making promises.
The very first thing the notebook does after authenticating is check the org’s daily API request usage. This matters because the notebook itself will consume API calls, and if you’re connecting to a production org that’s already running hot, you want to know before you fire off a few hundred SOQL queries.
limits = sf.limits()
api_usage = limits['DailyApiRequests']
consumed = api_usage['Max'] - api_usage['Remaining']
pct = round(consumed / api_usage['Max'], 2) * 100
if pct > 80:
print(f'WARNING: {pct}% consumed. Halting.')
raise StopExecution
else:
print(f'{pct}% of daily API requests consumed. Proceeding.')
I built in a StopExecution exception that halts the notebook if API consumption is above 80%. It’s a small thing, but it’s saved me from an awkward conversation at least once.
Next up: getting a sense of the codebase. I query all active Apex classes (excluding managed packages and test classes) and calculate each one’s size as a percentage of Salesforce’s character limit. This gives me a quick feel for how much custom code exists and where the big files live.
# Find test classes via SOSL
test_classes = sf.search(
"FIND {@isTest} IN ALL FIELDS RETURNING ApexClass(Id, Name)"
)
# Query all active, unpackaged Apex classes
all_classes = sf.query(
"SELECT Id, Name, Body, LengthWithoutComments "
"FROM ApexClass "
"WHERE Status = 'Active' AND NamespacePrefix = null"
)
From there, it’s just a matter of calculating byte sizes and sorting. A class sitting at 3% of the org’s total Apex limit is worth investigating. A handful of large utility classes from an open-source logging library? Probably fine. A 50KB class called DataFactory with no clear naming convention? That’s where you start asking questions.
This section is the one I find most valuable in early discovery. Salesforce orgs have a tendency to accumulate automation like a junk drawer accumulates takeout menus. Process Builders, record-triggered flows, Apex triggers… they all pile up. Understanding what fires when, and on which objects, is critical.
Process Builders:
process_builders = sf.query(
"SELECT ApiName, Label, InstalledPackageName, VersionNumber "
"FROM FlowDefinitionView "
"WHERE IsActive = TRUE AND ProcessType = 'Workflow'"
)
Just seeing the count here tells you something. A handful of Process Builders on a mature org? Normal. Fourteen of them, some from managed packages, some clearly hand-rolled years ago? That’s a migration conversation waiting to happen (since Salesforce is sunsetting Process Builder in favor of Flows).
Record-Triggered Flows:
flows = sf.query(
"SELECT ApiName, TriggerType, Label, TriggerObjectOrEventLabel, "
"RecordTriggerType, TriggerOrder "
"FROM FlowDefinitionView "
"WHERE IsActive = TRUE "
"AND (TriggerType='RecordAfterSave' OR TriggerType='RecordBeforeSave')"
)
I chart these by object label to see which objects are the busiest. If an object has multiple before-save flows, after-save flows, and Apex triggers, that’s a land mine. You need to understand the order of execution and whether they’re stepping on each other.
Apex Triggers:
triggers = sf.query(
"SELECT Name, TableEnumOrId, NamespacePrefix, Status, "
"UsageAfterInsert, UsageAfterUpdate, UsageBeforeInsert, UsageBeforeUpdate "
"FROM ApexTrigger WHERE Status = 'Active'"
)
I split these into managed package triggers versus unpackaged triggers. Managed package triggers (like the ones from CPQ, which can have dozens) are largely out of your control but important to be aware of. Unpackaged triggers are the ones you’ll be maintaining, and visualizing them by object and trigger event gives you an instant sense of complexity.
I query every customizable SObject in the org and get record counts for each. This takes a minute (it’s a lot of queries), but the result is a clear picture of which objects are heavily used and which are ghost towns.
entities = sf.query_all(
"SELECT QualifiedApiName "
"FROM EntityDefinition "
"WHERE IsCustomSetting = FALSE AND IsCustomizable = TRUE"
)
# Then, for each object:
count = sf.query(f"SELECT COUNT() FROM {object_api_name}")
Objects with zero records are interesting. Did someone build this out and never use it? Objects with millions of records are interesting too. Is storage going to be a concern? I also plot record growth over time for the top objects by querying CreatedDate on recent records. A sudden spike in record creation usually has a story behind it.
Finally, I look at licensing and user activity. License utilization tells you whether the org is right-sized or if you’re paying for seats nobody’s sitting in. User activity (specifically LastLoginDate) tells you who’s engaged and who’s not.
licenses = sf.query(
"SELECT Name, TotalLicenses, UsedLicenses, Status "
"FROM UserLicense"
)
users = sf.query(
"SELECT Name, IsActive, Profile.Name, UserRole.Name, "
"LastLoginDate, Profile.UserLicense.Name "
"FROM User "
"WHERE IsActive = TRUE AND UserType != 'AutomatedProcess'"
)
A user who hasn’t logged in for 90+ days is worth flagging. A pile of unused Salesforce Platform licenses? That’s a cost optimization conversation.
None of this replaces talking to people. You still need to sit down with admins, developers, and end users to understand why things are the way they are. But running this notebook gives you a massive head start. You walk into those conversations already knowing which objects are busiest, where the automation is tangled, and which corners of the org haven’t been touched in years.
It also earns you a bit of credibility. There’s something about saying “I noticed you have fourteen active Process Builders and a before-save flow on Opportunity that appears to overlap with an Apex trigger” that signals you’ve done your homework.
If you’re a Salesforce consultant or developer who hasn’t dabbled in Jupyter notebooks yet, I’d genuinely encourage you to give it a try. The learning curve for basic pandas and simple_salesforce is small, and the payoff in terms of discovery speed is enormous. And once you have your template notebook, every new org is just a Shift+Enter away from being a lot less mysterious.
Happy exploring.
]]>SQLite? More Like SQLightning!
Working with Salesforce data be daunting. Whether you’re migrating data from one org to another, transforming data for a new integration, or simply cleaning up data, the process can be time-consuming and error-prone. In this post, we’ll explore some tools and techniques that can help streamline the process and make your life easier.
Like any other development work, a clear set of goals is essential to success. Here are a few goals to consider when working with Salesforce data:
and, most importantly;
SQLite3 is a lightweight, serverless, and self-contained SQL database engine. It’s perfect for this kind of work, as we don’t need to worry about concurrent database writes or setting up a database server. Lastly, it’s extremely portable, as it’s just a single file that can be easily shared.
A common practice is to use SQLite3 as a staging area for the data we’ll work with. We can easily extract/import data from a source system or series of files and persist the data in dedicated SQLite tables. Then, we can run our transformations and validations against this data before loading it into Salesforce.
Lastly, a table JOIN beats the pants off of a VLOOKUP operation any day of the week.
In addition to SQLite, we’ll need a few additional tools. We’ll leverage Python throughout this post, but Ruby (with the Restforce gem) or Node.js (with the jsforce library) could be used as well.
Friendly reminder: never store your Salesforce credentials in your code. Use environment variables or a secure configuration file.
To get started, you’ll need to install the required Python libraries. You can do this using pip:
pip install simple-salesforce pandas
More detail on setting up a virtual environment can be found here.
Let’s walk through a simple example to illustrate how these tools can be used together.
Create a new Connected App in Salesforce
and obtain the client ID and client secret. You’ll also need your user’s security token.
You’ll need to enable OAuth settings for the app, and the scope for the connected app should include full
and refresh_token.
Store your credentials in a secure manner. I recommend using environment variables, but we’ll use a configuration file for this example:
{
"username": "",
"password": "",
"security_token": "",
"client_id": "",
"client_secret": "",
"api_version": "61.0",
"host": "test"
}
Authenticate with Salesforce using Simple-Salesforce. Here’s a simple function to build the connection using the configuration file:
from simple_salesforce import Salesforce
import json
env_file = open("./myconfigurationfile.json", 'r')
sf_creds = json.loads(env_file.read())
un = sf_creds['username']
pw = sf_creds['password']
st = sf_creds['security_token']
ck = sf_creds['client_id']
cs = sf_creds['client_secret']
en = sf_creds['host']
sf = Salesforce(username=un, password=pw, security_token=st, consumer_key=ck, consumer_secret=cs, domain=en)
(Yes, it’s really that straightforward.)
Extract data from Salesforce. In this example, we’ll extract data from the Account
SObject and dynamically retrieve all fields:
sobject_type = "Account"
sobject = getattr(sf, sobject_type)
description = sobject.describe()
field_names = []
compound_fields = set()
for field in description["fields"]:
field_names.append(field["name"])
if field["compoundFieldName"] is not None and field["name"] != "Name":
# compound fields should not be queried directly
compound_fields.add(field["compoundFieldName"])
# remove compound fields
field_names = [field for field in field_names if field not in compound_fields]
# dynamically define bulk2 type
sobject = getattr(sf_client.bulk2, sobject_type)
query = f"SELECT {', '.join(field_names)} FROM {sobject_name}"
sobject.download(query, path='./temp_data')
Now that we have our data locally as CSV files, we can load them into a SQLite3 table:
import sqlite3
import pandas as pd
import os
from pathlib import Path
# create a new SQLite3 database
db = sqlite3.connect('salesforce_db.sqlite3')
# for each CSV file in the data directory, load it into a SQLite3 table
for csv in Path('./temp_data').glob("*.csv"):
df = pd.read_csv(csv)
df.to_sql("Account", db, if_exists='append')
# then, we can clean up the CSV files
for filename in os.listdir('./temp_data'):
file_path = os.path.join('./temp_data', filename)
try:
if os.path.isfile(file_path) or os.path.islink(file_path):
os.unlink(file_path)
elif os.path.isdir(file_path):
shutil.rmtree(file_path)
except Exception as e:
print('Failed to delete %s. Reason: %s' % (file_path, e))
Now that we have our data in SQLite3, we can run our transformations and validate the outputs. There’s a number of ways to do this, including running additional Python scripts, but we’ll leverage plain ol’ SQL for this example:
Here, we’ll create a new table Account_Clean and remove any records where the Website or Industry field is empty:
CREATE TABLE Account_Clean AS
SELECT * FROM Account
WHERE Website IS NOT NULL OR Industry IS NOT NULL;
Finally, if we had modified data, we can load the records into Salesforce via the Bulk API:
pd.read_sql_query("SELECT * FROM Account_Clean", db).to_csv("temp-data.csv", sep=",", index=False)
sobject = salesforce_client.bulk2.__getattr__("Account")
results = sobject.upsert("temp-data.csv", external_id_field="Id")
SQLite3, Python, and Simple-Salesforce are incredibly powerful tools that can make your data work more efficient, repeatable, and accurate. By leveraging these tools, you can streamline your data migration, transformation, and cleanup processes and focus on the more important (and fun) aspects of your work.
]]>Through this blog, I aim to share my knowledge and experiences, spark discussions and connect with a community of like-minded individuals. By documenting my professional journey and personal interests, I hope to refine my communication skills and gain new perspectives that will enrich both my career and personal life.
]]>