Simon Data Integration Platform
A Boomi-based platform that centralizes Oxford's marketing data pipelines — ingesting, validating, securing, and delivering data from multiple vendors into Simon Data, replacing a patchwork of direct vendor connections into Upland.
Overview
- Project Name: Simon Data Integration Platform
- Project Type: Marketing data integration platform (Boomi)
- Source: Cinnamon Toast, Aislelabs, eGifter (Adeptmind planned)
- Target System: Simon Data (via SFTP/S3)
- Tech Stack: Boomi, Groovy, SFTP, PGP Encryption, SQL Server
- Status: Delivered
Retail Marketing decided to migrate its marketing data platform from Upland to Simon Data, and used the move as the moment to fix a bigger problem: multiple vendors had been sending data directly to Upland over a mix of APIs, CSV uploads, and manual processes, with little Oxford visibility into any of it. The target architecture centralizes that ownership inside Oxford instead — every marketing dataset is ingested, monitored, transformed, secured, and governed by Oxford's own integration platform before it's ever transmitted to Simon Data.
The platform was built around a small set of principles: centralize ownership of customer data pipelines, standardize how new sources get ingested, build reusable Boomi components instead of one-off integrations, support onboarding and offboarding vendors without rework, add monitoring, auditing, and operational controls that didn't exist before, and remove manual marketing operations wherever the data allowed it.
Architecture
Source Systems
The platform consolidates five datasets across four vendors, each landing through its own source-specific process before joining the shared pipeline:
- Cinnamon Toast — Newsletter Signups: delivered via API.
- Cinnamon Toast — User Profiles: delivered via API.
- Aislelabs — Guest Wi-Fi Users: delivered via API.
- eGifter — Gift Card Purchases: delivered via SFTP CSV.
- Adeptmind — Shopify / Product Data: the intended future API direction, not yet a live source.
One Platform, Not Three Integrations
The most important thing about this platform isn't any one source — it's that the newsletter, user-profile, and Wi-Fi integrations (ID202, ID203, and ID204) are deliberately built on the exact same framework. All three share a Boomi architecture, a Pipeline Control process, a database schema, monitoring tables, an error-handling framework, a Simon Data delivery process, and the same compression, encryption, and SFTP delivery mechanism. Onboarding a new vendor means writing one source-specific adapter, not rebuilding the platform underneath it.
Every source runs through the same Pipeline Control framework, transformation, compression, encryption, and Simon Data delivery — only the source-specific adapter changes.
Shared Platform Components
Three components in particular are what actually make this a platform rather than a pile of separate integrations.
Pipeline Control Framework: every integration writes its execution information into BoomiProcessing › Mrkt_Email_SimonData › Pipeline_Control — integration ID, batch ID, execution ID, start and end date, record counts, success/failure status, and an override flag. That's what makes incremental execution tracking, auditing, recovery, and reruns possible. Instead of hardcoding a "yesterday to today" window into each API call, the framework works out the execution window dynamically and supports replay through the override mechanism — a lightweight integration control framework underneath every source.
API Request Logging Framework: a second shared table, Common_Boomi.API_Request_Log, is used by all three integrations to record the process name, endpoint called, record count, endpoint URL, timestamp, and execution ID behind every API call — operational visibility that didn't exist before this platform.
Reusable Simon Data Delivery Process: every source eventually calls the same [Sub]_Send_Data_to_SimonData_SFTP process, which combines files, converts encoding to UTF-8, compresses to a ZIP archive, encrypts it with Simon's public key, creates a date-based directory, and uploads to Simon's SFTP/S3 landing area — likely the single most reused asset the project produced.
Process ID202 — Newsletter Signups
Newsletter signups used to go straight to Upland with no Oxford visibility into the flow. ID202 gives Oxford both: the API takes start_date and end_date parameters, so the integration processes true incremental windows instead of re-pulling everything on every run. The process calls a token endpoint for a JWT, calls the newsletter endpoint, and pages through results until every page is retrieved, logging record counts per property along the way. A custom pagination handler — [Sub]_ID202_Set_Page_for_Paginated_API — uses a Groovy script to generate each page request dynamically rather than hardcoding a page count.
Process ID203 — User Profiles
Before Simon Data, user profiles were a fully manual process — someone exported CSVs, reformatted them, and uploaded them to Upland by hand. ID203 removes that manual step entirely, but it can't do what ID202 does: the User Profile API has no Last Updated field, so incremental processing isn't possible — every run has to pull the full dataset. Boomi also receives the entire API payload as a single JSON document, so getting an accurate record count means splitting it by ID, flattening it to CSV, and counting lines, rather than reading a count off the response directly.
Process ID204 — Aislelabs Wi-Fi
This is the most technically interesting of the three. Aislelabs' API doesn't return an array of users — it returns a single object with each guest's email address used as the object's own key, so the schema shifts with every response and breaks Boomi's standard mapping outright. A Groovy transformation fixes it by turning each email-keyed object into a normal record with an id field holding the email, collected into a plain array Boomi can map dynamically no matter which emails show up:
{
"[email protected]": { "wifiSessions": 3, "lastSeen": "2026-08-01" },
"[email protected]": { "wifiSessions": 1, "lastSeen": "2026-07-28" }
}becomes:
[
{ "id": "[email protected]", "wifiSessions": 3, "lastSeen": "2026-08-01" },
{ "id": "[email protected]", "wifiSessions": 1, "lastSeen": "2026-07-28" }
]On top of that, the integration runs across 9 retail properties, each with its own org_id and swid, with record-level logging and aggregation across all of them.
Key Features
- Shared Platform, Not Separate Integrations: ID202, ID203, and ID204 share one Boomi architecture, database schema, monitoring tables, error handling, and delivery process — onboarding a new vendor means adding a source adapter, not rebuilding the pipeline.
- Pipeline Control Framework: every run logs its integration ID, batch ID, execution ID, date window, record counts, and success/failure status, with an override mechanism that supports replay instead of hardcoded date logic.
- API Request Logging: every call any of the three integrations makes is recorded — endpoint, record count, timestamp, execution ID — giving Oxford operational visibility that didn't exist under Upland.
- Reusable Simon Data Delivery: one shared process combines files, converts to UTF-8, compresses, PGP-encrypts with Simon's public key, and uploads to Simon's SFTP/S3 landing area for every source.
- Vendor API Quirks Solved in Middleware: a Groovy transformation normalizes Aislelabs' email-keyed JSON into a standard array, and a custom pagination handler drives ID202 through however many pages a given run needs.
- Security by Design: PGP encryption, SSH key-based SFTP authentication, and Oxford never holding Simon's private keys.
Technologies Used
The integration platform running every source adapter, the shared Pipeline Control framework, and the delivery process into Simon Data.
Powers the custom transformations Boomi's standard mapping can't handle — normalizing Aislelabs' email-keyed JSON and driving ID202's dynamic pagination.
The delivery channel into Simon Data's landing area, and the pickup channel for eGifter's CSV exports.
Hosts the Pipeline Control and API Request Log tables every integration writes its execution history to.
The marketing platform this entire pipeline exists to feed, replacing Upland as Retail Marketing's data destination.
Key Accomplishments
Platform Engineering
- Designed a reusable Simon Data ingestion framework instead of one-off vendor integrations.
- Built the shared Pipeline Control architecture every source runs through.
- Established a shared monitoring and audit pattern across all three integrations.
- Standardized the security and delivery model every source uses to reach Simon Data.
Operational Excellence
- Replay capability through override processing, instead of a failed run needing a manual fix.
- API request logging that gives visibility into every call any integration makes.
- Record count validation on every run.
- Site-level enable/disable controls.
Security
- PGP encryption on every file delivered to Simon Data.
- SSH key-based authentication on the SFTP delivery channel.
- Secure SFTP delivery throughout.
- Oxford never holds Simon Data's private keys.
Business Outcomes
- Eliminated manual CSV preparation for user profiles.
- Reduced operational risk across the marketing data pipeline.
- Improved visibility into customer marketing data that didn't exist under Upland.
- Created a repeatable onboarding pattern for future marketing data sources.
- Established Oxford ownership of its own data delivery pipelines.
Outcome
Retail Marketing's data now flows through one Oxford-owned platform instead of scattering across direct vendor connections and manual CSV handling. Every run is logged, every failure is recoverable through the override mechanism, and onboarding the next vendor means writing one adapter against a framework that already exists, not building a new integration from nothing. Manual CSV preparation for user profiles is gone entirely, and Oxford now has visibility into its marketing data pipelines that the Upland-era vendor-direct model never provided.