Turnkey Integrations
As I reflect on my career I see themes across projects I have worked on. Today I want to focus on the Data Connector Project.
Before my memory fades, I want to revisit a project I worked on years ago: a self-service integration tool designed to sync data between two enterprise SaaS platforms. Looking back, I’ll ask:
Were the decisions we made at the time justified?
What would I do differently today?
The product had two major components:
The Data Connector/ETL: Synced data between SaaS 1 (our product) and SaaS 2 (the external system).
The Self-Service Turn-Key Setup: A feature enabling customers to activate the integration with minimal clicks.
Let’s break these down one by one.
1. The Data Connector/ETL
The connector allowed customers to push data from SaaS 1 to SaaS 2 and sync changes bidirectionally. If an employee updated a field in one system, the change propagated to the other, eliminating manual reconciliation.
Diagram Reference:
Key Questions in Retrospect
Was a new connector necessary?
Likely not. The monolith’s complexity had bred an “allergy” to adding features directly to it, prompting decomposition efforts. But with hindsight, extending the monolith could have simplified parts of this project—if not for organizational resistance.The Node.js vs. Spring Boot Debate
The company was primarily a Java shop, with newer BFF services using Node.js. Early struggles arose because SaaS 2’s API lacked a robust Java SDK. While a JavaScript SDK worked seamlessly for prototyping, the Java alternative was poorly documented and required wrestling with non-standard auth (neither BASIC nor API key-driven). Worse, the API itself was slow and unreliable—a case of “lipstick on a pig.”Ultimately, we built the integration in Node.js. While functional, I question this choice today. The JVM’s slow startup times (mitigated by keeping instances “warm”) and Spring Boot’s boilerplate were tradeoffs, but Java’s dominance in our ecosystem might have justified the grind.
Syncing Logic
The first version required manual user clicks to trigger syncs. Ideally, a background “heartbeat” process should have handled this automatically. I can’t recall if this was ever implemented, but it highlights the importance of designing systems to minimize user friction.
2. The Self-Service Turn-Key Setup
The “holy grail” of this project was enabling customers to activate the integration with minimal effort. The goal was a seamless, self-service experience.
Diagram Reference:
Key Challenges
Activation Complexity: Despite the goal, enabling the integration still required 7–8 clicks. A major hurdle was the lack of a centralized identity provider (IDP) outside the monolith, forcing convoluted session management and orchestration.
Orchestrator Issues: The orchestrator wasn’t well-designed for self-healing. Debugging failures was difficult, especially since the database schema was generated directly from Java code. A cleaner, standalone service would have been preferable.
Overall Regrets and Lessons
ORM Overkill
We used an ORM to abstract database interactions. While ORMs save time upfront, they risk insulating developers from understanding SQL and database fundamentals—a disservice in the long run. Our queries were simple; raw SQL or a lightweight query builder would have sufficed.Agile vs. Kanban
We followed Scrum, but Kanban would have better suited the exploratory nature of the project. Scrum works for stable products balancing tech debt and features, but early-stage R&D demands flexibility.
The Bigger Picture
The integration “worked,” but only because enterprise SaaS often prioritizes “good enough” over elegance. Self-service tools demo well, but if your customers expect hand-holding, they’ll want to customize it. The next logical step—allowing users to define custom field mappings—was never pursued due to messy data models and lack of ambition.
Ironically, the company could have pushed for industry-wide data standardization. Instead, the founder cashed out via private equity, ending the era of product-led growth.
Final Thoughts
Would I rebuild this today? Absolutely—but with raw SQL, and a focus on true one-click activation. More importantly, I’d advocate for solving root problems (e.g., data model standardization) rather than papering over them.
In enterprise tech, “success” often means shipping the minimum viable solution under constraints. But as engineers, our duty is to balance pragmatism with the humility to learn from past shortcuts.