Solution options
Thinking through possible solutions. Each one is evaluated against the same checklist.
What the customer actually needs
From the brief: "They want raw or lightly modelled data, on a dependable schedule, with a schema they can build pipelines against and trust not to break."
- Raw or lightly modelled — not heavily transformed, close to source. Row-level records, not aggregated summaries
- Dependable schedule — arrives when promised, every time. SLA tracked and alerted
- Stable schema — columns don't disappear or rename without warning. Versioning absorbs changes
- Build pipelines against — can automate, not manual downloads. Machine-readable formats (Parquet, not PDF)
Note: The delivery mechanism options below (file drop, data share, API, warehouse access) are different ways to deliver this. They are Phase 3 decisions. The hard work is in Phases 0-2.
The real problems to solve
These are from the brief. They are blockers before any delivery mechanism matters.
| What the brief says | Why it's hard | What has to happen first |
|---|---|---|
| "The warehouse is not modelled for outsiders, not governed to the standard needed" | Before you can export anything, you need to fix the warehouse. That's a big internal project with no visible client value | Internal platform work. Hard to fund because clients can't see it |
| "Not properly separated client-by-client" | Tenant isolation doesn't exist. You can't export client A's data if you might accidentally include client B's | Security-critical fix. Blocker for everything else |
| "Has never had to withstand a client's scrutiny" | First security review will ask: "prove you can't leak data." They can't prove it today | Needs to pass penetration testing and audit before any client goes live |
| "They ship continuously and will not slow down" | The export mechanism is easy. Keeping the schema stable while shipping fast is the organisational challenge | Process and discipline, not just code. Who approves schema changes? |
| "Good margin at 10 clients can be no margin at a hundred" | Pricing model is unsolved. Costs scale with data volume. Revenue needs to scale too | Unit economics must be modelled before committing to pricing |
| "Data is partly monetised" | Some clients think they already paid for this. Commercial conversation is hard | Need clear positioning: what's included vs what's an add-on |
Phased approach
| Phase | What | Why it's hard | Effort |
|---|---|---|---|
| Phase 0 | Fix tenant isolation in warehouse | No client value. Hard to fund. But blocker for everything. Must pass security review | Depends on current state. Could be weeks or months |
| Phase 1 | Define scope — which tables, which columns | Product decision. What to include, what to exclude. Saying no is hard. Every field you expose is a field you must maintain forever | 2-4 weeks of product work, stakeholder alignment |
| Phase 2 | Build versioning layer (views on top of tables) | Technical work, but also needs process: who approves changes? What's the deprecation policy? | 4-6 weeks engineering |
| Phase 3 | Build delivery mechanism — pick one: file drop, data share, API, or warehouse access | Actually the easy part — well-understood patterns | 6-8 weeks engineering |
| Phase 4 | Onboard first client (pilot) | Proves it works. Finds gaps. Probably breaks things. Need a client willing to be a guinea pig | 4-8 weeks including fixes |
| Phase 5 | Productise — self-service config, pricing, documentation, support | Makes it scalable beyond "we configure each one manually" | Ongoing |
Why Phase 0 (multitenancy) is a must to sell a product
We want to sell a data product. But we can't sell it until clients' security teams approve it. Their security teams will ask how we isolate data. We need an answer that passes their audit. Product asks Engineering to build whatever is needed so we can say YES with evidence.
| Product asks | Engineering delivers |
|---|---|
| "Prove tenant isolation" | RLS policies on all tables |
| "Show it can't be bypassed" | Automated tests that try to access other tenants' data and fail |
| "Document it for auditors" | Security documentation explaining the controls |
| "Pass penetration testing" | Fix any gaps found in the pen test |
What happens if we get this wrong:
- Data breach — Client A sees Client B's repairs, residents, photos
- Regulatory disaster — GDPR breach, ICO investigation, fines
- Trust destroyed — one leak and every client loses confidence
- Contract termination — most contracts have clauses for data breaches
Regulatory and legal exposure:
| Risk | What it means |
|---|---|
| GDPR breach | Personal data exposed to wrong party. Must report to ICO within 72 hours |
| ICO fines | Up to £17.5 million or 4% of global annual turnover, whichever is higher |
| Client lawsuit | Breach of contract, negligence claims, damages |
| Reputational damage | Press coverage, other clients lose confidence, deals blocked |
| Retrofitting cost | Fixing it after a breach costs 10x more than doing it now — plus legal fees, plus fines |
Delivery mechanism 1: Nightly file drop
What it is (simple terms)
Every night, Plentific generates files (like spreadsheets but for machines — Parquet or CSV) and drops them into the client's cloud storage bucket (S3, Azure Blob, Google Cloud Storage). The client's data pipeline picks them up automatically.
How it works: one product, many destinations
Each client gets their own destination configured — but the same export code runs for everyone.
| Client | Destination | Config stored |
|---|---|---|
| Notting Hill Genesis | s3://nhg-data-bucket/plentific/ | Row in config table |
| Peabody | s3://peabody-warehouse/incoming/ | Row in config table |
| Southern Housing | azure://southern.blob.core.windows.net/plentific/ | Row in config table |
Not bespoke per client. Same code, different config. Adding a new client means adding a row to a database, not writing new code.
What happens when the schema changes?
This is the hard part. Two types of changes:
| Change type | Example | Client notice | Action required by client |
|---|---|---|---|
| Additive (non-breaking) | New column uprn added to locations |
Changelog only, no advance notice needed | None — their pipelines ignore new columns |
| Deprecation | Column legacy_category_code will be removed in 90 days |
90 days advance notice, changelog, direct email | Update their pipelines before deadline |
| Breaking | Column tenant_name renamed to resident_name, or type changed |
90+ days notice, changelog, direct email, migration support | Must update pipelines or break |
| New version | v2 schema released with restructured tables | Changelog, migration guide, support both versions | Migrate when ready, within support window |
We do not modify per client. We maintain one schema per version. All clients on
v2023-02-01 get identical files. Versioning absorbs the change, not manual work.
Tech and effort to build this
Today: someone gets a request, writes SQL, exports CSV, emails it. No tracking, no SLA, no consistency.
Automated version:
| Component | Tech options | Effort |
|---|---|---|
| Export job | Python script, Airflow DAG, or dbt job. Runs SQL, writes Parquet | 2-3 weeks |
| Scheduler | Airflow, Dagster, AWS Step Functions, or cron | 1 week |
| Destination connectors | AWS SDK (S3), Azure SDK (Blob), GCS SDK. Libraries exist | 1-2 weeks |
| Config management | Database table: client → destination → credentials → schedule | 1 week |
| Monitoring and alerts | Alert if job fails. Logs of what was sent, when, file sizes | 1 week |
| Client UI (optional) | Self-service: configure bucket, see history, trigger re-export | 2-4 weeks |
Total: 8-12 weeks for a solid v1 with one engineer.
Impact across product, commercial, engineering, governance
- Product: New add-on to sell. Need UI to configure exports. Need to decide: self-service or ops-managed?
- Product: Schema versioning becomes a product discipline, not just an API thing
- Commercial: New revenue line. Pricing: flat per client? Per GB? Per property count?
- Commercial: Upsell path: basic = nightly, premium = hourly or real-time
- Engineering: Build export jobs, scheduling, retry logic, monitoring. 8-12 weeks
- Engineering: Tenant isolation in queries is critical — one bad WHERE clause = data breach
- Governance: Clear handoff point. Once file lands, it's the client's copy, their responsibility
- Governance: Audit trail: what we sent, when, to whom. Required for disputes
- Compliance (GDPR): Personal data leaving Plentific's control. Need DPA per client
- Compliance (GDPR): Data minimisation: are we sending more than they need?
- Compliance (GDPR): Right to erasure: if resident requests deletion, client's copy is not updated automatically
- Compliance (GDPR): Cross-border: if client's bucket is in US, that's a transfer issue
- Compliance: Encryption in transit (HTTPS/TLS) and at rest (SSE-S3, Azure encryption)
Scenario: large housing association wants raw data
Example: Notting Hill Genesis (66,000 homes)
Their ask: "We have a data team. We want raw repairs data in our Snowflake warehouse every night so we can build our own dashboards and join it to our tenancy data."
What we give them:
- Parquet files dropped to their S3 bucket at 2am UTC
- Tables:
repairs,bookings,inspections,locations,reference_data - Schema version
v2023-02-01— same as the API version - Changelog notifications to
data-team@nhg.org
What they do with it:
- Their Airflow pipeline picks up files, loads into Snowflake
- They join to their tenancy data using UPRN
- They build dashboards showing repair times by estate, by vulnerability status
- They generate their own regulator returns using their definitions, not ours
Is this bespoke? No. The only thing specific to NHG is one row in the config table: their bucket, their credentials, their schedule. The export code is identical for every client.
Is it scalable? Yes. Client 2 is the same process: add config row, test connection, go live. No custom code. Onboarding takes days, not weeks.
Customer problem it solves
| Need (from the brief) | How this solves it |
|---|---|
| Raw or lightly modelled | Files are direct table exports. Minimal transformation. Client does their own modelling |
| Dependable schedule | Runs every night at configured time. Monitoring alerts on failure. SLA tracked |
| Stable schema | Versioned schema. Columns only change with notice. Breaking changes = new version |
| Build pipelines against | Files land in bucket automatically. Client's Airflow/dbt picks them up. No manual clicks |
Challenges and pushback
- Freshness — nightly means up to 24 hours stale. Some clients may want real-time. Answer: that's a premium tier
- Full exports are expensive — if we export everything every night, egress costs grow. Answer: incremental exports (only changed records)
- "We already get CSV" — need to show this is automated, reliable, versioned, and machine-readable. Not the same thing
- Schema changes break pipelines — even additive changes can break strict pipelines. Answer: clear notification process, 90-day deprecation windows
- "We already pay for our data" — this is the hard one. Answer: you pay for the platform, this is a new product with different costs
Scalability and pricing (10 to 100 clients)
| Clients | Cost drivers | Pricing model |
|---|---|---|
| 10 clients | Compute for export jobs (minimal). Egress to their buckets (main cost). Support for setup | Flat fee per client, included in premium platform tier. Or: per-property pricing |
| 100 clients | 100x the egress. Some clients have 10x more data than others. Photos are the killer (large files) | Usage-based: per GB exported monthly. Or: tier by property count. Photos excluded from base tier — separate add-on |
Key insight: The export code scales fine (same job, 100 configs). The cost that scales is egress — moving data out of our cloud. So pricing must track data volume, not client count.
Value, effort, risk
| Dimension | Rating | Why |
|---|---|---|
| Volume potential | High | Every client can use it. No warehouse compatibility required |
| Build effort | Medium (8-12 weeks) | Export jobs, scheduling, monitoring, per-client config. Not trivial, but well-understood |
| Ongoing effort | Low | Automated. Support tickets for setup issues, occasional failures |
| Risk | Low | Well-understood pattern. Many B2B SaaS companies do exactly this |
| Bespoke | Very low | Same export for everyone. Only config differs. No custom code per client |
Metrics to measure success
- Delivery SLA — % of exports that land on time (target: 99.5%)
- Pipeline failures reported — how often clients report breakage after an export
- Time to onboard — days from contract to first successful export (target: <5 days)
- Support tickets — data export tickets per client per month (target: <1)
- Adoption — % of eligible clients using the feature
- Revenue — revenue from data export add-on vs cost to serve
When to revert or try something else
- More than 50% of clients need fresher data (hourly or real-time) — nightly is not enough
- Egress costs exceed revenue from the feature — pricing model is wrong
- Too many schema-change breakages despite versioning — need tighter contract
- Clients want to query, not receive files — data sharing (Option 2) may be better fit
Technical: multitenancy and APIs
- Multitenancy: Each export job has
WHERE client_id = 'nhg'baked in. One query per client. Never a shared query that filters later — that's how breaches happen - Tenant isolation: Must pass security review. Unit tests that prove cross-client queries return nothing
- APIs for clients: Check export status, view history (what files, when, how big), trigger manual re-export, update destination config
- APIs for ops: Pause a client's export, view all failures, retry failed jobs
User journey example
- Client signs contract including data export add-on
- Client's IT provides an S3 bucket ARN with write permissions for Plentific's IAM role
- Plentific ops adds config: tables = repairs, bookings, inspections. Schedule = 2am UTC. Destination = client's bucket
- Test export runs. Client's data engineer confirms files appeared
- Production export starts running nightly
- Client builds pipeline in dbt/Airflow to load files into Snowflake
- Every morning, fresh data is in their warehouse, ready for dashboards
- When schema changes: client gets email, updates their pipeline if needed, no disruption
Delivery mechanism 2: Data sharing (zero-copy)
What it is (simple terms)
Instead of copying files, the client sees a live view of their data that lives in Plentific's warehouse. Snowflake calls this "Data Sharing", BigQuery calls it "Analytics Hub". No files move. The client queries the data directly, but can only see their own records.
Why it could work
- Real-time (or near real-time) — no waiting for nightly export
- No egress costs — data doesn't leave the warehouse
- Client doesn't need to build ingestion pipelines
- Schema changes are instant — but also instantly visible
Impact by area
| Area | Impact |
|---|---|
| Product | Premium tier feature. Need UI to manage shares, see usage |
| Engineering | Build secure views per client. Ensure row-level security is bulletproof |
| Governance | Data stays in Plentific's warehouse — more control, but also more liability |
| Compliance | Need to verify client can only see their data. Audit logging essential |
Customer problem it solves
| Need | How this solves it |
|---|---|
| Raw or lightly modelled | Direct access to tables (via views), minimal transformation |
| Dependable schedule | No schedule needed — always live |
| Stable schema | Versioned views. Old views stay available during transition |
| Build pipelines | Client queries the share from their own warehouse. Native integration |
Challenges and pushback
- Lock-in — only works if client uses same warehouse (Snowflake-to-Snowflake, BigQuery-to-BigQuery)
- Query costs — who pays when client runs expensive queries? Their compute, but our data
- Security scrutiny — "you're sharing our live database?" Trust barrier is higher
- Schema changes hurt instantly — no buffer, client sees changes immediately
Scalability and pricing (10 to 100 clients)
| Clients | Cost drivers | Pricing model |
|---|---|---|
| 10 clients | Storage (shared), compute for views. Low marginal cost per client | Premium add-on fee. Flat per client or per-property |
| 100 clients | Managing 100 secure shares. Row-level security complexity. Support load increases | Tiered by usage or property count. Consider query volume caps |
Value, effort, risk
| Dimension | Rating | Why |
|---|---|---|
| Volume | Medium | Only works for clients on compatible warehouse |
| Effort | Medium-High | Row-level security, view management, cross-account sharing |
| Risk | Medium | Security failure = data breach. Must be bulletproof |
| Bespoke | Low | Same views for everyone, filtered by tenant |
Metrics to measure success
- Query activity — are clients actually using the share?
- Time to first query — how fast from setup to client running queries
- Security incidents — any cross-tenant data exposure (target: zero)
- Support tickets — questions about access, permissions, schema
When to revert or try something else
- Most clients use different warehouse than Plentific (incompatible)
- Security reviews consistently fail or take too long
- Clients prefer to own the data, not access a view
Technical: multitenancy and APIs
- Multitenancy — row-level security (RLS) on every table. Client ID filter on every query
- APIs — API to check share status, list available tables, see schema versions
User journey example
- Client (on Snowflake) asks for real-time access
- Plentific creates a Snowflake share scoped to that client's data
- Client's admin accepts the share in their Snowflake account
- Client sees tables appear as a new database in their Snowflake
- Client queries directly:
SELECT * FROM plentific_share.repairs WHERE ... - Data is always current — no waiting for exports
Delivery mechanism 3: Bulk API endpoints
What it is (simple terms)
Add new API endpoints designed for pulling large amounts of data at once. Instead of calling the API 10,000 times for 10,000 jobs, the client calls one endpoint and gets all jobs in one response (paginated). They pull when they want, as often as they want.
Why it could work
- Extends existing API — clients already know how to use it
- Pull-based — client controls timing, no push infrastructure needed
- Works for any client, regardless of their warehouse
- Change capture possible — "give me everything changed since X"
Impact by area
| Area | Impact |
|---|---|
| Product | New API tier or add-on. Rate limits, authentication, documentation |
| Engineering | Build bulk endpoints, pagination, change tracking (watermarks). Medium effort |
| Governance | Data leaves via API. Same as current API, but higher volume |
| Compliance | Existing API security applies. Need rate limiting to prevent abuse |
Customer problem it solves
| Need | How this solves it |
|---|---|
| Raw or lightly modelled | API returns records as-is, JSON format |
| Dependable schedule | Client pulls on their schedule. Availability SLA on the API |
| Stable schema | API versioning (already exists: 2023-02-01). Additive-only within version |
| Build pipelines | Client calls API from Airflow/dbt, loads into warehouse |
Challenges and pushback
- Client has to build the pipeline — more work on their side
- Compute cost on Plentific — every API call hits our servers
- Rate limits — if too restrictive, clients can't get data fast enough
- Not truly "bulk" — still JSON over HTTP, not as efficient as Parquet files
Scalability and pricing (10 to 100 clients)
| Clients | Cost drivers | Pricing model |
|---|---|---|
| 10 clients | API compute, bandwidth. Depends on call frequency | Include in premium API tier. Rate limit per client |
| 100 clients | 100 clients all pulling at midnight = spike. Need infrastructure to handle | Per-call or per-record pricing. Or monthly quota |
Value, effort, risk
| Dimension | Rating | Why |
|---|---|---|
| Volume | High | Any client can use it. No warehouse compatibility needed |
| Effort | Medium | New endpoints, but builds on existing API infrastructure |
| Risk | Low-Medium | API patterns well understood. Risk is in scaling |
| Bespoke | Low | Same endpoints for everyone |
Metrics to measure success
- API call volume — how much are bulk endpoints being used?
- Latency — p95 response time for bulk endpoints
- Error rate — % of calls that fail
- Data freshness achieved — how current is the data clients end up with?
When to revert or try something else
- Clients don't want to build their own pipelines
- API infrastructure can't handle the load cost-effectively
- Clients need data faster than API can serve it
Technical: multitenancy and APIs
- Multitenancy — API already tenant-scoped via authentication. Each client only sees their data
- APIs — this IS the API. New endpoints:
/v1/bulk/jobs,/v1/bulk/bookings, etc.
User journey example
- Client's data engineer gets API credentials with bulk access enabled
- They write a script: call
GET /v1/bulk/jobs?since=2026-09-10 - API returns paginated JSON with all jobs changed since that date
- Script loads JSON into their warehouse (Snowflake, BigQuery, whatever)
- They schedule this to run every hour via Airflow
- Their dashboards stay fresh within ~1 hour
Delivery mechanism 4: Read-only warehouse access
What it is (simple terms)
Give clients a username and password to connect directly to a read-only copy of their data in a database. They can run SQL queries, connect BI tools (Tableau, Power BI), or extract data however they want.
Why it could work
- Maximum flexibility — client can do anything SQL allows
- Familiar to data teams — just a database connection
- Real-time (or near real-time) access
- Client doesn't need to build pipelines — they query directly
Impact by area
| Area | Impact |
|---|---|
| Product | Premium feature. Credential management, usage monitoring, query limits |
| Engineering | Provision isolated schemas per client. Read replicas. Query governors. High effort |
| Governance | Data stays in Plentific's infrastructure. Full audit trail possible |
| Compliance | Highest scrutiny. Client running arbitrary SQL against live data. Security critical |
Customer problem it solves
| Need | How this solves it |
|---|---|
| Raw or lightly modelled | Direct table access — as raw as it gets |
| Dependable schedule | Always available. 99.9% uptime SLA |
| Stable schema | Versioned views layer on top of tables. Old versions maintained |
| Build pipelines | Connect any tool that speaks SQL/JDBC/ODBC |
Challenges and pushback
- Security risk — direct database access is scary for security teams
- Runaway queries — client writes bad SQL, impacts performance for everyone
- Support burden — "why is my query slow?" becomes Plentific's problem
- Isolation complexity — ensuring one client can never see another's data is hard
Scalability and pricing (10 to 100 clients)
| Clients | Cost drivers | Pricing model |
|---|---|---|
| 10 clients | Read replica capacity, query compute, storage for per-client schemas | Premium tier, flat fee. Query time limits included |
| 100 clients | Infrastructure scales significantly. Query isolation becomes critical. Support scales too | Per-seat or per-query pricing. Compute quotas. Overage charges |
Value, effort, risk
| Dimension | Rating | Why |
|---|---|---|
| Volume | Medium | Only for sophisticated clients with data teams |
| Effort | High | Per-client isolation, query governors, credential management |
| Risk | High | Security breach, performance issues, support load |
| Bespoke | Medium | Each client gets their own schema, some config |
Metrics to measure success
- Query volume — are clients actually using it?
- Query performance — p95 query time, slow query rate
- Security incidents — any unauthorized access (target: zero)
- Support tickets — access issues, slow queries, "how do I..." questions
When to revert or try something else
- Security reviews consistently block clients from using it
- Support cost exceeds revenue from the feature
- Performance issues affect other clients or the main platform
Technical: multitenancy and APIs
- Multitenancy — per-client schemas or row-level security. Must pass penetration testing
- APIs — API to manage credentials, view query history, check quotas
User journey example
- Client signs up for warehouse access tier
- Plentific provisions a read-only user with access to views filtered to their data
- Client receives connection string:
host=data.plentific.com, user=nhg_readonly, ... - Client's BI analyst connects Tableau directly to the database
- They build dashboards that refresh in real-time from live data
- No pipelines, no exports, just SQL
Quick comparison
| Dimension | Nightly file drop | Data sharing | Bulk API | Warehouse access |
|---|---|---|---|---|
| Freshness | Daily | Real-time | Hourly possible | Real-time |
| Client effort | Low — just receive files | Low — just query | Medium — build pipeline | Low — just connect |
| Plentific effort | Medium | Medium-High | Medium | High |
| Works for any client | Yes | No — same warehouse only | Yes | Yes |
| Security risk | Low | Medium | Low | High |
| Cost at scale | Egress grows | Low marginal | Compute grows | Infrastructure grows |
| My recommendation | Start here | Phase 2 for Snowflake clients | Already have API — extend it | Only if clients demand it |
Forward-looking: AI and automation
Once the basic data product exists, there are opportunities to add value with AI and automation.
AI-powered features
- Natural language queries — client asks "how many emergency repairs last month?" and gets an answer without writing SQL. Uses the data product as the source.
- Anomaly detection — "your completion rate dropped 15% this week" alerts, automatically surfaced from the data.
- Report generation — AI drafts the regulator return based on the data, human reviews and submits.
- Schema change assistant — when Plentific changes the schema, AI generates migration scripts for the client's pipeline.
MCP (Model Context Protocol) and agent integration
- MCP server for Plentific data — expose the data product as an MCP server. AI agents can query repairs, bookings, compliance data as context for their tasks.
- Agent handover — when a human can't answer a question, hand off to an AI agent that has access to the full data history. "Why was this repair delayed?" — agent investigates.
- Automated compliance checks — agent monitors data for Awaab's Law deadlines, flags at-risk cases before they become breaches.
SaaS-style features on top of the data product
- Embedded dashboards — Plentific hosts dashboards the client can embed in their own portals. Data product powers them, client doesn't need their own BI tool.
- Scheduled reports — client configures "send me a PDF summary every Monday". Data product generates it automatically.
- Alerts and notifications — "tell me when any repair is overdue by more than 7 days". Runs against the data product, sends email/Slack/webhook.
- Self-service data exploration — a simple UI where non-technical users can filter, group, and export data without SQL.
Why this matters for the case study
The data product is not the end state — it's the foundation. Once data flows reliably to clients, you can build services on top. The question is: what do you build yourself vs. what do you let clients build in their own warehouse?
- Build yourself — things that need Plentific's domain knowledge (compliance logic, repair categorisation)
- Let clients build — things specific to their organisation (joining to their tenancy data, their own KPIs)