Picture a sprawling enterprise catalog system, born in an era when monolithic applications ruled. It processes thousands of product queries daily across multiple regions. It has served its purpose for years — but it’s now showing its age in ways that directly impact the business.
I worked on exactly this kind of system recently. The story is generic enough to share, and the modernization patterns apply to any legacy catalog with the same symptoms.
Watch the 30-second summary
Auto-generated summary — read the full article below for details.
The Current State
The user interface is reminiscent of Web 1.0:
- An advanced search form with a labyrinth of filtering options
- A paginated list of search results
- Detailed product pages with specifications and images
- Supplementary category navigation screens
The database handles an exponentially growing combination of search parameters — price ranges, categories, technical specifications, availability status. Response times are getting slower. Maintenance headaches are getting worse.
The real problem is the database. At 500 GB, it’s bloated with binary image data — a decision that made sense in the early 2000s but is now a significant liability. Every product image sits directly in the database. A moderately coordinated DDoS attack targeting image endpoints can bring the entire system down, as each request hammers both the application server and the database.
The deployment architecture:
- A single application server on an EC2 instance
- A separate EC2 instance hosting PostgreSQL
- No CDN, no caching layer, no separation of concerns
This worked when the catalog contained a few thousand items. Today, with hundreds of thousands of products and users expecting sub-second responses, it’s mission-critical to modernize.
The Pain Points
Performance
- 500 GB database dominated by binary image data
- Slow query performance with complex search parameters
- High memory usage on EC2 instances
- No caching — repeated identical queries hit the database every time
- DDoS vulnerability through image endpoints
Technical Debt
- Monolithic codebase making changes risky
- Tight coupling between components
- No separation between read and write operations
- Limited scalability options
- High maintenance costs
The Modernization Strategy
Three phases, each delivering value independently. No big bang.
Phase 1: Image Management Transformation
The biggest win for the least risk. Extract images from the database.
S3 Migration:
- Create S3 bucket infrastructure with appropriate security policies
- Develop migration scripts for batch processing
- Execute staged migration during off-peak hours
- Verify data integrity post-migration
Image Microservice:
- Lambda-based image processing service
- Automatic image optimization pipeline
- Format conversion for modern formats (WebP, AVIF)
- RESTful API for image operations
CloudFront Integration:
- Global CDN distribution
- Caching strategies tuned per content type
- Image compression at edge locations
- SSL/TLS everywhere
Result: Database drops from 500 GB to ~50 GB. Image requests never hit the application server. DDoS surface is eliminated.
Phase 2: Search Engine Enhancement
Replace the SQL-based search with a purpose-built engine.
OpenSearch Implementation:
- Design search indices optimized for the catalog’s access patterns
- Data synchronization from PostgreSQL to OpenSearch
- Faceted search capabilities out of the box
- Sub-second response times for complex queries
GenAI Integration:
- Natural language query processing — users describe what they want instead of filling a 15-field form
- Semantic search using embeddings
- Product recommendation engine based on search patterns
- A/B testing framework to measure relevance improvements
Phase 3: Caching and Event-Driven Architecture
Eliminate redundant work. React to changes instead of polling.
ElastiCache (Redis):
- Cache warming for popular queries
- Intelligent invalidation rules
- Dramatic reduction in database load
Event-Driven Updates:
- EventBridge integration for product changes
- Lambda functions for event processing
- Dead letter queues and retry mechanisms
- OpenSearch index stays fresh without batch reindexing
The Target Architecture
┌─────────────┐
│ CloudFront │
│ (Images) │
└──────┬──────┘
│
┌──────────┐ ┌─────────┴─────────┐ ┌──────────────┐
│ Client │───▶│ API Gateway │───▶│ OpenSearch │
└──────────┘ └─────────┬─────────┘ └──────────────┘
│
┌───────────┴───────────┐
│ │
┌────────┴───────┐ ┌─────────┴────────┐
│ ElastiCache │ │ PostgreSQL │
│ (Redis) │ │ (50 GB, no imgs)│
└────────────────┘ └─────────┬────────┘
│
┌─────────┴─────────┐
│ EventBridge │
│ (change events) │
└───────────────────┘
What Makes This Work
- Each phase delivers value independently. Phase 1 alone eliminates the DDoS risk and cuts database size by 90%.
- No big bang cutover. The legacy application keeps running. You migrate traffic gradually.
- GenAI search is additive. The traditional faceted search stays as fallback. Natural language queries layer on top.
- Event-driven architecture keeps everything consistent without the fragility of batch synchronization jobs.
Key Takeaways
- Images in a relational database is technical debt that compounds daily — extract them first
- A CDN is not optional for anything serving static assets at scale
- Replace SQL-based search with a purpose-built engine (OpenSearch, Elasticsearch) when query complexity outgrows what a relational DB can handle efficiently
- GenAI-powered natural language search doesn’t replace faceted search — it complements it
- Event-driven architectures eliminate the “eventual consistency” gap that plagues batch sync jobs
- Modernize incrementally — three phases that each stand alone beats one big-bang migration that might never ship
Any opinions in this article are my own.