Clinical document digitisation
Ten million clinical documents, searchable in under 100 ms
- Year
- 2024–26
- Client
- IQVIA
- Domain
- Clinical trials · HIPAA
- My role
- Lead AI/ML Engineer · team of 4
- Stack
10M+documents processed
0.94F1 on medical entities
10×pipeline throughput
The problem
Brochures, study reports and safety data were being processed by hand — around 20 hours per batch — slowing down regulatory submissions, with patient data that had to stay HIPAA-compliant.
What I built
- A BERT-based extraction pipeline for adverse events, lab values and demographics, with PHI de-identification and audit logging.
- A distributed Elasticsearch cluster serving 50,000 queries a day at under 100 ms p99.
- An asynchronous RabbitMQ pipeline that took throughput from 500 to 5,000 documents an hour.
- MLOps practices for a team of four that halved deployment time, with 95% test coverage on the ML pipelines.
The hard part
Accuracy and privacy pull in opposite directions on clinical text. The pipeline de-identifies first, extracts second, and keeps an audit trail of both — so speed never comes at the cost of compliance.
Outcome
Batch processing fell from 20 hours to 3, submission cycles ran 75% faster, and approval bottlenecks shrank from 3 days to 4 hours.
Client data and code are confidential; figures are from the delivered system.
Next caseLegal contract intelligence →