Full-stack data processing system, from intake forms, to data-cleaning, to reporting and dashboards.
I was tasked with designing and implemented a full-scope data system for an inspection program with multiple different constituencies.
The solution had many constraints, including the requirement to work seamlessly with a loosely affiliated network of consultants and members.
The data-intake form was built on top of Excel, as it was a tool that everyone was familiar with. Extensive controls and guidelines were built into the Excel form. Then a data-ingestion pipeline was built in python to clean and review data the data and put it into an SQL database.
From there, numerous python scripts were developed to manage the intake forms, report on the data, and manage the program.
In parallel, another SQL database was developed to host an inspection management and scheduling platform, from which a front-end database portal app was built using the app-builder DronaHQ.
Requirements:
- data intake
- data quality assurance
- data warehousing
- reporting
- live data dashboards
Tools:
- Python (pandas, openpyxl, sqlalchemy)
- MySQL
- BOX.com
- Excel
- DronaHQ