Our client is a leading private research university in the U.S. It has revolutionized higher education by integrating teaching and research. The university was looking for an alternate solution for better managing their humungous data. The solution had to be scalable and cost effective.
The Astrophysics department of the university was conducting research on large datasets of particle physics data. Data was held on SQL Server and Python programs and SQL queries were used to process the data. Data was also made available to other universities and government institutions for their analysis.
Ellicium facilitated adoption of Hadoop and Spark for data storage and processing, thereby reducing data management costs by over 50%.
Challenges with previous approach
- Managing several Terabytes of data on a single SQL Server database was becoming unmanageable
- Data had to be split into snapshots but that made the data processing logic complex
- Getting a single consolidated view of the data was becoming difficult