2400 Meadowbrook Parkway, Duluth, GA 30096 | +1 770-493-5588 Follow Us
Select Page

Enabling complex joins between datasets and faster iteration of ML models for an AI tech company

 

Client Background

Client is an AI technology firm that helps large enterprises integrate data from a variety of sources and build their machine learning models both faster and more precise

Challenges

  • The application hosts hundreds and thousands of datasets (either free or paid) sourced from thousands of providers.
  • Enable the enterprise users, decision scientists, and data analysts to upload their organizational datasets.
  • Facilitate joins between the uploaded transactional/non-transactional datasets and the other publicly hosted datasets.
  • These joins should be executed within a few seconds for seamless user experience.

Solutions

  • Implemented responsive and modern frontend app for data scientists using Redux React.
  • Designed and implemented all middle-tier services that include APIs and data access layer on Python Django
  • Wrote ANSI SQL code generator in Python that considers all user selections, connects with the metadata system, and generates the final query that runs on Snowflake.
  • Built search and recommendation systems on Neo4j that help users find features pertinent to their own uploaded datasets.

Tools & Technologies

Numpy, Django, Redux, React, AWS, Snowflake

Key benefits

  • Snowflake allows complex joins that include running various math functions between large datasets to happen within seconds, giving an output of billions of rows
  • It auto-creates multiple clusters depending on the count of concurrent queries as the workload increases
  • Data Scientists can quickly iterate over their models and thus move towards higher accuracy levels since they now save a significant amount of time finding the most relevant features.
Key Benefits - Data Engineering for Pipeline Management

    captcha

    Services

    Digital Product Engineering

    Cloud Services

    Data & Analytics

    Intelligent Automation

    Cyber Security

    Build Operate Transfer

    Talent Solutions

    Industries

    Banking & Financial Services

    Communication, Media
    and Technology

    Energy and Utilities

    Healthcare

    Life Sciences

    Manufacturing

    Retail

    Transportation and Logistics

    Travel & Hospitality

    Innovation @ Work

    Blogs and Insights

    Research and Whitepapers

    Case Studies

    Company

    About Us

    Leadership

    Strategic Partnerships

    Office Locations

    News and Events

    The Foundation

    Careers

    Open Positions

    Life @ Innova Solutions
    Candidate Resource Library

    Let's Connect