Download Case Study

EDI ETL – Serverless Cloud Application

serverless-cloud-application

Introduction

This module is implemented for Us client, for consolidating the electrical data from various electricity provider. Data from the various provider is supplied in EDI 867 format.

This application is responsible for performing ETL operation on EDI data and store in centralized data. After storing data in centralized database, a job is responsible for sending the user data to Salesforce using SF API.

Application has 3 layers for extracting HU data from various provider – following are the details:

Layer 1

  • In this layer we have data request and Raw data received from various sources.
  • Data is requested by generating a CSV for EDI file or a Web scrapper.
  • All the data request will be stored in HU request queue in RDS.
  • A web interface is provided for requesting HU data.

Layer 2

  • In this layer data will be extracted from Raw data received i.e. HU Data file 867, Acknowledgement 814 and Web scrapper.
  • All the extracted data will be stored in RDS database.

Layer 3

  • In this layer data captured from 2nd Layer RDS DB will be fetched and stored in structured format in another DB in RDS using ETL jobs.
  • A web interface is provided for fetching this data.

Electronic HU Data - Architecture

edi-architecture

Features

Dashboard

This is the dashboard of application, which shows progress of EDI/Web scraper Request. User can see list of status of jobs running.

dashboard

HU Request

Using this feature user can view list of jobs that has been queued by user. For requesting jobs, user can click on Add Request or alternatively an API is available using which user can submit the batch request -

hu-request

User can also view completed HU request, by switching to View Completed HU Request

Inbound EDI

Using this feature user can view list of EDI files which has been received from electricity provider for further processing –

inbound-edi

User can also view completed EDI request, by switching to View Completed EDI Request

HU/IU Data

Historical/Interval Usage data – using this feature user can search for data captured using ETL process. User can search by single/multiple account number. Alternatively, user can search using consumer service provider –

hu-data

Service Provider

Using this feature user can manage configuration related to service provider –

service-provider

Job Schedular

These are the background service, responsible for processing the requested jobs. There are separate jobs for processing webscrapper and EDI processing.

EDI Jobs – EDI files are uploaded by 3rd party service provider on box.com ftp location. Box.com triggers EDI Jobs schedular which is responsible for processing EDI ETL request.

Webscrapper Jobs – This job is responsible for fetching electricity data from website. There are multiple website from which we fetch the electricity data.

Contact Us

info@aress.com