Skip to Content
Skip to Content

improved

forecast 

ACCURACY

Using big data to drive smarter ways to save energy consumption and improve forecasting accuracy. Future proofing the network and delivering high service levels.


Project Snapshot

AEMO

Location:
Melbourne, Australia

Industry:
Utilities

Project Challenges

Data used in isolation across many systems. Business units working in silos with no consolidation or single source of the truth. Enquiries taking days or weeks to answer.

The Solution

Microsoft Azure’s data platform with rich data components catered for evolving needs. Big data warehouses were designed and built to centralise data for quick access.

Results

Capacity needed to analyse and report from centralised data that forms a consolidated single source of truth quickly. Better decision making and forecasting and reduced costs.

Download case study

The situation

The Australian Energy Market Operator’s (AEMO) vision is to deliver energy security for all Australians. It administers and operates the country’s wholesale national energy markets, with around 18 terabytes of data stored across a myriad of different on-premises systems, including some high-volume transactional systems. 

With energy consumption on the rise around the world, recent heatwaves saw Australians use more electricity than ever before.  This anomaly highlighted how difficult it is to gain insights into how electricity grids cope with the ebb and flow of energy consumption across all consumer and commercial meters. 

The opportunity

The opportunity

Historically, AEMO’s data was used in isolation across its many systems. Different business units acted in silos, generating their own reports without a consolidated, single source of truth, meaning energy grids couldn’t be compared and increases or declines were not noticed. An internal initiative to aggregate data in a pseudo data warehouse was carried out, however the volume of data proved too much as certain queries were taking  

days to execute and ministerials were taking weeks or even months to answer. 

Empired helped AEMO use big data to drive smarter ways to save energy consumption and improve forecasting accuracy across Australia. This solution is key to planning how the network will look in the future and the infrastructure required to ensure continued high levels of service to energy consumers. 

The technology

Using Microsoft Azure’s best-of-breed data platform with rich ecosystems of data components, catering for AEMO’s evolving needs, Empired assisted AEMO in tapping into their energy consumption data. This data intelligence solution allows data to be analysed right across the country and across grids every two minutes in a flexible way. 

Energy data, population data and weather data can all be factored in and assessed to forecast a future approximate for the next 20 years in 30 minute intervals. Furthermore, Azure provides the elasticity to be scalable depending on AEMO’s needs, achieving cost savings by scaling up only when required, then pausing or powering down when not in use, such as weekends. This is one of the most complicated and sophisticated uses of data intelligence in the world. 

Empired designed and built big data warehouses using Azure SQL Data Warehouse, Azure Data Lake, Power BI, machine learning, Python, Spark, R and Spark-R, Data Factory and HDInsight. The data flows were designed to load data from AEMO’s on-premises systems to the Azure-based data warehouses, automating the build of the data flows from more than 700 tables (some larger than one terabyte) from on-premises systems in Azure. Australian-based data centres ensured all information was stored in-country. 

The result

AEMO now has the capacity it needs to analyse and report on data from a repository that can form a consolidated single source of truth at a much faster pace, letting the business make better decisions and reducing costs considerably. 

AEMO can now: 

  • Let users analyse datasets at an unprecedented scale, using tools they are familiar with (such as R) 
  • Use the data in the solution to plan how the network will look in the future and what infrastructure is required to ensure continued high levels of service to Australian energy consumers 
  • Compare and contrast grids or groups of grids in their own right or against each other in order to uncover hidden trends in the data 
  • Create unit testable highly scalable forecasting models and able to automate their executions (this is a highly manual labour intensive process at the moment) 
  • Allow their analysts to focus on their strengths in model creation and algorithms instead of focusing on data engineering elements