Big data solution drives smarter energy consumption and improved forecasting accuracy across Australia

The situation

The Australian Energy Market Operator’s (AEMO) vision is to deliver energy security for all Australians. It administers and operates the country’s wholesale national energy markets, with around 18 terabytes of data stored across a myriad of different on-premises systems, including some high-volume transactional systems.

With energy consumption on the rise around the world, recent heatwaves saw Australians use more electricity than ever before.  This anomaly highlighted how difficult it is to gain insights into how electricity grids cope with the ebb and flow of energy consumption across all consumer and commercial meters.

The opportunity

Historically, AEMO’s data was used in isolation across its many systems. Different business units acted in silos, generating their own reports without a consolidated, single source of truth, meaning energy grids couldn’t be compared and increases or declines were not noticed. An internal initiative to aggregate data in a pseudo data warehouse was carried out, however the volume of data proved too much as certain queries were taking days to execute and ministerials were taking weeks or even months to answer.

Empired helped AEMO use big data to drive smarter ways to save energy consumption and improve forecasting accuracy across Australia. This solution is key to planning how the network will look in the future and the infrastructure required to ensure continued high levels of service to energy consumers.

The technology

Using Microsoft Azure’s best-of-breed data platform with rich ecosystems of data components, catering for AEMO’s evolving needs, Empired assisted AEMO in tapping into their energy consumption data. This data intelligence solution allows data to be analysed right across the country and across grids every two minutes in a flexible way.

Energy data, population data and weather data can all be factored in and assessed to forecast a future approximate for the next 20 years in 30 minute intervals. Furthermore, Azure provides the elasticity to be scalable depending on AEMO’s needs, achieving cost savings by scaling up only when required, then pausing or powering down when not in use, such as weekends. This is one of the most complicated and sophisticated uses of data intelligence in the world.

Empired designed and built big data warehouses using Azure SQL Data Warehouse, Azure Data Lake, Power BI, machine learning, Python, Spark, R and Spark-R, Data Factory and HDInsight. The data flows were designed to load data from AEMO’s on-premises systems to the Azure-based data warehouses, automating the build of the data flows from more than 700 tables (some larger than one terabyte) from on-premises systems in Azure. Australian-based data centres ensured all information was stored in-country.

The result

AEMO now has the capacity it needs to analyse and report on data from a repository that can form a consolidated single source of truth at a much faster pace, letting the business make better decisions and reducing costs considerably.

AEMO can now:

  • Let users analyse datasets at an unprecedented scale, using tools they are familiar with (such as R)
  • Use the data in the solution to plan how the network will look in the future and what infrastructure is required to ensure continued high levels of service to Australian energy consumers
  • Compare and contrast grids or groups of grids in their own right or against each other in order to uncover hidden trends in the data
  • Create unit testable highly scalable forecasting models and able to automate their executions (this is a highly manual labour intensive process at the moment)
  • Allow their analysts to focus on their strengths in model creation and algorithms instead of focusing on data engineering elements

This case study is part of the #cxreimagine series. For more experts' insights, clients' experiences and to download the whitepaper, click the banner.

For more experts' insights, clients' experience and to download the whitepaper, click the banner #cxreimagine

#cxreimagine series

This case study is part of our Digital Customer Experience Transformation series. Click here to download the whitepaper, read our experts' views and read other case studies. #cxreimagine


Data Insights, Business Intelligence and Advanced Analytics

Giving you actionable and meaningful insight into the workings of your organisation.


Microsoft Azure

Microsoft Azure is an open and flexible cloud platform that enables you to quickly build, deploy and manage applications that support your business.

Intergen RSS

Subscribe to our full RSS feed and stay up to date with all the latest content (blogs, news, videos, slide presentations, jobs) we post in our site.

Stay up to date with all insights from the Intergen blog