Using big data to drive smarter ways to save energy consumption and improve forecasting accuracy. Future proofing the network and delivering high service levels.
Data used in isolation across many systems. Business units working in silos with no consolidation or single source of the truth. Enquiries taking days or weeks to answer.
Microsoft Azure’s data platform with rich data components catered for evolving needs. Big data warehouses were designed and built to centralise data for quick access.
Capacity needed to analyse and report from centralised data that forms a consolidated single source of truth quickly. Better decision making and forecasting and reduced costs.
The Australian Energy Market Operator’s (AEMO) vision is to deliver energy security for all Australians. It administers and operates the country’s wholesale national energy markets, with around 18 terabytes of data stored across a myriad of different on-premises systems, including some high-volume transactional systems.
With energy consumption on the rise around the world, recent heatwaves saw Australians use more electricity than ever before. This anomaly highlighted how difficult it is to gain insights into how electricity grids cope with the ebb and flow of energy consumption across all consumer and commercial meters.
Historically, AEMO’s data was used in isolation across its many systems. Different business units acted in silos, generating their own reports without a consolidated, single source of truth, meaning energy grids couldn’t be compared and increases or declines were not noticed. An internal initiative to aggregate data in a pseudo data warehouse was carried out, however the volume of data proved too much as certain queries were taking
days to execute and ministerials were taking weeks or even months to answer.
Empired helped AEMO use big data to drive smarter ways to save energy consumption and improve forecasting accuracy across Australia. This solution is key to planning how the network will look in the future and the infrastructure required to ensure continued high levels of service to energy consumers.
Using Microsoft Azure’s best-of-breed data platform with rich ecosystems of data components, catering for AEMO’s evolving needs, Empired assisted AEMO in tapping into their energy consumption data. This data intelligence solution allows data to be analysed right across the country and across grids every two minutes in a flexible way.
Energy data, population data and weather data can all be factored in and assessed to forecast a future approximate for the next 20 years in 30 minute intervals. Furthermore, Azure provides the elasticity to be scalable depending on AEMO’s needs, achieving cost savings by scaling up only when required, then pausing or powering down when not in use, such as weekends. This is one of the most complicated and sophisticated uses of data intelligence in the world.
Empired designed and built big data warehouses using Azure SQL Data Warehouse, Azure Data Lake, Power BI, machine learning, Python, Spark, R and Spark-R, Data Factory and HDInsight. The data flows were designed to load data from AEMO’s on-premises systems to the Azure-based data warehouses, automating the build of the data flows from more than 700 tables (some larger than one terabyte) from on-premises systems in Azure. Australian-based data centres ensured all information was stored in-country.
AEMO now has the capacity it needs to analyse and report on data from a repository that can form a consolidated single source of truth at a much faster pace, letting the business make better decisions and reducing costs considerably.
AEMO can now: