Introduction

On the afternoon of October 12, 2022, the Chinese Society for Electrical Engineering organized and held an online technical appraisal meeting for the achievement of “Million--node-level Electromagnetic Transient High-performance Supercomputing Cloud Simulation Platform for New-type Power Systems” completed under the leadership of Tsinghua Sichuan Energy Internet Research Institute. Experts of the appraisal committee agreed that the achievement is internationally leading. The features and technical indicators are introduced below.

Development background

In September 2020, China announced to the world at the United Nations General Assembly the goals of realizing carbon peaking by 2030 and carbon neutrality by 2060. Driven by the carbon neutral policy, China’s energy structure will gradually transform and promote the development of new power systems with a high proportion of renewable energy and a high proportion of power electronics equipment. Significant changes have taken place in the physical form and operating characteristics of new power systems, the microsecond-level power electronic switching process interacts with the millisecond and second-level AC motor transition process, resulting in increased nonlinearity and uncertainty, and thus the scientific research, construction and operation of new power systems are more dependent on the full electromagnetic transient simulation analysis.

Simple calculations show that the total number of electrical nodes and control nodes in the electromagnetic transient model of a large-scale AC/DC hybrid power system considering the detailed model of new energy stations will increase sharply, and the simulation scale of a regional power grid can easily reach a million-node scale. Therefore, there is an urgent need to develop a new full-electromagnetic transient simulation tool for million-node power systems.

Simulation scale of typical new power systems

Estimated scales of simulation computation for new power systems

Facing the ever-increasing demand for high-performance all-electromagnetic transient simulation of new power systems, current commercial simulation tools encounter technical bottlenecks such as limited calculation scale, low simulation efficiency, and dependence on imported computing chips, making it difficult to realize full electromagnetic transient simulation for million-node new power systems. To this end, this project has studied the theory, computation method and simulation platform of the electromagnetic transient modeling and simulation of new million-node power systems, carried out the full electromagnetic transient multi-scale modeling of AC and DC power grids, high-efficiency parallel simulation algorithms and domestic supercomputing high-performance cloud simulation platform, created a completely independent and controllable high-performance supercomputing cloud simulation platform suitable for ultra-large-scale new power systems, and formed an open, shared and flexible power system cloud simulation model and application ecosystem. The research achievement will be applied to multi-time scale, dynamic, accurate and fast simulation and safety analysis of new power systems, which is of great significance for guarding the safe and stable operation of new power systems.

Achievement features

The project achievement have realized the complete localization replacement and performance surpassing of the ultra-large-scale electromagnetic transient simulation software and hardware platform. The most critical and differentiated technical parameters of the superior platform are:

(1) Large-scale simulation: supporting ultra-large-scale electromagnetic transient simulation with a scale of one million computing nodes;

(2) New modeling method: with the multi-time scale electromagnetic transient modeling and multi-rate simulation method under the same simulation accuracy, the simulation step length of a large-scale new power system can be increased by 40~100 times;

(3) High simulation efficiency: compatible with x86, ARM, Shenwei many-core, domestic c86, GPU and other architecture processors, multi-level hybrid parallel simulation algorithms, suitable for heterogeneous processors and domestic supercomputing platforms; million-node 2000-scenario 2s process full electromagnetic transient simulation (step length 50μs), 64,000 cores for concurrent simulation, process < 15 minutes, and satisfying the demand for online dynamic security analysis (verified at Chengdu Supercomputing Center).

(4) Strong platform expansibility: native cloud computing architecture, providing models, algorithms and full-dimensional secondary application development tools, as well as an open, shared, flexible and efficient application ecosystem.

Achievement application

Up to now, the project achievement has been applied in many power grid companies, scientific research institutes and universities, and it has realized the replacement of foreign simulation equipment, reduced cost on equipment purchase, improved work efficiency, and saved labour. The cloud simulation platform, selected as a shared campus software of Tsinghua University to significantly improve the teaching effect, has been applied to electrical engineering teaching in South China University of Technology, Beijing Jiaotong University, China Agricultural University, Shanghai Jiaotong University and North China Electric Power University, etc.

In addition, the project team has adapted the platform to domestic processors, completed functional verification and performance test and built a large-scale electromagnetic transient supercomputing cloud simulation platform based on Chengdu Supercomputing Center, which can satisfy the demand for online dynamic safety analysis of million-node new power systems.

The research achievement can be extensively used in:

(1) Regional AC and DC power grids to support system optimization planning and safety assessment, large-scale new energy grid-incorporated design and optimization regulation, massive power electronics equipment interaction and stability mechanism analysis, etc, as well as key applications in new power system planning, design, test and detection, safety assessment, optimization and regulation, etc to improve the low-carbon, economical and safe operation level of China’s new power systems;

(2) Electrical equipment innovation and development enterprises to develop simulation-driven R&D tool chains for digital products, improve the optimal design and manufacturing capabilities of AC and DC electrical equipment, accelerate the development of key electrical equipment in new power systems, reduce grid-incorporated debugging cost, and support intelligent operation and maintenance business;

(3) Power and energy-related universities and scientific research institutions to support the construction of digital simulation laboratories for new power systems, form a full-digital scientific research accelerator solution for theoretical analysis, simulation and closed-loop verification, and improve China’s basic R&D and technological innovation capability in the field of power and energy.

Category	Technical indicator		Platform of this project
Modeling and simulation methods	AC system frequency-shifted electromagnetic transient model and simulation	Sequence-component frequency-shifted electromagnetic transient model and simulation algorithm	Computation efficiency for 3-phase AC systems improved by 2.7 times at least
		Frequency-shifted electromagnetic transient model and simulation algorithm based on improved root matching	Error < 5% when the step length expanded to 5ms
		Step length of motor electromagnetic transient model simulation	≤5ms
	DC system fast electromagnetic transient model and simulation	Modularized efficient modeling and simulation method for voltage source converter	Switch prediction supported, step length expanded by 2 times at least, efficiency improved by 3 times
		MMC modularized efficient modeling and simulation method	Model equivalent to constant admittance interface, efficiency improved by 5 times
	Multi-dimensional integrated modeling simulation Modeling frame	Interface type and accuracy	Ideal transmission interface supported MATE1 interface supported
			Max. step length/Min. step length >500 Interface accuracy loss < 0.5%
		Multi-partition differentiated modeling	“Routine - routine”, “frequency shifted - frequency shifted” and “frequency shifted - routine” multi-partition electromagnetic transient modeling supported
		Partition differentiated simulation step length	Multi-rate simulation supported
Simulation accelerating technology	Ultra-large-scale electromagnetic transient heterogeneous parallel accelerated computing method	Processor architecture	x86, arm64, domestic x86 c86), domestic SW many-core (alpha), NVIDIA GPU
		Parallel computing mode	Support coarse and fine-grained mixed parallel computing for heterogeneous many-core processors
		Batch simulation acceleration for massive scenarios	Multi-scenario fine-grained batch parallelism based on multi-layer computational graph model
	Steady-state initialization method for power flow section	Designated power flow section startup	Supported
		Million-node electromagnetic transient initialization time	< 30s
		Steady-state startup error	< 0.1%
Supercomputing platform construction and application technology	Communication architecture	Communication and synchronization mechanisms	Implicit synchronization mechanism supported Synchronization time can be reduced by up to 80%
		Multiple simulators Co-simulation interface	Multi-emulator interconnection supporting FC-AE2 and AURORA3 protocols
	Supercomputing cloud platform framework	Platform architecture	Native cloud architecture Deployed in computing environments such as stand-alone computers, private clouds, and public clouds
		Kernel openness	Provide tools for model and application development Support third-party computing kernel access
		Application development and deployment	Support agile construction and flexible deployment of simulation applications
		Technical independence	Completely independent intellectual property rights
	Performance indicator	10s process simulation of 10,000-node system (50μs)	60-core parallel (main frequency 4.3G) Time <50s
		10s process simulation of 100,000-node system (50μs)	512-core concurrent (main frequency 4.3G) Time <18s
		2000-scenario 2s simulation of million-node system (50μs)	60-core parallel (main frequency 4.3G)4 Time <15min, PUE6=1.04
Note: 1. MATE: multi-area Thevenin equivalent interface 2. FC-AE protocol: fiber channel communication protocol dedicated to avionics, an open communication protocol with the characteristics of high speed, certainty and low delay, and networking communication. 3. AURORA protocol: a link layer high-speed communication protocol developed by Xilinx, which supports point-to-point low-latency communication 4. Based on the test of the privatized supercomputing cloud simulation platform, the processor is AMD3990x 5. Test on the open supercomputing cloud simulation platform in the National Supercomputing Center in Chengdu, and the processor is Sugon HygonC86-7285 6. PUE: Power Usage Effectiveness, supercomputing energy consumption index, the closer to 1, the higher the energy efficiency

Attachment: Main technical specifications and technical level of the achievement

(1) Basic simulation functions:

1) Electromagnetic transient modeling and simulation;

2) Three-phase symmetrical and asymmetrical power flow analysis of AC and DC power grids;

3) Analysis of short-circuit current of AC and DC power grid;

4) Analysis of single/multi-port frequency sweep and impedance characteristics of electrical equipment and networks;

5) control protection model packaging and closed-loop test of new energy and power electronics equipment manufacturers;

6) Model conversion function, which supports importing simulation examples from computation example files generated by simulation software such as PSASP, BPA and PSCAD;

7) Transient simulation power flow initialization and simulation section preservation;

8) It has an industrial control data exchange interface, which can output simulation results and simulate closed-loop system control through protocols such as 485, DNP and 61850;

9) Support data conversion through the interface between SCADA/GIS;

10) Graphical simulation modeling and data analysis.

 

Electromagnetic transient modeling and simulation

 

Three-phase symmetrical power flow analysis

 

Three-phase asymmetric modeling and asymmetric power flow analysis

 

Conversion of large-scale system model of a regional power grid based on BPA

(2) Advanced simulation functions:

1) Frequency-shifting electromagnetic transient modeling and simulation;

2) Multi-partition frequency shift, conventional electromagnetic transient modeling and hybrid simulation;

3) “Device-partition-system-scenario” multi-level hybrid parallel simulation, supporting single-unit multi-core grid-partitioned parallel operation and multi-unit multi-core grid-incorporated parallel operation, and providing automatic grid partitioning and load balancing, manual configuration of load balancing, and automatic generation of codes for parallel operation, etc;

4) Real-time simulation and hardware-in-the-loop test;

5) Multi-simulator joint simulation based on high-speed data exchange, and joint simulation application with RTDS, RT-Lab and ADPSS;

6) Event-driven cyber-physical power system simulation and in-the-loop test.

 

Kernel configuration interface of Multi-level hybrid parallel simulation

 

Joint real-time simulation interface

(3) Scale of simulating computation:

1) There is no upper limit for the number of various components in the simulation case;

2) In the engineering cases which have been put into practical use, there are more than 1.2 million simulating computation nodes at maximum, including more than 200,000 electrical computation nodes and more than 1 million control computation nodes; not less than 5,000 new energy devices which can be simulated (specific electromagnetic transient simulation model) at maximum; not less than 50 conventional DC and flexible DC transmission systems which can be simulated at maximum;

3) The simulation step length is 10ns~10ms, supporting multi-partition and multi-rate simulation; the maximum step length of the AC system partition is >5ms, and the maximum step length of the DC system partition is >100us;

(4) Efficiency of simulating computation

1) Benchmark simulation efficiency: with a 4.3GHz main frequency processor as the benchmark computing device and considering single-core and multi-core parallel concurrent operation scenarios, the simulation efficiency of AC, DC and new energy systems of different scales and complexities are shown in the table below:

 

Efficiency of basic simulation

2) Efficiency of electromagnetic transient simulation on ultra-large-scale systems, a 10s process simulation (step length 50us) of 100,000-node electromagnetic transient simulation takes less than 18s in 512-core (4.3GHz main frequency) parallel concurrent operation; a 2s process simulation (step length 50us) of 1,000,000-node electromagnetic transient simulation in 64000-core (2GHz main frequency) parallel grid-incorporated operation takes less than 15min;

3) Efficiency of simulation power flow initialization: the initialization of electromagnetic transient simulation of a million-node system takes less than 30s;

4) Analysis of simulation examples and generation of executable (parallel) programs: the initialization of electromagnetic transient simulation of a million-node system takes less than 2min.

(5) Accuracy of simulation analysis:

1) Accuracy of the simulation model: unit test of all the equipment models (independent equipment plus excitation source test) reveals an equivalent simulation accuracy compared with foreign commercial simulators under the same settings;

2) Numerically stable in Long-term simulation, no damage to numerical stability in continuous simulation of any system for 3600s, and cumulative relative error within 0.1%.

(6) Secondary development function:

1) It supports user and third-party custom models, and can customize simulation models in Python, C, C++, Matlab, Octave and other languages;

2) It can be connected to the black box model components provided by the manufacturer;

3) Provide various simulation function API interfaces, support Python, Java and other languages to customize simulation process and carry out simulation result analysis;

4) Provide functional microservice encapsulation tools, which can encapsulate basic simulation functions, custom simulation processes, simulation result conversion and analysis into distributed execution microservice;

5) Provide visualized and low-code simulation application development tools, and support visualized simulation application interface design and application release.

 

User-defined model (C/C++, Matlab/Octave, Python and other interfaces)

 

Access to the manufacturer’s black box model (directly imported into the Simulink s-function black box model)

 

CloudPSS simulation function API (SDK)

 

Functional microservice development, packaging and distributed deployment

 

Visualized, low-code simulation application development and release platform

 

Example of application developed based on the simulation application development platform