Speech | Case Sharing - ClusterTech CHESS Helps China's Coal-to-Oil Technology Development Research
2022-04-06

 

 

"In the past, the clusters we built were relatively independent and managed with scattered open-source software. Now, with the use of ClusterTech's independently developed cluster management software CHESS, we have adjusted the cluster architecture, optimized resource utilization and scheduling, and improved user perception and overall cluster utilization rate through ClusterTech's professional and mature HPC products and services."

---- a Senior Engineer at the Synfuels China Technology Co., Ltd., Qiao Peixin

 

Introduction

In the industrial and engineering fields, the practical application of high-performance computing in simulation and modelling can not only reduce the number and cost of physical prototypes but also improve design quality and efficiency, enhance the means and capabilities of enterprises to solve complex technical problems, and ultimately shorten the time to market for new products, accelerate the introduction of new product types, and enhance the domestic and international competitiveness of enterprises in related industries.

On March 22, ClusterTech, in conjunction with Dell Technologies and China Synthetic Oil, explored the combination of High-Performance Computing and CAE, sharing a practical guide for the application of High-Performance Computing in the field of simulation and modelling.

Now we have organized the presentation transcript of this live broadcast for everyone's reference and learning. The following is the original text shared:

 

PART01 Overview of Synfuels China Technology Co., Ltd.

 

First, allow me to introduce Synfuels China Technology Co., Ltd.:

Established in 2006, Synfuels China Technology Co., Ltd. is mainly engaged in national special project research and development, coal-to-oil technology research and development, and proprietary equipment research and development for coal-to-oil. It is a joint venture with the Shanxi Institute of Coal Chemistry, Chinese Academy of Sciences, and has two national key laboratories for coal.

Synfuels China Technology Co., Ltd. is a technology-oriented company that integrates production, education, and research. It conducts basic research, develops coal-to-oil technology, produces catalysts and fine chemicals, and has engineering design and technical support capabilities. It also cooperates with foreign companies in technology development and promotion.

 

Relying on the Chinese Academy of Sciences system, Synfuels China Technology Co., Ltd. has been deeply involved in the coal chemical industry for many years and has solid fundamental theoretical research and analytical testing capabilities. By continuously upgrading coal-to-oil development technology, it supports coal-to-oil and catalyst factories, constantly reserves knowledge and talent for coal-to-oil technology development and engineering research, and has developed a set of technology-integrated innovation systems.

 

PART02 Overview of China Technology Co., Ltd.

 

Synfuels China Technology Co., Ltd. attaches great importance to fundamental theoretical research and makes investments in High-Performance Computing almost every year.

Over the course of ten years, the company has established 11 clusters and over 400 High-Performance Computing servers. These servers are equipped with applications such as VASP, Lammps, CP2K, Ansys, and others, serving the company's fundamental research and engineering development studies.

The computing power of the clusters has increased from an initial capacity of 3 trillion calculations per second to the current capacity of 30 trillion calculations per second. The computing network speed has also increased from 40 GB to the current 100 GB, providing an efficient and stable computing environment for quantitative calculations and fluid mechanics.

 

Previously, the clusters were relatively independent and utilized open-source software, resulting in relatively decentralized management. Now, the clusters utilize the self-developed CHESS system by LinkTech, which, along with their professional and mature HPC products and services, allows for the adjustment of cluster architecture, optimization of resource utilization and scheduling, and improvement of user perception and overall cluster utilization.

Cluster Topology Diagram

The existing old cluster continues to use InfiniBand (IB) network, while the new cluster adopts an OPA networking method and distributed storage. This allows for unified management and storage of data across all clusters, making it easier for future cluster expansions.

By connecting the management network for information and data statistics, it greatly facilitates user data management and downloads.

 

PART03: Usage of CHESS in China Technology Co., Ltd

 

Resource Management: Unified resource management for real-time dynamic utilization of resources

Through the CHESS system, resources are managed in a unified manner, integrating user requests, resource allocation, and resource monitoring. By standardizing user job submissions, the chances of job failures due to input errors are reduced, which greatly helps new users become familiar with the cluster.

Of course, experienced users who are familiar with the cluster often prefer to submit jobs using scripts, as it is a personal preference.

Users can accurately select the appropriate queue and number of nodes for their computations, and they can also view the queue status.

 

Based on years of experience in cluster usage, we have established scheduling and management rules. The CHESS system collects data on user usage hours and submitted job counts, monitors the start-to-finish time and resource usage of each job, and provides basic data for resource allocation. This ensures that every user can access computational resources for their submitted jobs. Any excess job requests are allocated resources based on their assigned weights.

 

The monitoring feature of the CHESS system allows administrators to track the activity status of servers. This enables prompt detection of server failures and easy identification of the problematic server's location, facilitating timely troubleshooting and issue resolution by operations and maintenance personnel.

 

Intelligent Scheduling: Establishing scheduling rules to achieve intelligent allocation of system resources

The high or low utilization of the cluster is reflected in the formulation and optimization of job scheduling rules.

We have divided the scheduling rules into four modules: scoring module, strategy module, queuing module, and service module.

The scoring module utilizes historical statistical data and summarizes user submission habits and job types over the years. The compute nodes are divided into multiple queues with different priorities based on business requirements. When a user submits multiple jobs, the system ensures that at least one job can be processed on the cluster. Then, it analyzes the user's recent resource usage and assigns a score to the current user's jobs. The scored jobs are then passed to the queuing module, where the strategy module determines the job queuing order according to predefined rules.

 

Statistical Analysis: Enhancing the evaluation system to monitor cluster usage

It's great to hear that you have provided feedback and suggestions during your usage of the CHESS system. One of your suggestions is to enhance the business reporting capabilities within the CHESS system by including metrics such as system utilization, job count, average wait time, and average response time.

These metrics can indeed provide valuable insights for analyzing and adjusting cluster scheduling strategies. In times of resource scarcity, adopting a tighter scheduling strategy can help optimize resource allocation. Conversely, when job submissions are low, a more relaxed strategy can be applied to ensure efficient cluster operation while maintaining a utilization rate of around 80 percent.

Moreover, it's excellent that you have implemented the functionality to query and download these metrics within the CHESS system.

 

Statistical data

Every month, we consolidate this data and generate reports to be delivered to the project team leaders, enabling them to understand the usage of the cluster in their respective domains. Additionally, during the usage process, users can also conveniently and visually monitor the cluster usage through the web interface of the CHESS system at any time.

 

Three-dimensional visualization for remote monitoring

In addition, CHESS is configured with and fine-tuned the remote desktop software NiceDCV, which integrates heterogeneous High-Performance Computing resources. This configuration enhances the efficiency and convenience of utilizing graphical processing resources.

The CHESS system can monitor the dynamic usage of job resources, facilitating the diagnosis of anomalies and providing valuable insights for debugging engineering designs, simulation computations, and efficient resource allocation.

 

The points mentioned above summarize several key aspects of the practical application of the CHESS system in the China Technology Co., Ltd.

 

PART04: Presentation of Achievements

 

Quantum computing

Quantum computing is one of the main areas of research in China Technology Co., Ltd.

In recent years, numerous research articles in the field of theoretical computation have been published in prestigious domestic and international journals such as Nature, totaling over 200 publications. Additionally, eight patent applications and two software copyrights have been filed, indicating the recognition of the theoretical research work.

As an essential supporting component, the achievements in theoretical research have received several awards and honors. These include the First Prize for Technological Invention from the China Petroleum and Chemical Industry Federation in 2019, the Second Prize for Scientific and Technological Progress in the Chinese Coal Industry in 2020, and recognition as one of China's Top Ten Scientific Advances in 2017.

 

Computational Fluid Dynamics (CFD) calculations

The field of engineering development primarily focuses on computational fluid dynamics calculations. Through numerical simulations, it investigates the complex multiphase turbulent flow field inside industrial slurry bed reactors, examines the performance of various internal component modules, and establishes a virtual, computationally based, large-scale slurry bed F-T synthesis reactor model. These calculations guide the scale-up of industrial slurry bed reactors and the development of internal component technologies. They provide research and design personnel with high spatiotemporal resolution multiphase flow field information that cannot be obtained through industrial experiments alone. This accelerates the development and design process while reducing development risks.

 

The development and research of coal-to-oil technology require high-quality scientific research achievements, and improving research capabilities is inseparable from High-Performance Computing.

We would like to express our gratitude to the technical personnel at ClusterTech for their support in our work. They possess a high level of professionalism and technical competence. We also hope that ClusterTech's technical personnel will continue to upgrade and optimize the CHESS system, contributing to the development of High-Performance Computing.

 

Thank you, everyone.