[PrivateAI] Machine Learning with Fully Homomorphic Encryption on real neural network example

Introduction

This report details the performance evaluation of the ResNet18 model utilizing Fully Homomorphic Encryption (FHE) with the Concrete-ML library from Zama. The evaluation leverages the ImageNet-1k dataset and assesses performance across various stages, including original model evaluation on GPU and CPU, model quantization, and FHE simulation. The hardware used includes an AMD EPYC 7R32 CPU with 8 threads and an NVIDIA A10G GPU.

‍

Experimental Setup

Hardware Configuration

• CPU: AMD EPYC 7R32 with 8 threads
• GPU: NVIDIA A10G

‍

Dataset

• Data: ImageNet-1k
• Evaluation Samples: 100 test samples

‍

Results

‍

1. Evaluation of Original Model on GPU

• Accuracy: 66%
• Top-5 Accuracy: 87%
• Time: 0.5998 seconds

‍

2. Evaluation of Original Model on CPU

• Accuracy: 66%
• Top-5 Accuracy: 87%
• Time: 2.5067 seconds

‍

3. Model Quantization

• Time: 150.8716 seconds

‍

4. Evaluation of Quantized Model

• Accuracy: 58%
• Top-5 Accuracy: 82%
• Time: 115.8231 seconds

‍

5. Evaluation of FHE Simulation

• Accuracy: 56%
• Top-5 Accuracy: 82%
• Time: 699.8050 seconds

‍

6. Evaluation of FHE on Single Image

At the moment it was not possible to perceive the results on FHE without simulation (code execution aborts with an error)

• Accuracy: None
• Top-5 Accuracy: None
• Time: None

‍

Analysis

Performance Comparison

1. Original Model Performance
   
   The original model's evaluation on the GPU demonstrates a high accuracy of 66% and top-5 accuracy of 87%, with a swift processing time of approximately 0.6 seconds. In contrast, the same model evaluated on the CPU shows identical accuracy metrics but takes significantly longer, at around 2.5 seconds.
   
   
2. Quantization Impact
   
   The quantization process, although time-intensive (taking approximately 150.9 seconds), reduces the model's accuracy to 58% and top-5 accuracy to 82%. The evaluation time for the quantized model increases substantially to approximately 115.8 seconds, indicating the computational overhead introduced by quantization.
   ‍

3. FHE Simulation Performance
   
   The FHE simulation further impacts performance, with a slight drop in accuracy to 56% and maintaining the top-5 accuracy at 82%. However, the time taken for evaluation balloons to about 699.8 seconds, highlighting the significant computational demands of FHE.

‍