Q:

Speedup by AVX2 and AVX512

I am trying to visualize the speedup for incorporating AVX2 and AVX512
#include <stdio.h>
#include <stdlib.h>
#include <immintrin.h>
#include <omp.h>
#include <time.h>
int main()
{
  long i, N = 160000000;
  int * A = (int *)aligned_alloc(sizeof(__m256), sizeof(int) * N);
  int * B = (int *)aligned_alloc(sizeof(__m256), sizeof(int) * N);
  int * C = (int *)aligned_alloc(sizeof(__m256), sizeof(int) * N);

  int * E = (int *)aligned_alloc(sizeof(__m512), sizeof(int) * N);
  int * F = (int *)aligned_alloc(sizeof(__m512), sizeof(int) * N);
  int * G = (int *)aligned_alloc(sizeof(__m512), sizeof(int) * N);

  srand(time(0));

  for(i=0;i<N;i++)
  {
    A[i] = rand();
    B[i] = rand();
    E[i] = rand();
    F[i] = rand();
  }

  double time = omp_get_wtime();
  for(i=0;i<N;i++)
  {
    C[i] = A[i] + B[i];
  }
  time = omp_get_wtime() - time;
  printf("General Time taken %lf\n", time);

  __m256i A_256_VEC, B_256_VEC, C_256_VEC;
  time = omp_get_wtime();
  for(i=0;i<N;i+=8)
  {
    A_256_VEC = _mm256_load_si256((__m256i *)&A[i]);
    B_256_VEC = _mm256_load_si256((__m256i *)&B[i]);
    C_256_VEC = _mm256_add_epi32(A_256_VEC, B_256_VEC);
    _mm256_store_si256((__m256i *)&C[i],C_256_VEC);
  }
  time = omp_get_wtime() - time;
  printf("AVX2 Time taken %lf\n", time);

  free(A);
  free(B);
  free(C);

  __m512i A_512_VEC, B_512_VEC, C_512_VEC;
  time = omp_get_wtime();
  for(i=0;i<N;i+=16)
  {
    A_512_VEC = _mm512_load_si512((__m512i *)&E[i]);
    B_512_VEC = _mm512_load_si512((__m512i *)&F[i]);
    C_512_VEC = _mm512_add_epi32(A_512_VEC, B_512_VEC);
    _mm512_store_si512((__m512i *)&G[i],C_512_VEC);
  }
  time = omp_get_wtime() - time;
  printf("AVX512 Time taken %lf\n", time);

  for(i=0;i<N;i++)
  {
    if(G[i] != E[i] + F[i])
    {
      printf("Not Matched !!!\n");
      break;
    }
  }
  free(E);
  free(F);
  free(G);

  return 1;
}

So, the code is distributed in three phase. Three arrays are present. It is just a simple array addition. First we perform it using general loop, then using AVX2 and followed by AVX 512. I am using Intel Xeon 6130 processor.
The code is compiled using the command,
gcc -o test.o test.c -mavx512f -fopenmp -mavx2

The output is,
General Time taken 0.532550
AVX2 Time taken 0.175549
AVX512 Time taken 0.264475

Now, the speedup is visible incase of general loop and intrinsic implementations. But the time is increased from AVX2 to AVX512, which therotically should not be.
I have checked individual load, add, store operations. The store operation of AVX512 takes the maximum timings.
Just for the sake of checking if I remove the store operation from both of the code segment, the resultant timings are,
General Time taken 0.530248
AVX2 Time taken 0.115234
AVX512 Time taken 0.107062

Can anybody throw some light into such a behavior or is it expected ?
********* UPDATE 1 *********
After compiling with -O3 -march=native extensions, the new timings are,
General Time taken 0.014887
AVX2 Time taken 0.008072
AVX512 Time taken 0.014630

These are with all load, add, store instructions.
********* UPDATE 2 *********
Test 1 :
General loop has been modified as follows,
for(i=0;i<N;i++)
{
    //C[i] = A[i] + B[i];
    //G[i] = E[i] + F[i];
}

The output is,
General Time taken 0.000003
AVX2 Time taken 0.014877
AVX512 Time taken 0.014334

Therefore in both the cases page-fault is happening
Test 2 :
General loop has been modified as follows,
for(i=0;i<N;i++)
{
    C[i] = A[i] + B[i];
    G[i] = E[i] + F[i];
}

So, caching is done in both the cases.
Output is,
General Time taken 0.029703
AVX2 Time taken 0.008500
AVX512 Time taken 0.008560

Test 3 :
A dummy outer loop is added in all of the scenarios and the size of N is reduced to 160000.
for(j=0;j<N;j++)
{
    for(i=0;i<N;i+= /* 1 or 8 or 16 */)
    {
         // Code
    }
}

Now the output is,
General Time taken 6.969532
AVX2 Time taken 0.871133
AVX512 Time taken 0.447317

A:

Your AVX2 test reuses the same array you already wrote with the "general" test.  So it's already page-faulted in.
Your AVX512 test is writing to an array that hasn't yet been touched, and has to pay the cost of those pagefaults in the timed region.  Either dirty it outside the timed region, or just reuse C[] again.  Or mmap(MAP_POPULATE) also works, wiring up writeable pages.  (For real-world use, lazy page-faults may be better.  Letting the kernel zero a few pages right before you write them might reduce total cost by letting your real writes hit in L1d cache before the kernel's zeroing stores write back to outer caches.)
Notice that the "general" time (for the auto-vectorized first loop) is almost identical to the "AVX512" time.  (With gcc -O3 -march=native, GCC will auto-vectorize the "general" loop with 256-bit vectors, as per the default tuning of -mprefer-vector-width=256 for -march=skylake-avx512).
Those loops are doing basically the same work: reading 2 initialized arrays and writing a not-yet-touched array, causing page faults.

Lower clock speeds from using 512-bit vectors (limiting max turbo) shouldn't lower memory bandwidth much.  (You're going to bottleneck on memory with this 2 read / 1 write access pattern.)  If the uncore (L3 / mesh) slows down to match the fastest core, that could reduce bandwidth some, but it seems that effect is tiny if present at all.
Memory bandwidth for this STREAM-like test should be pretty much the same with 256 vs 512-bit vectors.  If you want to see a measurable speedup from 512-bit vectors for a problem with so little computation per memory bandwidth, you're going to need your arrays to fit in L1d cache and already be hot.  Or possibly L2 cache.    (Use a repeat loop around an inner loop that iterates over the array so it can run long enough for good timing precision).  AVX2 can easily keep up with L3 or memory for this so AVX512 won't help with big arrays unless you're doing more work per vector.

There's nothing weird about the asm loops once you enable optimization (https://godbolt.org/z/w4zcrC) so I had to take a closer look at what arrays you were actually writing.
A and B are probably fully evicted from cache even before the AVX2 loop runs (because your N is so large; like 662 MiB each for A, B, and C).  But it's still a bit weird to init different arrays for AVX2 vs. AVX512, and not to run any warm-up loop to make sure the CPU's at max turbo.
The "general" time basically acts as a warm-up loop for both clock speed and for page-faulting in the C[] array, so the actual time measured for it won't be indicative of memory bandwidth for writing to already-dirty memory.  You might be able to use perf to see how much time is spent in the kernel.

