HiveBrain v1.2.0
Get Started
← Back to all entries
patterncppMinor

Reducing memory usage of nested vectors

Submitted by: @import:stackexchange-codereview··
0
Viewed 0 times
vectorsreducingnestedusagememory

Problem

I recently asked this question about performance of an unordered_map, but I realize my question is probably more about this piece of code here.

I create a 8 x 10 x 30 x 30 x 24000 nested vectors of floats. This structure is used to keep track of votes in this 5D space. Many of the bins are never actually incremented, though (0 votes). The structure takes up 7.2GB of memory.

I initially thought to switch to a hash map (hence, my other question), but there's memory overhead in storing the 5D keys, and it's actually slower.

How can I improve this solution? I'd like to reduce the memory used, without giving up much speed.

Things I've thought of:

  • Switching to an unordered_map (was slower and used more memory)



  • Dropping to half-precision floats (I only need about 3-4 digits of precision, and the votes accumulated in any of the bins are in approximately [-3,3]) - haven't tried this



  • Replacing the inner vector with map - haven't tried this



I'd appreciate feedback about any of these ideas.

#include 
#include 

int main(int argc, char** argv) {
  std::default_random_engine generator;
  std::uniform_int_distribution random_z(1,24000);
  std::uniform_real_distribution random_vote(0.0, 1.0);

  // This makes an 8 x 10 x 30 x 30 x 24000 vector of vectors... of vectors.
  std::vector > > > > votes;
  for (size_t a = 0; a  > > > a_grid;
    for (size_t b = 0; b  > > b_grid;
      for (size_t y = 0; y  > y_grid;
        for (size_t x = 0; x (24000, 0));
        }
        b_grid.push_back(y_grid);
      }
      a_grid.push_back(b_grid);
    }
    votes.push_back(a_grid);
  }

  for (int i = 0; i  0.8) {
              votes[a][b][y][x][z] += this_vote;
            }
          }
        }
      }
    }
  }
  return 0;
}

Solution

Since your overall number of indices just fits into 31 bits, you could try a std::map:

uint32_t keyFrom5DIndex(uint32_t a, uint32_t b, 
                        uint32_t x, uint32_t y, uint32_t z)
{
    return z+24000*(x+30*(y+30*(b+10*a)));
}

std::map votes;
// Compute indices
votes[keyFrom5DIndex(a,b,z,y,z)] += this_vote;

Code Snippets

uint32_t keyFrom5DIndex(uint32_t a, uint32_t b, 
                        uint32_t x, uint32_t y, uint32_t z)
{
    return z+24000*(x+30*(y+30*(b+10*a)));
}

std::map<uint32_t, float> votes;
// Compute indices
votes[keyFrom5DIndex(a,b,z,y,z)] += this_vote;

Context

StackExchange Code Review Q#27066, answer score: 3

Revisions (0)

No revisions yet.