Connduino Projects

An Arduino Uno combatible board with enhanced connectivity

Fast EEPROM operations

This is the second article about using an external EEPROM I2C chip with an Arduino or any compatible board, like ConnDuino. In a previous article we covered the basics of reading and writing data to the EEPROM. We were able to read and write any data-type, having arbitrary size, but we had to decompose and transfer them to the EEPROM byte after byte.

In this article, the writing and reading of continuous blocks will be explained. These operations are a little bit more complicated to implement and some better understanding of the EEPROM is required, but for large objects they are a magnitude faster.

The problem

Writing and reading of single bytes to/from EEPROM, causes a considerable overhead. For writing, the most significant reason is the required delay for the operation to be completed. According to the datasheet of the 24LC256 chip the maximum write delay is 5ms. If a second write operation is issued within this time, the first write will be rather aborted. Indeed, you may have noticed, inside the main loop of the writeObjectSimple function, that is responsible for writing any data to the EEPROM, the statement delay(5), after issuing the write command to the I2C bus. Here is the code snippet:

for (i = 0; i  < sizeof(value); i++){
    Wire.beginTransmission(i2cAddr);
         Wire.write((uint16_t)(addr >> 8));  // MSB
         Wire.write((uint16_t)(addr & 0xFF));// LSB
         Wire.write(*p++);
    Wire.endTransmission();

    addr++;
    delay(5); // < < < < < < < < < overhead
}

Indeed, a 5ms delay is not that long, and the above loop should be lightning fast, if we were to write one or two variables. A float, that is 4 bytes in size, would take 20ms of total delays to be written. But what if we had to write 300 floats? That would take 300*4*5ms = 6000ms of delays, six whole seconds! This is tremendous overhead.

A second source of overhead, when writing or reading single bytes is the additional data that get streamed with our byte. Look at the next figure, taken form the 24LC256 datasheet, which describes byte writing. The byte we write appears at the right, as “Data”. This byte is preceded by a “Control byte” and two bytes for the memory address, named “Address High Byte” and “Address Low Byte”. Without getting too technical, a single byte of data causes the transfer of four bytes through the I2C bus. In other words, we can achieve a utilization factor, no more than 25%. Similar is the situation for reading. This time five bytes should be transferred for a single byte read.

The solution

The solution to the above problems is to write or read blocks of bytes. The 24LC256 EEPROM supports the so called “page writing” for sending a whole page of data to it, that is equal to 64 bytes. The 5ms write delay is again required, but not for each byte, but for the whole page! Using the same example, if we try to write 300 float variables, in blocks of 64 bytes we should send: (300*4)/64 = 19 pages that require 19*5 = 95ms delay. Compare this to the 6000ms delay required when writing single bytes…

Furthermore, the final data stream now looks like a mass transportation system, as shown in the figure below, for page writing. If we want to write a full page, of 64 bytes, we end up to a packed size of 67 bytes, achieving a utilization factor close to unity (64/67 = 0.96). Perfect.

Page operations are relevant for writing only, because reading can be done sequentially without the page limit of 64 bytes. We can read the entire EEPROM from the first byte, up to the last one, as one block; provided we have a large enough buffer to store the read bytes.

How many bytes we may transfer in one block really

As described above, from hardware perspective, we may write 64 bytes and read unlimited ones, in one block. But from the software perspective, these numbers are probably not realized. In an Arduino program, any communication using the I2C interface is handled by the Wire library. Any data, we send or get from an I2C device are stored in a buffer by this library. This buffer is 32 bytes long, with its first two bytes reserved to store the buffer size. Thus, the maximum number of bytes we may transfer, without modifying the Wire library, is 30 bytes. For reasons explained later though, we strongly want our block length, to be an integer division of the 64 byte page and with this we reach a final block of 64/4=16 bytes.

Hacking the Wire Library

For some projects, modifying the buffer in the Wire library may be preferable. This is easily accomplished if we edit the following line in the Wire.h file:

#define BUFFER_LENGTH 32 

To take advantage of full 64 byte page writing we should change the above value to 66 (we add 2 bytes for the buffer size). The Wire.h file is located in normally in the following path: “Arduino dir / libraries / Wire”, where “Arduino dir” is the Arduino installation folder. You may have to copy the file to your desktop, edit it, and then paste it back, to overcome the operating system protection.

How to write the EEPROM in blocks

Hopefully enabling page writing couldn’t be more straightforward. Let’s compare the data streams for byte and page writing to the EEPROM, extracted from the 24LC256 datasheet.

The key difference is that for page writing the STOP bit is postponed until all data bytes have been transferred. Also the control and address bytes are streamed at the beginning only once. It is very easy to modify the writeObjectSimple function we have shown in the previous article, in order to write multiple bytes in one operation. The new function is called writeObject, and may be defined like this:

#define I2CADDR 0x50

template  <class T>
uint16_t writeObject (uint16_t addr, const T& value){
      const uint8_t* p = (const uint8_t*)(const void*)&value;
                
     //Outside the loop. Streamed once at the beginning
     Wire.beginTransmission(I2CADDR);
     Wire.write((uint16_t)(addr >> 8));  // MSB
     Wire.write((uint16_t)(addr & 0xFF));// LSB           

     uint16_t i;
     for (i = 0; i  < sizeof(value); i++){     
          Wire.write(*p++); //dispatch the data bytes
     }

     //Outside the loop. Streamed once after all 
     //data bytes have been dispatched
     Wire.endTransmission();
     delay(5);   //required write delay 5ms
     return i;
}

What are the changes, compared to the writeObjectSimple? The Wire.beginTransmission statement and the address writing have been moved before the main loop, in order to write to the Wire buffer the control and address bytes, only once at the beginning. Also, the Wire.endTransmission() has been moved after the main loop in order to dispatch the Wire buffer, after all data bytes have been written to it inside the main loop.

But what happens if the size of the sent data is bigger than the Wire library buffer capacity, typically being 30 bytes? Probably nothing will be written to the EEPROM. The above function is not safe in this regard because it’s not enforcing a block size and can easily be misused, if sending large objects to it. Instead, we should handle such large objects, by writing them to the EEPROM in multiple blocks. The code below does that, by keeping track of the available space in a block, using the blockBytes variable. If the space is depleted (blockBytes==0) and more data remain to be sent, a new block is setup.

#define BLOCKSIZE 16
#define I2CADDR 0x50

template  <class T>
uint16_t writeObject(uint16_t addr, const T& value){
      const uint8_t* p = (const uint8_t*)(const void*)&value;

      Wire.beginTransmission(I2CADDR);
      Wire.write((uint16_t)(addr >> 8));        // MSB
      Wire.write((uint16_t)(addr & 0xFF));      // LSB

      //counts the bytes we may send before our block becomes full
      uint8_t blockBytes = BLOCKSIZE;      
      uint16_t i;
      for (i = 0; i  < sizeof(value); i++){
            if (blockBytes == 0){
                  //block is full;
                  Wire.endTransmission(); //dispatch the buffer
                  delay(5);
                  //restart new block
                  addr += BLOCKSIZE;
                  blockBytes = BLOCKSIZE;
                  Wire.beginTransmission(I2CADDR);
                  Wire.write((uint16_t)(addr >> 8));  // MSB
                  Wire.write((uint16_t)(addr & 0xFF));// LSB
            }
            Wire.write(*p++); //dispatch the data byte
            blockBytes--;     //decrement the block size
      }

      Wire.endTransmission();
      delay(5);   //required write delay 5ms
      return i;
}

Let’s test the above function with an example. First, we will define a new class, containing enough data members, and some functions. The size of an object of this class should be 27 bytes, larger than our 16 byte block size.

class MyData{
      long result;  
public:
      float  f1;
      float  f2;
      double f3;
      int    i1;
      long   i2;
      long   i3;
      bool  negative;
  
      long getResult(){
            return result;
      }
      long calcResult(){
            //makes a calculation with its data
            //and stores the result to a private variable
            long    i = (i2+i3)/i1;
            double  f = f3/(f1+f2);
            result = f/i;
            //round off
            result += result>0 ? 0.5 : -0.5;
            if (negative) result=-result;
            return result;
      }    
      MyData() : result(0) {}
};

Next we will write a test, inside the setup function.

#define MEM_ADDRESS 0

void setup(){
     Wire.begin();    
     Serial.begin(9600);

      //data - result should be 20
            MyData data;
            data.i1 = 10;
            data.i2 = 150000;
            data.i3 = 350000;
            data.f1 = 0.5;
            data.f2 = 2.5;
            data.f3 = 3.0E+6;
            data.negative = false;

      Serial.print("Size of each object: "); 
      Serial.println(sizeof(data)) ;
      Serial.println ("---------------------");
      Serial.println ("Data before writing to eeprom");
            data.calcResult();
            Serial.print("Data result = ");
            Serial.println(data.getResult());

      //Write to eeprom
      Serial.print ("Writing data to eeprom");
      writeObject (MEM_ADDRESS, data);
      Serial.println ("...ok.");
           
      //Read data back from eeprom
      delay(1000);
      MyData newdata;
      readObjectSimple(0x50, MEM_ADDRESS, newdata);

      //Evaluate
      Serial.println ("---------------------");
      Serial.println ("Data read from eeprom");
      Serial.print ("Stored result = ");
      long storedResult = newdata.getResult(); 
      Serial.println(storedResult);
      Serial.print ("Recalculated result = ");
      long newResult = newdata.calcResult();
      Serial.println(newResult);
      if (storedResult== newResult)
            Serial.println ("SUCCESS");
      else Serial.println ("FAIL");
}

For this test we create and initialize an object of the MyData class and write it to the EEPROM, at the address 0, using the writeObject function. Then the EEPROM is read back, using the readObjectSimple shown in the previous article (an updated sequential read function will be described later). The read bytes are stored to a new MyData object. Finally, the new object is evaluated. The output to the serial port should be like this:

    Size of each object: 27
    ---------------------
    Data before writing to eeprom
    Data result = 20
    Writing data to eeprom...ok.
    ---------------------
    Data read from eeprom
    Stored result = 20
    Recalculated result = 20
    SUCCESS

Apparently, the MyData object was written in two blocks:

  • the 1st starting at address 0 and having a full size of 16 bytes.
  • the 2nd starting at address 16 and having a size of 27-16 = 11 bytes.

The matching of the result variable before and after, proves that the data was handled correctly by the writeObject function. To be sure, to let’s retry the same test with a different memory address, let’s say 56. We have to modify the MEM_ADDRESS definition to the next one:

#define MEM_ADDRESS 56

The output to the serial port should be like this:

    Size of each object: 27
    ---------------------
    Data before writing to eeprom
    Data result = 20
    Writing data to eeprom...ok.
    ---------------------
    Data read from eeprom
    Stored result = 20
    Recalculated result = 0
    FAIL

Something went wrong! The stored result was read correctly but the recalculated one is wrong. This means that some of the data members haven't been written correctly. If we examine the read object, we will find that its data members have the following values:

      f1 = 0.50
      f2 = 0.00
      f3 = 0.00
      i1 = 10
      i2 = 150000
      i3 = 350000
      negative = 0

The f2 and f3 values are wrong. Whatever caused this behavior, is surely related with the new address we used because this is the only change we made.

Back to the 24LC256 datasheet, we find that we cant’t write a block anywhere we want. The EEPROM memory space is partitioned in pages of 64 bytes. The 1st page occupies the address space 0-63 the second one 64~127, etc.

It is not allowed to cross the border of a page when writing a block of bytes, in a single pass. If we try to do so, writing will not continue to the next page but to the start address of the same one, overwriting whatever lies there. This is what happened to our 2nd test. We attempted to write the 27 bytes of our object starting at address 56. The page border was at address 63, thus we were able to write correctly the first 63-56 = 7 bytes of the first 16 byte block. The next 9 bytes of the first block, ended up at the addresses 0~8. We supposed they were written at addresses 64~73 though, and that’s why they failed to get read back to members f2, f3.

A function that works for writing any object at any address

The above problem can be solved in two ways:

  • Write all objects strictly at addresses that are aligned with the block size of 16 bytes. For example 0, 16, 32, 48, 64, 80, …. No page border will be crossed because a page of 64 bytes contains an exact number of blocks: 64/16=4.
  • Write at any address, but use for the first block a customized size (1~16 bytes), that would make us reach at an address, exactly aligned to our block size of 16 bytes (an address exactly divided by 16). Thus all subsequent blocks would not cross a page border.

The second option is friendlier and fool proof, so it will be implemented as our final write function which is shown below. This time the blockBytes variable is not set to 16 bytes initially. Instead, it is set to the number of bytes up to the next 16-byte aligned address. Subsequent blocks though have constant 16-byte length. Since the block size is not constant now, care should be taken how many bytes to advance the address once a new block is initiated. A solution is shown inside the if (blockBytes == 0){...} case below.

#define BLOCKSIZE 16
#define I2CADDR 0x50

template  <class T>
uint16_t writeObject(uint16_t addr, const T& value){
      const uint8_t* p = (const uint8_t*)(const void*)&value;               

      Wire.beginTransmission(I2CADDR);
      Wire.write((uint16_t)(addr >> 8));        // MSB
      Wire.write((uint16_t)(addr & 0xFF));      // LSB

      //in the loop: counts the bytes we may send before 
      //our block becomes full
      //but initialise it to the number of bytes up to the
      //next 16-byte aligned address
      uint8_t blockBytes = (addr/BLOCKSIZE + 1)*BLOCKSIZE - addr;       
      uint16_t i;
      for (i = 0; i  < sizeof(value); i++){
            if (blockBytes == 0){
                  //block is full;
                  Wire.endTransmission(); //dispatch the buffer
                  delay(5);
                  //restart new block
                  addr = (addr/BLOCKSIZE + 1)*BLOCKSIZE;
                  blockBytes = BLOCKSIZE;
                  Wire.beginTransmission(I2CADDR);
                  Wire.write((uint16_t)(addr >> 8));  // MSB
                  Wire.write((uint16_t)(addr & 0xFF));// LSB
            }
            Wire.write(*p++); //dispatch the data byte
            blockBytes--;     //decrement the block space
      }
      Wire.endTransmission();
      delay(5);   //required write delay 5ms
      return i;
}

Running the failed test, again with this function should result a correct outcome. This is the function we want.

How to read sequentially from EEPROM

Reading multiple bytes from the EEPROM in a single operation is similar to writing, but with a significant advantage: there are no page border limitations. The memory can be read from any start address and continue to the next page with no problem. The only limit that remains is the Wire Library buffer capacity of 30 bytes. This time though, no benefit exists in selecting a block size exactly dividing the 64 byte page. We can use a block size of 30 bytes for reading, which the default buffer capacity is.

The next function reads from EEPROM an object of arbitrary size, starting from any address. The operation is done in blocks of 30 bytes length. The blockbytes variable counts the bytes we may read before the Wire library buffer is depleted. When blockbytes becomes zero, a new block is requested form the EEPROM, inside the loop. It is quite possible to read some bytes past the end of the requested object, but this means no trouble, because the loop will stop when the required bytes for the object are reached. However, for optimal performance, when dealing with small objects, a condition is inserted, not to request blocks larger than the size of the object.

#define BLOCKSIZE_READ 30

template  <class T>
uint16_t readObject(uint16_t addr, T& value){
      uint8_t* p = (uint8_t*)(void*)&value;

      Wire.beginTransmission(0x50);
            Wire.write((uint16_t)(addr >> 8));        // MSB
            Wire.write((uint16_t)(addr & 0xFF));      // LSB
      Wire.endTransmission();

      //counts the bytes we may read before buffer is depleted
      uint8_t   blockBytes = 0;
      uint16_t  objSize = sizeof(value);
      uint16_t  i;
      for (i = 0; i  < objSize; i++){
            if (blockBytes==0){
               //we need a new block
               blockBytes = BLOCKSIZE_READ;
               if (objSize  < blockBytes) blockBytes = objSize;
               //get the new block
                Wire.requestFrom((uint8_t)0x50, blockBytes);
            }
            if(Wire.available()){
                  //read a byte from buffer
                  *p++ = Wire.read();
                  blockBytes--;
            }
      }
      return i;
}

Grand test

This is quite a big example. Five MyData objects are created, then written to the EPPROM, and then read back to different variables. The final outcome is evaluated and some messages should appear to the serial port. But there is more. The EEPROM operations may be accomplished either using blocks, as described in this article, or with single bytes as described in a previous article. The program measures the time it takes for each operation, and we should make some comparisons about the actual benefits of the block EEPROM operations.

You have to change the value of the BLOCK_OPERATIONS definition to 0 and recompile the program, in order to use the single byte functions. Don’t forget to include these in the "eeprom_routines.h" header file.

#include  <Wire.h>
#include "eeprom_routines.h" //definitions of the functions:   
                             //writeObject, readObject, 
                             //writeObjectSimple, readObjectSimple
#define BLOCK_OPERATIONS 1

#if BLOCK_OPERATIONS>0
  #define __WRITE__(ad, obj) writeObject(ad,(obj))
  #define __READ__(ad, obj) readObject(ad,(obj))
#else
  #define __WRITE__(ad, obj) writeObjectSimple(0x50,ad,(obj))
  #define __READ__(ad, obj)  readObjectSimple(0x50,ad,(obj))
#endif
#define MEM_ADDR_MULTI 3458

//This is the structure we write and read from EEPROM
class MyData{
      long result;  
public:
      float  f1;
      float  f2;
      double f3;
      int    i1;
      long   i2;
      long   i3;
      bool  negative;

      long getResult(){
            return result;
      }
      long calcResult(){
            //makes a calculation with its data
            //and stores the result to a private variable
            long    i = (i2+i3)/i1;
            double  f = f3/(f1+f2);
            result = f/i;
            //round off
            result += result>0 ? 0.5 : -0.5;
            if (negative) result=-result;
            return result;
      }    
      MyData() : result(0){   }
};

void setup(){
      Serial.begin(9600);
      Wire.begin();

      MyData data[5];

      //data[0] - result should be 20
      data[0].i1 = 10;
      data[0].i2 = 150000;
      data[0].i3 = 350000; 
      data[0].f1 = 0.5;
      data[0].f2 = 2.5;
      data[0].f3 = 3.0E+6;
      data[0].negative = false;

      //data[1] - result should be -40
      data[1].i1 = data[0].i1;
      data[1].i2 = data[0].i2;
      data[1].i3 = data[0].i3;
      data[1].f1 = data[0].f1;
      data[1].f2 = data[0].f2;
      data[1].f3 = 2*data[0].f3;
      data[1].negative = true;

      //data[2] - result should be 80
      data[2].i1 = data[0].i1;
      data[2].i2 = data[0].i2;
      data[2].i3 = data[0].i3;
      data[2].f1 = data[0].f1;
      data[2].f2 = data[0].f2;
      data[2].f3 = 4*data[0].f3;
      data[2].negative = false;

      //data[3] - result should be -20
      data[3].i1 = data[0].i1;
      data[3].i2 = -data[0].i2;
      data[3].i3 = -data[0].i3;
      data[3].f1 = data[0].f1;
      data[3].f2 = data[0].f2;
      data[3].f3 = data[0].f3;
      data[3].negative = false;

      //data[4] - result should be -2
      data[4].i1 = data[0].i1;
      data[4].i2 = -data[0].i2;
      data[4].i3 = -data[0].i3;
      data[4].f1 = data[0].f1*10;
      data[4].f2 = data[0].f2*10;
      data[4].f3 = -data[0].f3;
      data[4].negative = true;

      Serial.println ("Data before writing to eeprom");
      for (int i=0; i <5; i++){
            data[i].calcResult();
            Serial.print("data-"); Serial.print(i); 
            Serial.print(": result = ");
            Serial.println(data[i].getResult());
      }
      Serial.print("Size of each object: "); 
      Serial.println(sizeof(MyData)) ;

      Serial.println ("Writing data to eeprom");
      int memaddr = MEM_ADDR_MULTI;
      long t0 = millis();
      for (int i=0; i <5; i++){
            memaddr += __WRITE__(memaddr, data[i]);
      }
      long t1 = millis();
      Serial.print ("...writing finished. Elapsed time: "); 
      Serial.println(t1-t0);
      delay(1000);

      Serial.println ("Reading data from eeprom");
      MyData read[5];
      t0 = millis();
      memaddr = MEM_ADDR_MULTI;
      for (int i=0; i <5; i++){
            memaddr += __READ__(memaddr, read[i]);
      }
      t1 = millis();
      Serial.print ("Time to read: "); Serial.println(t1-t0);
     
      for (int i=0; i <5; i++){
            Serial.print ("Read object: #"); 
            Serial.println(i);
            long storedResult = read[i].getResult();
            Serial.print ("  >stored result = ");     
            Serial.println(storedResult);
            long newResult = read[i].calcResult();
            Serial.print ("  >recalculated result = ");     
            Serial.println(newResult);
            if (storedResult== newResult && 
                  storedResult==data[i].getResult() )
                  Serial.println ("SUCCESS");
            else  Serial.println ("FAIL");
      }
}

void loop() {
}

Here are the time measurements, in milli-seconds, using a ConnDuino board:

Write time

Read time

Bytes

734

75

Blocks

82

15

So for this example, writing in blocks is approximately 900% faster while reading about 500% faster.

Useful links

Using EEPROM with Arduino and ConnDuino

Adding External I2C EEPROM to Arduino (24LC256)

MICROCHIP 24LC256 product page

Tags:

#eeprom #i2c #arduino #connduino #programming #code