MagPi Article

 

Abstract :

Logging systems are often used in industries in order to perform post analysis and monitoring. Studying logged data, which could be large, lead to a better understanding of  the  monitored system behavior. Nowadays the Raspberry Pi makes possible an affordable, accessible and low consumption system that can log, present and pull online data. Energy consumption is concerning everybody, for example the Raspberry Pi can be use to better understand the behavior of a house.

There I will present an independent logging system made of several technological choices (essentially from my personal  knowledge). The Raspberry Pi operating system is Raspibian and the programming language is Java.

Presentation of the base station requirements

As the aim of the system is to run 24/7, particular attention should be taken for the choice of the power supply. It should be efficient and reliable. If only few low power consumption sensors are connected, probably a mobile phone charger is enough. It is up to you to decide if a screen/keyboard/mouse or remote access is used to interact with the Raspberry Pi. I Choose to record all the data on the SD card. This allows further analysis and less write access to the flash memory compare to a rotating buffer. The consequence is one year of recording could represent hundreds of mega-bytes of data, and then a large SD card is mandatory if a lot of sensors are logged.

Raspibian is a really convenient operating system. It now includes the Oracle Java virtual machine which is very efficient. It also includes system driver for serial ports on GPIO pins, such as I2C, SPI and Serial (RS232). To install Raspibian on the SD card, follow the official Raspberry Pi tutorials. A java library called Pi4J will be installed to access these serial ports within Java, and will be describe later. Java programming language knowledge is also required to make sensors recorded.

Connecting the first sensor, BMP085:

Bosch provides with this little component a very precise and affordable solution to measure temperature and air pressure. It usually comes mounted on a board and can be connected directly to the Raspberry Pi I2C port following figure 1. A first test could be directly run on a console by typing sudo i2cdetect -y 0 and the response should show 77  on the address list. Adafruit made a very nice tutorial, you can also order the probe it self and get the datasheet : http://learn.adafruit.com/using-the-bmp085-with-raspberry-pi/overview

Figure 1 – Connecting BMP085 to Raspberry pi – Courtesy Adafruit.com

An example of a Java logging program

Java programming language can really be used on the Raspberry Pi since Oracle provided an efficient virtual machine. Execution time is fast and even if memory footprint is not optimum, programs run smoothly. Check if your Java version is Oracle Java7 by typing java -version and get as response java version “1.7.0_40”. The Java project including the needed libraries can be check out from this svn google code https://code.google.com/p/pilogger/source/checkout

But remember the source code must but customized according the sensors connected, and the online server used.

All the details of the Pi4J librairy is accessible from the official web page : http://pi4j.com/ . Pi4J give direct access to SPI for Java programs – and a lot of GPIO facilities.

Goals of the program

• At start-up, configure the communications to the probes and then the probes themselves.
• Load the previous recorded data from log files at start-up.
• Retrieve data from probes and record them to a log plain text file on the SD card.
• Make averaging with different time scale in order to present historical data in a convenient way.
• Upload via FTP these averaged data on a web server in order to makes them visible on Internet.
• Use a chart as graphical user interface to make local visualization easy.

By recording all the data we keep the possibility to change in further development the method used to average or analyze the log without losing information.

Structure

Figure 2 – Java program general structure

Probes connected to the Raspberry Pi can provide one or several values which may be independent. The chosen solution consists of an abstract class for probes and must be implemented for each sensors connected to the system. A probe implementation must provide then one or several DataChannels object, which can represent the data model. On a model-view-controller pattern, the ProbeManager class can be seen as the controller, customizing the view.

DataChannel class do the common work for each type of data, it load previous data from the log file at construction, log the new data, average data on diferent time scale.

The different implementations of AbstractProbe do the specific work for a type of sensor, like configure it, process or convert data, and then add them to the right DataChannel.

Add a sensor, the BMP085.

The dedicated class BMP085probe extends AbstractProbe. Then it is needed to provide the data channels. In this case, it is a first one for temperature, and a second one for air pressure. The two arguments of DataChannel constructor are the name for display and the name for the file.

private DataChannel pressureChannel = new DataChannel("Atmospheric Pressure", "Atmospheric_Pressure");
private DataChannel temperatureChannel = new DataChannel("Room Temperature", "Room_Temperature");

Then overide the abstract method which give the channels accessible to the ProbeManager  :

@Override
public DataChannel[] getChannels() {
   return new DataChannel[]{pressureChannel, temperatureChannel};
}

The class BMP085probe receive as constructor argument the I2CBus object from Pi4J library, as it can be several peripherals on the same I2C bus. With this bus, we can configure our I2C device. BMP085 has the 0x77 address :

public static final int BMP085_I2C_ADDR = 0x77;
private I2CDevice bmp085device;
public BMP085probe(I2CBus bus) throws IOException {
   bmp085device = bus.getDevice(BMP085_I2C_ADDR);
   readCalibrationData();
   DataReaderThread dataReaderThread = new DataReaderThread();
   dataReaderThread.start();
}

Within our reading thread, after sending the corresponding read command for temperature and pressure, the 2 bytes of raw temperature and the 3 bytes of raw pressure can be retrieved :

bmp085device.write(CONTROL, READTEMPCMD);            //Send read temperature command
sleep(50);                                           //wait the convertion time
rawTemperature = readU16(DATA_REG);                  //retrieve the 2 bytes

bmp085device.write(CONTROL, (byte) READPRESSURECMD); //Send read pressure command
sleep(50);                                           //wait the convertion time
msb = readU8(DATA_REG);                              //retrieve the 3 bytes
lsb = readU8(DATA_REG+1);
xlsb = readU8(DATA_REG+2);
rawPressure = ((msb << 16) + (lsb << 8) + xlsb) >> (8-OSS);    //make raw pressure integer

Then these two raw data can be converted in Pascal and Degrees following the BMP085 datasheet instructions, using the calibration values read before. Java not supports natively unsigned interger types, for convenience bit shifting can be replaced by division/multiplication.  http://www.adafruit.com/datasheets/BMP085_DataSheet_Rev.1.0_01July2008.pdf

double temperature = 0.0;
double pressure = 0.0;
double x1 = ((rawTemperature - cal_AC6) * cal_AC5) / 32768;
double x2 = (cal_MC *2048) / (x1 + cal_MD);
double b5 = x1 + x2;
temperature = ((b5 + 8) / 16) / 10.0;

double b6 = b5 - 4000;
x1 = (cal_B2 * (b6 * b6 / 4096)) / 2048;
x2 = cal_AC2 * b6 / 2048;
double x3 = x1 + x2;
double b3 = (((cal_AC1 * 4 + x3) * Math.pow(2, OSS) )+2) / 4;
x1 = cal_AC3 * b6 / 8192;
x2 = (cal_B1 * (b6 * b6 / 4096)) / 65536;
x3 = ((x1 + x2) + 2) / 4;
double b4 = cal_AC4 * (x3 + 32768) / 32768;
double b7 = (rawPressure - b3) * (50000 / Math.pow(2, OSS));
if (b7 < 0x80000000) pressure = (b7 * 2) / b4;
else pressure = (b7 / b4) * 2;
x1 = (pressure / 256) * (pressure / 256);
x1 = (x1 * 3038) / 65536;
x2 = (-7375 * pressure) / 65536;
pressure = pressure + (x1 + x2 + 3791) / 16;

This sensor can provide high rate of data. To enhance the precision, a thread take 5 data points, average them and add this new measurement every second to the dataChannel object :

pressureChannel.newData(pressureSum/OVER_SAMPLING);
temperatureChannel.newData(temperatureSum/OVER_SAMPLING);

To add the Bosch BMP085 on the final program, on the main class of the program, we get an instance of the Pi4J I2CBus, instantiate the BMP085probe, and add it to the ProbeManager:

private void initI2CandBMP085probe() {
   try {
      final I2CBus bus = I2CFactory.getInstance(I2CBus.BUS_0);
      bmp085Probe = new BMP085probe(bus);
      probeManager.addProbe(bmp085Probe);
   } catch (IOException e) {
      e.printStackTrace();
   }
}

Now the two data channel temperature and air pressure should be acquired, logged, and print on the graphical user interface as a chart.

— End of first article —

— beginning of second article —

In the previous article, a Java example of collecting and logging data from an I2C air pressure sensor was shown. A convenient way to see and analyse the data is to use the javaScript Google chart library on a web page. Depending on the needs, these charts may be accessed locally or stored on the raspberry pi and accessed remotely, or stored and accessed remotely.

Generate chart data files

In the class DataChannel, a scheduled task is started to periodically average the real time data collected. This initiate the generation of different time scale files in comma separeted value format. Files for real time, an hour, a day, a month and a year summary are stored on the raspberry pi SD card.

private static final int MS_TO_HOUR_POINT = 15000;
Timer t = new Timer();
averagingTask = new AveragingTask();
t.schedule(averagingTask, MS_TO_HOUR_POINT, MS_TO_HOUR_POINT);

Uploading files Online

In order to avoid direct access from internet to a personal home network to reach the raspberry pi, a solution among other consist of uploading the files from the raspberry pi to your favorite web server using FTP protocol. Apache provide a Java library to handle FTP protocol file uploading and can be download here : http://commons.apache.org/proper/commons-net/download_net.cgi

The class DataChannel use a static method of the class UploadFTP to put the files on an online web server :

private static final FTPClient ftp = new FTPClient();
private static final String HOSTNAME = "your_web_server.com";
public static synchronized void store(Path localFilePath) throws SocketException, IOException {

		ftp.connect(HOSTNAME);
		if (! FTPReply.isPositiveCompletion(  ftp.getReplyCode() )) {
			 ftp.disconnect();
			 return;
		}

		ftp.login("your_login", your_password");
		ftp.setFileType(FTP.BINARY_FILE_TYPE);
		ftp.enterLocalPassiveMode();

		InputStream input = new FileInputStream(localFilePath.toFile());
		ftp.storeFile("pilogger/" + localFilePath.getFileName(), input);
		ftp.disconnect();
}

An example of a web page

A HTML web page can be used in local or online to plot the data on a chart. Google provide a javaScript charting library, GoogleChart:
https://developers.google.com/chart/
To import .csv file the jquery.csv JavaScript library is also used:
https://code.google.com/p/jquery-csv/

Different HTML web page can be made to show the different time scales. Javascript language is then used to make the charts. To import csv data on the drawChartFromCSV function from the web page :

var arrayData = $.csv.toArrays(csvString, {onParseValue: $.csv.hooks.castToScalar});

and convert to Google Chart data Table :

var data = new google.visualization.DataTable();
data.addColumn('datetime', 'Date');
data.addColumn('number', 'Min');
data.addColumn('number', 'Average');
data.addColumn('number', 'Max');
data.addColumn('number', 'Max');
for (var i = 0; i < arrayData.length-1; i++) {
   var d = new Date();
   d.setTime(arrayData[i+1][0]);
   data.addRow([d, arrayData[i+1][2], arrayData[i+1][1],
      (arrayData[i+1][3] - arrayData[i+1][2]), arrayData[i+1][3] ]);
}

Some option are set, and then create the chart object :

var chart = new google.visualization.ComboChart(document.getElementById(elementId));
chart.draw(data, options);

Check the source code of the page here:
https://code.google.com/p/pilogger/wiki/OnlinePilogger
and an example here :
http://muth.inc.free.fr/pilogger/index.php?scale=Day&grouped=0

Outlook

You can follow the story and evolution of the system via the blog page. I added wireless temperature probes, and others are planed :
https://pierremuth.wordpress.com/rpiadventure/