Wednesday, June 28, 2017

LabPlot getting "live support"

As I told you guys in my previous post, I'm here now to show you some new, cool stuff about LabPlot. First let me introduce you the synopsis of my project's proposal: 

"Currently, the visualization and analysis of data is only possible on static data that was imported into or generated in one of LabPlot's data containers. The goal of the project is to add support for streaming data. At the moment LabPlot has no support for this kind of data processing even though it is very important for this feature to be available in a scientific data plotting software."

As you can see one of the main tasks here is to be able to handle changing, streaming data. Somebody may plug in their physical device and then watch as their datas are being plotted. For this to work nicely and user friendly, we collected a handful of options to control this kind of data source. The first thing you want to do is to create a new FileDataSource, you can do that this way:

FileDataSource creation

After this you get this nice looking dialog with the new options, controls. We decided to support more types of live data sources, as you can see in the picture below.

FileDataSources options
For "File or named pipe" source type we have the following options:

File or named pipe options

This kind of source consists of reading from a local ascii or binary file or a named pipe. A named pipe is a bit more than a traditional pipe because it has a name and unlike the traditional one it can last till the system is up. In Unix systems one has to explicitly create a named pipe using mkfifo or mknod. After creation one process can write to it and one can read from it by name. Also for "Local socket" live data source type we have the same options, but this consist of reading from a local socket using QLocalSocket. To work with QLocalSocket we just need the name of the server the socket is connected to.

The next type of live data source is the Network socket one. The following options we have right now to configure this kind of live data source:
Network socket options

A network socket is an endpoint for sending/receiving data in a computer network. In our situation we want to read from it. To access a socket we must have the socket host's address and port number. 

The last type of live data source for now is the Serial port one. In the following picture you can see the options for this kind of data source:

Serial port options
This kind of live data source consist of reading from a port to which a physical device is connected. We just have to select the port our device is connected to, set the baud rate and voilá, it should work nicely soon. We use QSerialPort to access, work with serial ports. Using QSerialPortInfo class we can have informations easily about the available ports on our system. Also the supported baud rates can be acquired using QSerialPortInfo.

These options are necessary to set up the wanted kind of live data source. Beside this we must have some control above this kind of data source, right now we have the following options to control the update a FileDataSource:

Update options
Here you can see that we have two types of updates. The "Time interval" one will update the curves on a specified time interval (every second, every five seconds or so) which is a value in milliseconds. The "New data" update type will update the curves only when there's new data available. In the "Update frequency" we set the interval between the updates when using "Time interval" update type. The "Keep last N values" will control how much of the data we want to update. For this kind of data source one usually want to observe how the data is changing, to take a snapshot of their data. 

Another thing we might want is to have control over this kind of data source even after creation. For this we'll have a dock widget, it's not yet used, but currently it looks like this in Creator:

FileDataSource dock widget

With these options we will be able to modify the number of values we want to keep on every update, the time between updates. If the user uses a long interval between the updates then a functionality like the "Update now" button will provide it's very useful because there might be moments we didn't think we'd need to check the actual state of our data. Also the pausing and stopping the updates/data reading helps in the control of this kind of data source.

I have created a demo video which shows the creation of a FileDataSource, setting the X and Y axis data of some xy curves. Then a python script writes Brownians motion data to a file on which the FileDataSource was created. We can see there the curves being updated on new data:

An issue you can see here is that I have to set for every curve separately the X and Y value columns. We will use the PlotDataDialog which is used for plotting spreadsheet columns to solve this issue. PlotDataDialog is a very user friendly dialog which can be accessed from the spreadsheet's context menu. In this dialog we can select the X data and Y data for any curves we might want to plot from spreadsheet column data. This is the PlotDataDialog:


This is for now, hope you guys enjoyed it! The next step is to implement all the logic behind these options and make them do their job. A greater step after implementing the options will be to optimize our code on larger amount of data. Thanks to my mentors, Alexander and Stefan for their great help. Every time I am unsure on something they take their time to talk about it so together we can make LabPlot even cooler than it is right now :).

See you guys soon again! Have fun on your GSoC project!

Saturday, June 17, 2017

A harder beginning brings cool changes

Hi folks, it's been a while since my last post here. I had a hard time to start on GSoC but now that I have successfully graduated from the university (Computer Science BSc) I have all my time for my project. While learning for the exams and preparing for my final exam I've been talking to my mentors often to share our ideas about the project. Thanks to the fact that I have already worked previously (and continously after GSoC) on LabPlot I can implement the needed features much easier and quicker now than last year :) Now that I have my own branch and a few new classes added I can continue the adventure with LabPlot! See you soon! :)

Thursday, August 18, 2016

LabPlot is comfortable with FITS

So we are here, GSoC 2016 is officially over, it's time for the final evaluations, so this will be a closing post for this GSoC. I've learned many things while I was working on my project and I'm really thankful to my mentors, Stefan and Alexander (not officially my mentor, but we talked many times and he helped me many times too). I'll be happy to help in integration of this project in the next release of LabPlot :)
Since my last post I've fixed some minor bugs, and implemented some new features too.

Let's say we want to add a new keyword to one of our FITS extension:

FITS header editor's new keyword adding dialog
 And then we can add units to our keyword, which will appear in squared bracelets in the beginning of the comment:

Keyword unit adding dialog
I added some predefined units, such as meters, kilograms, seconds, parsec, solar mass, astronomical unit, kelvin, light year.
The newly added keyword unit

If a keyword has some kind of unit defined already, then the dialog will be shown with the unit of the keyword:

Saving keyword modifications

We have modified this value, the original is "lw", as you can see in the treeview.

The saved extension with the new keywords value
As you can see, after saving the modifications, and reopening the file, the name of the extension is modified.

Import a subset of data 

Let's say we have this image, but we'd like to import just a portion of it.

We just have to set the start/end row/columns in the import dialog,

..and voilá

File info dialog improved for FITS files

Import FITS tables to matrices

If the selected table extension has numeric columns (at least one) then we can import those in matrices. (btw notice the nice icons in the treeview, thanks to Alexander for the idea)

As you can see, the second and third column contains only values, so after import our matrix:

I think this was it. I hope you enjoyed it too. I'm really happy that I was part of GSoC '16, it's one of my greatest experiences. I look forward on reading other GSoC students blogs and I wish good luck to everybody on the final evaluation! :)

Friday, July 8, 2016

Further exports, delete extensions, FITS for LabPlot

I have implemented some features since my last post, soon I'll be polishing the features, searching for bugs. So, let's have a look at them:

Delete FITS extensions from files:

Context menu for deleting extensions
You just right-click on the extensions, and you can delete them, and after you save the file, reopen it, you can check the file again:

(I deleted the other extensions this time) And you can see that some extensions are removed from the file.

I've improved the export dialog for FITS files:

Now you can select whether you want to export your container to an image extension, or a table. If the spreadsheet contains non-numeric columns, then exporting to a FITS image it's not possible:

You can check the "Columns comments as units" checkbox if you'd like to export the column's comments as FITS table units, check comment:

And here you can see it as the unit of the column in the FITS table:

Other important it's that when you import tables to spreadsheets, then columnmodes are set properly, for example text/numeric columns:

To prove that:

In short, "A1" stands for a column which contains strings, with length of 1.
That's for now, I'd be happy if you could test the features/give feedback. See you soon.:)

Friday, June 17, 2016

Import, export FITS

There are some new features that I've implemented since my last posts, let's see them!
FITS tables can be imported to spreadsheets now. Let's see how it works:

FITS table imported to a spreadsheet
As before, I compare my results with the results of FV (FITS Verify). I set the column comments to the units defined for the columns in the FITS if they're available.

You can now export your matrices to FITS images! If the file exists, it just appends a new image extension, if not, it creates a new FITS file, check out the header of an exported matrix, and the tree of the file:

Header keywords of the exported matrix

You can export your spreadsheets to FITS tables too!

Currently the export dialog for FITS files

And then you can check out your file's header:

Header of the exported FITS file
 Great, isn't is?! If the file didn't exist yet, it creates one with an empty primary array, and appends an ASCII table. It still needs more improvements of course.

And then let's import back our exported spreadsheet:

The exported spreadsheet imported

There are still plenty of things to fix, and others to implement of course :)

Friday, June 10, 2016

Let's modify your FITS files

Another great step for the project, now you can add new keywords, or remove them from the extensions!
There are some restrictions:
  • you're not allowed to remove mandatory keywords (still needs some polishing)
  • you're not allowed to add mandatory keywords, since those must be available when you create the extension
  • no duplicates
Take a look at the newly added keywords, the DATE keyword proves, that it was written to the file, since cfitsio has a special method for adding DATE keywords, which sets the value and comment fields:

The new keywords
There are still many things to do, to refine:)

Thursday, June 2, 2016

Strong kickoff

Ever since I've connected with my mentor (Stefan) I've worked on LabPlot. Even before the community bonding period I have implemented LaTeX exporting support for spreadsheets and matrices. You can export now your datas in LaTeX tables, this is the export's dialog:

LaTeX export options

And then you can create some nice looking pdf's, like:

Exported LaTeX file converted to a PDF

In the community bonding period I did some modifications on it and just a few days after my project got accepted I started working on my project as my mentor suggested, because I was already familiar with the code and I had everything set up. My project's main goal is to enable importing of FITS images and tables in LabPlot.
In extra I had some ideas:
  • have a widget for editing (or just listing) the content of a FITS primary header or extension's  header unit. 
  • hopefully, if there will be time, export to FITS
I've done some of them already, but of course everything is changing yet.
The header edit dialog can be accessed through the Tool -> Edit FITS file header menu. This is how it looks like now:

FITS header preview

Currently you can open files, and for every file a nice tree is created, showing separately the Image and Table (ASCII table, binary table) extensions. In the table you can see the keywords of the currently selected header. I know the layout needs some improvements, soon that will be fixed too. It will be possible here to modify the entries, add new ones, or remove some. I've added some standard keywords from the FITS standard to be accessible from the "Key" KTextEdit. Adding a new keyword:

Dialog for adding a new keyword

After clicking on "Add keyword" you can see the newly added keyword:

Keyword added to the header's preview table

Currently the modifications appear only in the dialog, soon implementing the saving of the changes to the files.
Other thing I have achieved is that it's possible to import already FITS images. As you can see I use the same tree as in the header edit dialog. This is how the import dialog for FITS files looks now:

FITS import dialog - Image preview
The preview works nicely for FITS tables too:

FITS import dialog - Table preview
And then you can import some cool images from FITS files, like:

Imported FITS image
I compare my results with the results of FITS Liberator 3, as you can see, they are pretty the same.
The importing of FITS tables will work soon nicely too, but I think these are already some big steps for the project.

Have fun other GSoC participants!