Tobias recently blogged about Nepomuk, and from the comments it seems that people are a bit in the dark about what Akonadi, Nepomuk and Strigi actually do and how they interact with each other. So if you want to understand those technologies, read on! This blog post is an attempt to clear things up a bit.
Soprano
Let me start with Soprano. Soprano is a Qt library for accessing semantic storage (RDF). In many ways, Soprano can be compared to the QtSql module, the key difference is that QtSql accesses relational data with SQL as the query language, whereas Soprano accesses semantic data with SPARQL as the query language.
Semantic Data
But what is semantic data, and what is the difference to relational data? You probably all know relational databases, which use tables to store data. Semantic databases, on the other hand, use statements, also sometime referred to as sentences, to store data. Statements consist, just like real-world sentences, of a subject (noun), a predicate (verb) and an object. By storing many sentences in a database, one can create a big network of data. The best way to make this clear is probably by examples, so here are some:
.
- “image.jpg” “has the width” “800 pixels”
- “image.jpg” “is tagged with” “example-tag”
- “image.jpg” “was photographed by” “Max Mustermann”
- “example-tag” “has the title” “Holidays”
- “example-tag” “has the icon” “beach.svg”
- “Max Mustermann” “lives in” “London”
- “Max Mustermann” “was born on” “01.01.1970”
As you can see in those examples, it is possible to link together totally different topics, such as information about a file, a tag and a person. Those examples are of course a bit fabricated, real data would look different. However, you can see the basic sentence structure consisting of subjects, predicates and objects. One can add an arbitrary number of statements about things to a semantic database. The powerful aspect about semantic data is the way it is linked together. With the above statements, you could for example search for all images tagged with “example-tag”. You could also do much more interesting searches, like searching for all pictures taken by people living in London, or searching for all files that were sent as e-mail attachments from your boss.
Read the RDF Primer from the W3C for an in-depth introduction about RDF and semantic data (RDF is one way to describe semantic data, but not the only way).
Backends
Like QtSql, which supports different backends such as SQLite or MySQL, Soprano also supports different backends. Currently, there are three backends for Soprano: Redland, Sesame2 and Virtuoso. Redland is C++ based and orders of magnitudes too slow for what we need in Akonadi, so basically Redland should never be used at all. Sesame2 is Java based and performs well. This currently seems to be the only usable backend for Soprano. Read Tobias’ blog entry for the details about this. The third backend is Virtuoso, which combines the strength of the other two backends: It is C++ based and performs well. To my knowledge, this backend is currently in development and therefore not usable, but it will certainly be an interesting choice in the future.
Nepomuk
Ok, now you should have a basic understanding about what semantic data is and what Soprano is. Read on to find out what Nepomuk is!
Nepomuk is the KDE library for accessing semantic data. It uses Soprano for storage access. Nepomuk provides a KDE API for many high-level functions such as tagging and annotating. An important point is that Nepomuk also provides a set of standard ontologies, and convenience classes to use them.
Let me explain what an ontology is. Although you can store arbitrary statements in a semantic database, that rarely is useful. Consider the case that you store the sentence “Laura lives in Leeds” and the sentence “Ralf resides in Leeds”. Notice that the predicate, i.e. the verb, is different in those sentences, once it is “resides in”, and once it is “lives in”. Now, if you attempt to do a semantic search for all people living in Leeds, you will not find Ralf, since the statement about him uses a different predicate. Therefore it is a good idea to have a set of standard predicates and other terms, to have a clearly defined vocabulary about things. This is what ontologies are. Nepomuk comes with a set of standard ontologies which define vocabulary for talking about annotations, files, contacts, mails, calendars, music and more. These ontologies are now also a freedesktop standard, GNOME’s Tracker uses them as well.
Now, Nepomuk would be useless without any data. There are basically two ways of getting data into Nepomuk: One way is by manual user action, for example when a user tags a file in Dolphin. The other way is automatic indexing, which is done by both Strigi and Akonadi. Read the next sections for details.
Strigi
Strigi is the file indexer for KDE. It looks at every file on your hard disk, extracts semantic data out of the file, and then feeds the data into Nepomuk. When saying that Nepomuk uses a lot of CPU or IO, that is usually because Strigi is indexing files in the background. There are however many settings to improve this, for example indexing is disabled while on battery, and the IO niceness is set to a low level. Also, Strigi indexing can be disabled completely, without disabling any other parts of Nepomuk, because the file indexing is just one way to get data into Nepomuk.
What currently is badly missing, in my opinion, is a good GUI client to actually search for all the data that has been indexed. It seems that there was some very nice progress during this year’s summer of code with that, so I am sure the situation will get a lot better in the future
Akonadi
Now, on to the last technology of this blog post! Akonadi is a framework to access PIM data like mails, contacts and calendar events. Think of it as a cache or a proxy to your PIM data: The real data is still stored in local Maildir folders, local vCard files, IMAP servers or in your Google address book. Akonadi provides an easy API to access that PIM data in an uniform way. Additionally, it can act as a cache, for things like disconnected IMAP or for offline access to your Google address book. Another advantage is that the PIM data can easily be shared between applications. Now not only KMail can access your mail, but also LionMail or Mailody. Additionally, there is no need to have KMail running to access your calendars and contacts on your Kolab server. Akonadi furthermore replaces the brittle system of index files in KMail, and the new Akonadi IMAP resource is already much faster than KMail’s old IMAP code.
So as you can see, Akonadi will bring many advantages to the end user, once the applications are ported to use Akonadi. For KDE 4.4, only the new KAddressbook and KPilot will use Akonadi natively. Ports of KOrganizer, Akregator and KMail are in progress and (hopefully) expected to be released with KDE 4.5.
Akonadi of course needs to store information about the PIM items and folders it knows about somewhere. For this, we use a classical SQL database. For now, we support only MySQL, but there is work done on PostgreSQL and SQLite support. Those two database backends are both work in progress, help there would be very welcome.
Now, how is Akonadi related to Nepomuk? Applications which use Akonadi require a fast search and good support for virtual folders. Now, we didn’t want to code our own search support into Akonadi. It is quite a lot of work and difficult to get right. The virtual folders in KMail 1 are for example too slow to be useful for larger volumes of mail. What we did for searching instead was to use a technology that is actually good at finding stuff: Nepomuk.
We use Akonadi agents to feed information about contacts, events and mails into Nepomuk. So just like Strigi, those Akonadi agents will put data into Nepomuk. We use the standard ontologies, like the NMO (Nepomuk Mail Ontology) to store the mails. This data is then used for searches and virtual folders. By using Nepomuk, we hope to overcome many of the KMail 1 shortcomings, like the slow virtual folders mentioned earlier or the inability to search in base64-encoded attachments. It already works quite good, for example we have a working tag resource to show all your mails tagged with specific tags, and searches with SPARQL are also working (although there is no GUI for it, yet, for now you need to use the development tool akonadiconsole to see them).
That’s it, folks. I hope I made some things clearer to you. If anything is unclear, please ask in the comments section.
My next blog post will have screenshots again, I promise 🙂
Thomas McGuire's Blog 
Well explained, but now it’s all about starting to incorporate these technologies.
I started a forum thread about this (available here http://forum.kde.org/viewtopic.php?f=15&t=82595 ) hoping to start tackling real-life scenarios to make it so that users can start benefiting from this as soon as possible.
Personally I believe what’s important is to create a prioritised roadmap from these case studies. What’s your view on this?
Oh, and just to rant about a personal problem, whenever I enable strigi, it indexes no matter if I tell it to suspend or not. This jumps my CPU up to 100% and I have to kill it (the process actually listed as nepomukstorage).
looks interesting and exciting to develop
as a user, for the moment, it just makes my desktop crawl even more 🙂
If I read this correcty, there’s no way to synchronize my addressbook/calendar entries with another system (say, OS X for example) until KDE 4.5 i.E. approximately one year from now? Even very simple syncs (like adding a remote ics based calendar (or using webdav) is out of the question?
That sucks 😦 There used to be quite an easy way to add new calendars to Kontact, but if I understand you correctly, that will come again in 4.5 😦
I do understand the usefulness of Akonadi, but right unfortunately it’s next to useless – until it’s got some *real* functionality, there should be something (even if that means some ugly, temporary hacks) to bring back the old functions of kontact.
Mark, remote ics based calendar is available. Look under resources and add a “remote file” resource. I personally use this to ensure that my calendar is up-to-date, backed up on my server, and available everywhere, including my Windows Mobile phone with a nifty import tool.
openSUSE did have a virtuoso backend. I think it even worked but not any more. Perhaps it is still available in unstable.
Thanks very much for your post, things are much clearer now (especially with Strigi) and now I understand the rough concept of semantic data.
@Dion: Yes, roadmaps for Nepomuk functionality that implement real-life scenarios are important. As a member of the PIM team, I can only speak about our plans for Nepomuk. We will focus on mail searching, virtual folders and the tag resource. Those are all real world scenarios and actually requirements for a good mail client.
@Mark: Using a remote ics file as a calendar is possible with the Akonadi ical resource, and in fact Akonadi is designed to make it easy to write new resources that support synchronization and offline access. One example is the Google Calendar resource for Akonadi.
You say that “there used to be quite an easy way to add new calendars to Kontact”. That is still the case! First of all, KOrganizer is not ported to Akonadi yet, so nothing has changed, you can add calendars like it was always the case.
Furthermore, the Akonadi-based KOrganizer in KDE 4.5 of course allows to add new calendars in the same easy way. Each calendar will be an Akonadi resource then.
Even better: Although KOrganizer is not yet ported to Akonadi, it can you Akonadi-based resources like the Google Calendar resource through so-called bridges. And the other way around as well: Should there be any old-style calendar resource in KDE 4.5 that is not ported to Akonadi, it can also be used in an Akonadi-based KOrganizer through bridges.
So there is really is no loss of functions related to calendars with Akonadi. On the contrary: The current, non-Akonadi Kontact does not offer any synchronization support at all, while for Akonadi, we have SyncML support.
@all Aah, thanks – I’ve found out how to this, my mistake, sorry about that
I know that might be a little bit off-topic (on the other hand, this is how akonadi is supposed to work), I’ll explain what I want to do:
A couple of days before I was given an old iBook. I’d like to use it as a netbook and synchronize Apple’s iCal/Adressbook with my main machine running KDE 4.3.1, considering that KPilot (I’ve got a Palm OS device, too) isn’t really working – Apple’s iSync should work nicely. Is there any way (right now) to accomplish this? Let’s say I’m travelling, add some new appointments on the iBook and synchronize them later (or whenever I’ve got internet connectivity) to Kontact?
I wouldn’t mind setting up something on a remote server to accomplish this, right now there seems to be kolab plugin (in Akkonadi), but apparently there’s no way to configure it
Unfortunately, I don’t know anything on the topic of synchronization, so I can’t answer you here.
About Kolab:
The Akonadi Kolab resource is not configurable because it works automatically, given that you have configured an IMAP resource, since Kolab stores contacts and events as attachments to mails on an IMAP server. The Akonadi Kolab Resource is work in progress currently, I’m not sure how well it works out of the box.
Given that an IMAP resource without KMail support is not very useful before KDE 4.5, when KMail uses it, I wouldn’t recommend using it for now. Right now, using the traditional Kolab setup with KMail configured with an dIMAP account and addressbook/calendar KResources that talk to KMail over DBus works well, I use that as well.
Thomas, any chance we can lift and reformat some of this content for the dot? It seems that a lot of people have a hard time understanding the whole concept of the semantic desktop (including the developers of other parts of KDE), and this was very well explained.
Cheers
Hey Troy! Feel free to take any of this blog post and reformat it, that is perfectly fine. A small note linking to this blog entry would be nice. So if you and the promo team want to make a dot article out of it, I would be glad, you are certainly better writers than me. Ping me if you need any help with that.
Well put!
I still can’t help but feel all of this is horribly overengineered though. Do you really need a Java server to maintain a simple triple store? Shouldn’t that be a piece of cake to hack together over sqlite/whatever?
Apparently it is not a piece of cake, otherwise redland would be fast 🙂
However, I think Tracker uses SQLite internally, so apparently what you say is possible, and IMHO that would be a good idea.
According to the Redland site(http://librdf.org/) The Redland RDF API is in C not C++. Assuming that to be relevant and true, I am interested in working on the Redland backend to speed it up. I’ll contact the Redland people separately to get inserted into their way of doing things.
Helping redland would be very welcome, thanks!
I don’t think it is the language that matters, it doesn’t really matter if it is C, C++ or Java. It is the algorithms used that matter, and it seems there is a lot of room for improvement in redland in that area.
So, for users like me, that use GMail and never search for files in the desktop, what advantages does Nepomuk bring?
Are there plans to use Nepomuk for other things besides messaging and searches?
If you do not use desktop search, do not use Kontact and do not do any rating, tagging or annotating, then indeed Nepomuk will be of little use for you.
There might be plans to use Nepomuk in other areas, but I don’t know that.
For PIM, we first plan to use Nepomuk for searches and for tagging. Using it for annotating mail is something I have thought as well, since that is a popular feature request.
I agree with David. This is just more software bloat crammed into KDE base. Akonadi steals resources (disk and RAM) even if you do use it.
It’s really disappointing to see how it has infected KDE 4.6 like the registry infected Windows.
It needs to be completely optional.
What sane person wants to run mysql or postgres on their laptop to hold PIM data?
I recognize there are some users who will see this as something useful, but it should be optional and not forced upon those KDE users who users of kapps.
Whoa! I was searching for bits of information what Akonadi actually is after my KMail updated to KMail2.
From what I read it actually should speed up my system, but it did something completely different. Not only KMail2 is so unstable so it’s barely usable, but it’s now slow as hell in most operations (even showing contents of a folders can take ike 5 seconds). IO load generated by Akonadi processes makes my system completely unresponsive. Where were the QAs before it was released?
I’ve posted about 7 or 8 bugs to KDE. Will see what will happen about this.
Many thanks for this article! I need it really bad…
I really hope to see this tecnologies working. It will be a fantastic improvement.
For now I’ll be content to have akonadi running in order to access my kaddressbook entries…
This days trunk is not usable for daily work… I’ll have to install a stable KDE release 🙂
I always use KDEPIM from trunk for my daily work, so far it always worked for me, including the new Akonadi-based KAddressbook.
But I would recommend normal users to not do that 🙂
I use trunk everyday too (and I’m happy with it)…
The problem I’ve had is that on this days mysql-global.conf file is not automatically created on ~/.config/akonadi/ and in the userbase.kde.org akonadi page it is written to copy the default configuration file to ~/.config/akonadi/mysql-local.conf but akonadi server instead looks for ~/.config/akonadi/mysql-global.conf.
Now kaddressbook is working again. I’ve had it unworking only for 1 day because I’ve had cleaned my local config files, in the past all files were automatically created. I’ll inform devs about this.
Bye 🙂
Hi, really nice article, can I as well take it, translate to Spanish and publish in KDE-Hispano? (web page for KDE spanish speakers users). I think the article really worth it.
Sure, feel free to translate this into Spanish, that would be nice.
You might consider submitting an article to Linux Format they need some KDE input there. I just read and re-read their latest issue #124 on vacation and they are seriously Gnome heads there and not to mention the U* word.
Nice article. I see some of my forum members have been here already! Good work and nice Pic!
I’m in the Lake District.
I’m not a writer myself, only a programmer who started to blog a bit, I don’t think I’m the right person to submit articles to magazines. Maybe the KDE promo teams wants to take this up?
Lake District is in England, right? The picture in this blog is from the west coast in Ireland, where I’ve often been on holidays.
Hi there,
here is a possible (a little hacky) solution to synchronize Kontact with scheduleworld/funambol.
I’m using it and it’s okay:
http://wiki.scheduleworld.com/wiki/Linux_KDE_4_Configuration
One thing I want to mention about the nepomuk/strigi thing:
If I try sopranocmd examples or any nepomuk internal apps like nepomukshell it is really lightning fast. I currently have 1GB nepomuk size and I think about 162352 Files in nepomuk storage.
So to me the problem is definitly not the speed of soprano or nepomuk itself. My cpu usage raises like the most others told by nepomukstorage. I think this process stores actually information and mostly raises up if I change (copy,move,create,delete) files.
One possible Solution is to exclude more directories from nepomuk.
But I think the basic settings of nepomuk are kinda bad. With just “files” nepomuk/strigi is in a very bad situation.
A file can be anything. If a application internal uses nepomuk it knows much more about the ontology which is related to the current task/object you are working with.
If kget downloads a file it knows that it was “downloaded from site ‘bla'”. And it also knows the exact date you downloaded it. Nepomuk know knows you downloaded the file from a site/person on that date.
If strigi indexes the whole drive it does not know much about it. Pimo is possibly a good approach to solve that but I don’t believe fully in this because it’s like trieing to fix a bad audiofile. Once you have no exact idea about a file you want get data exact.
So to my opinion indexing the whole drive with strigi is kinda contra approach to nepomuk and its big advantage against other desktop search systems.
Akonadi knows much more about the task/object you are working with. It knows that person X is sending or doing or whatever.
If I look whith nepomukshell (can be found in playground) the data my computer collects is not very intelligent. It knows a view Persons like contacs or artists from my music collection.
So this data isn’t very valuable for me.
It is from a semantic view better than any other indexed search I know but far from beeing that what nepomuk should be.
So for me the most important point is:
Nepomuk is no desktop search system. Bringing strigi into it and index the whole drive mixes two different things. So if I really search for a keyword and only want to know what files contain that a simple search over the clucene index with a fast c++ programm would be the best.
If I want to know who sended this file to me or what foo is or which things are tagged with “Girlfriend” the approach have to be different.
So to me at the moment strigi is pulling a lot of “garbage” (sorry) into nepomuk with no big gain for me.
I really believe in nepomuk and it will change the desktop experience dramaticly but I hate it when it blocks dolphin or my whole computer.
[…] pointers on how to enable the java-based sesame2 Soprano backend. Thomas McGuire gives a very good introduction into what Soprano, Nepomuk, Strigi, and Akonadi are and how they relate. This was a much needed post. Thank you for that, Thomas! And finally mat69 […]
[…] McGuire, un desarrollador de KDE, ha publicado en su blog personal un completo artículo explicando cada uno de ellos, así que vamos a aprovechar para hacer nosotros también un […]
Nice article. Thanks!
What I wonder is, will it be possible to load and use “foreign” ontologies (dbpedia, geonames etc.) into nepomuk/soprano? That would be _really_ cool since then, to stay with your example, London would not only be a meaningless string for my desktop/computer, but it would “know” that it is a city, the capital of England (which is in Europe), with a certain longitude and latitude, a certain population, a certain postcode etc. Or if strigi finds an audio file that has the artist and title track set, it could learn the unique id from the musicbrainz database and, if asked, could tell me about all albums this artist has ever created. Or, if the file has no tags, it could send the fingerprint to musicbrainz, receive the track and artist IDs and tag the file for me…
Loading foreign ontologies in Nepomuk is perfectly possible, there is even a tutorial at http://techbase.kde.org/Development/Tutorials/Metadata/Nepomuk/OntologyLoaderService
Yes, I’ve seen this tutorial in the meanwhile, but wasn’t able to load any ontologies into nepomuk by using the method described there. At least nothing happened, it just returned without any notification, no matter if I gave a real filename or a made up one.
I only could load some statements (which are NOT the same as ontologies, if I understand correctly) into nepomuk/soprano via sopranocmd, but I got an error when I tried to use onto2vocabulary to load/convert a ontology. I guess I need to read a lot more to even begin to understand what’s going on…
Oh, what I also did was to use Soprano as a SPARQL client and query the dbpedia endpoint. Quite interesting! For me this is where the real attraction of semantic technologies lies as I do not care that much about filesizes, imagewidths etc., though I understand the usefullness this has.
[…] 4 es la inclusión de Nepomuk, el motor semántico que se empieza a aprovechar en aplicaciones como Akonadi/Strigi y que permite añadir ciertas etiquetas y metadatos a cada uno de los archivos que manejamos para […]
[…] There is a good blog explained akonadi, nepomuk and strigi. Check it https://thomasmcguire.wordpress.com/2009/10/03/akonadi-nepomuk-and-strigi-explained/ […]
An incredible, MS-style embarrasment: dolphin’s ability to display an image’s width and height information DEPENDS on the file having been indexed by strigi/nepomuk, BUT strigi/nepomuk index files when they see fit, NOT on demand.
How can I order strigi/nepomuk to index a particular file NOW, and gimme gimme gimme the information I want RIGHT NOW, without waiting?
Alternative question: how can I boost the priority of strigi/nepomuk indexing? Currently it is SO incredibly careful not to interfere with desktop performance that it stops working when I approach the computer, I don’t even have to touch the mouse. How do I adjust the tradeoff? I want my file indexers to grind like crazy and finish ASAP.
[…] Re: KDE4.5: akonadi, nepomuk, strigi. Linux desktop options. Thoughtful input. Thank you. I think I'll be looking at LXDE. Some distros (i.e., Knoppix) went that route, because of the resource requirements of KDE. It's interesting that Lucid Ubuntu/Kubuntu, which is a LTS release (that's 3 years!), is going with KDE 4.4. As far as mail, I'll be looking at software that supports the maildir format. I've never been a big PIM user, but I definitely need an efficient/stable email client. Doesn't need a lot of bells and whistles. I was planning on turning off my Windows email and converting to Kmail until I started looking at KMail in KDE 4.5. I sort of like KDE 4.3.5. It took me awhile to get organized from KDE 3.5, going through options, trying things, enabling/disabling options. There are some issues, but I felt there was a nice marriage between what I liked and made me productive in KDE 3.5 and what's new in KDE 4. I'm less excited with KDE 4.5. It's change in configuration files, use of MySQL (Just to read my email. Icks!). The whole semantic desktop, good for those users who find it increases their productivity. I don't have a need for it, but it appears I'll still have to carry the baggage. There's a sense in the air that this goes against the Unix philosophy (ie., small, compartmentalize, user decides). Most changes I don't fret, especially in the Unix world, because if users don't like it, there is always a techie with a timely tweak. KDE 4.5 will not be tweakable in that fashion because it's the foundation/framework. Helpful links:Akonadi, Nepomuk and Strigi explained […]
[…] https://thomasmcguire.wordpress.com/2…igi-explained/ seems to be an indexer. I have it running, but have never really looked into what it does. […]
[…] 1 Akonadi, Nepomuk and Strigi explained […]
[…] to read these two articles first, one being two years old and the other a bit more recent… https://thomasmcguire.wordpress.com/2009/10/03/akonadi-nepomuk-and-strigi-explained/ and […]
[…] akonadi in KDE was and is a real pita, as the boundaries within KDE applications such as konversation or kontact are way too much, and decrease performance entirely. disabling akonadi (as well as nepomouk and strigii by not wanting an indexed desktop search – see this explaination). […]
Well explained, very much appreciable..
what is the difference of the information stored in nepomuk and mysql (or sqlite)?
[…] Is there anything that I should be concern? Has anyone else tried (removing) them? I read this blog https://thomasmcguire.wordpress.com/2…igi-explained/ and other topics on this forum and google it and it seems that there is not problem removing them, […]
Still an interesting article in 2012, but I wonder if the Nepomuk idea tries to accomplish too much? Perhaps if parts of the code where useful to applications outside of the KDE sphere, (Book catalogue software, MP3, Torrents) then development and uptake would be greater?
[…] https://thomasmcguire.wordpress.com/2…igi-explained/ https://kdenepomukmanual.wordpress.c…epomuk-manual/ (given that .kde4 is in /home/username/… for each user, if you format the volume where that is, you start again from fresh; if you re-use the existing partition to preserve the data in /home, this usually does bite…it didn't used to bite (or, at least not hard) back in the kde3 days, from what I remember….progress…. […]
About 3 1/2 years later Akonadi, Nepomuk, and Strigi still suck and are 100% worthless CPU wasters.
It is past time to pull this crap out of KDE. These are the 3 biggest reasons why KDE sucks today.
A pox of the houses of all the “developers” who thought this was a good idea.