Discussion:
ClientInterface (was Re: Fwd: [PATCH] kernel / UI separation - DCOP example)
(too old to reply)
Ingo Klöcker
2003-07-13 23:00:57 UTC
Permalink
Regarding the comment.
+// TODO - this needs effort to separate kernel and GUI
This issue should be resolved before the patch is committed to cvs.
Is it possible to use a SYNC dcop method to show a Yes/No messagebox
and return the result? Eventually we might be able to architect
KMail so that only ASYNC dcop methods are used, but I think it makes
sense to have a transitional period where SYNC methods are used.
I've already thought about this issue, and I agree with you. It can
get solved using a sync dcop call right now, but I also thought about
a way to avoid such sync calls later. But this requires deeper changes
in the existing codebase, I would like to concentrate on the dcop
issue for now. I'll commit as soon as I have the TODOs eliminated.
I just returned from LinuxTag 2003 in Karlsruhe. During LinuxTag I
talked with Cornelius, Tobias (tokoe) and Marc about the UI separation.
No decisions where made but we definitely see the need to talk
about this some more because this is a very serious decision which will
have a very high impact on the future of KMail (So why Cornelius and
Tobias? Because they would like to use some of KMail's functionality in
KO resp. in KAB). Although I was at first quite enthusiastic about the
DCOP approach, the others don't think that using DCOP is a good idea.
So we really need to talk about this in a larger group. I would prefer
to postpone any decision until Nove Hrady where we will have the unique
opportunity to talk about this from face to face.

I'm very sorry, Andreas, for all the work that you put into this, but
please understand that we mustn't rush such a radical change into KMail
without giving it a lot of thought.

For KDE 3.2 we primarily have to concentrate on making Kontact a
full-featured and rock-stable Kolab client because that's what the
corporate users need. LinuxTag showed that a lot of people are waiting
for Kontact because without a working Outlook/Exchange replacement it's
impossible for them to even think about migrating from Windows to
Linux.

Regards,
Ingo
Don Sanders
2003-07-15 07:43:45 UTC
Permalink
If you (Ingo) and Marc object to Andreas' patch, Zack and I support it
and Carsten has no comment then effectively that's a deadlock
according to the KMail decision making process.

In which case I'm ok with leaving the decision as to whether to commit
the patch or not up to Andreas himself. Andreas has
been contributing to KMail for a long time and I trust him to make
changes without breaking things. I also doubt he has any intention to
rush things.

Whether or not Andreas get to make the decision I think it's a good
idea for people to voice any technical objections on the list. So far
I'm not aware of any outstanding technical objections. I believe
Andreas isn't scheduled to attend Nove Hrady so since he's the one
doing the coding it does make sense to voice objections on the list,
where he can read them.

To recap the basic idea of the patch is to work towards creating a
KMail client library that can be used by other apps to create
composer/message view/message list view/mail widgets in that apps
process space. Other generally useful mail services could also be
implemented in the client library.

Except when running KMail standalone, this client library would
communicate with the KMail server/kernel via a KMail dcop interface.
This interface would provide general mail services that other KDE
applications could also use, but this interface would be subject to
change without notice and hence would not initially be part of the
standard KDE API.

When running KMail standalone the client library could directly call
kernel methods rather than using the dcop interface this would avoid
dcop complicating the KMail debugging process.

At least that's my understanding of the approach. Ultimately if
Andreas is to be trusted with implementing this change then I believe
he should also be trusted to make the final decision on how the code
should be written.

Don.
Post by Ingo Klöcker
Regarding the comment.
+// TODO - this needs effort to separate kernel and GUI
This issue should be resolved before the patch is committed to
cvs. Is it possible to use a SYNC dcop method to show a Yes/No
messagebox and return the result? Eventually we might be able
to architect KMail so that only ASYNC dcop methods are used,
but I think it makes sense to have a transitional period where
SYNC methods are used.
I've already thought about this issue, and I agree with you. It
can get solved using a sync dcop call right now, but I also
thought about a way to avoid such sync calls later. But this
requires deeper changes in the existing codebase, I would like to
concentrate on the dcop issue for now. I'll commit as soon as I
have the TODOs eliminated.
I just returned from LinuxTag 2003 in Karlsruhe. During LinuxTag I
talked with Cornelius, Tobias (tokoe) and Marc about the UI
separation. No decisions where made but we definitely see the need
to talk about this some more because this is a very serious
decision which will have a very high impact on the future of KMail
(So why Cornelius and Tobias? Because they would like to use some
of KMail's functionality in KO resp. in KAB). Although I was at
first quite enthusiastic about the DCOP approach, the others don't
think that using DCOP is a good idea. So we really need to talk
about this in a larger group. I would prefer to postpone any
decision until Nove Hrady where we will have the unique opportunity
to talk about this from face to face.
I'm very sorry, Andreas, for all the work that you put into this,
but please understand that we mustn't rush such a radical change
into KMail without giving it a lot of thought.
For KDE 3.2 we primarily have to concentrate on making Kontact a
full-featured and rock-stable Kolab client because that's what the
corporate users need. LinuxTag showed that a lot of people are
waiting for Kontact because without a working Outlook/Exchange
replacement it's impossible for them to even think about migrating
from Windows to Linux.
Regards,
Ingo
Andreas Gungl
2003-07-15 20:02:05 UTC
Permalink
Cornelius has supported my take on this "design" and Coolo did so, too.
Till hasn't yet commented on this issue, neither has Danimo. I hope
they speak up before it's too late and this stuff is pressed into KDE
3.2. If at all, this is post-3.2 stuff, IMO.
Marc, calm down. Nobody ever wanted to get something pressed into KDE 3.2. I
think that all my contributions to KMail (although most of them before you
joined the team) were made serious and carefully. I won't commit my current
code before a consensus have been found, although I think that my changes
wouldn't break anything. And I want to point out that I'm also very
interested in a solid Kolab-enabled Kontact.

I'm just wondering why you didn't take part in the discussion of the UI
separation issue some weeks ago. And I don't understand why you don't
uncover your technical arguments pro and contra whatever solution.

Okay, what have I done so far? I've provided two kind of prototypes
realizing two very different approaches for the UI separation. Technical
details can get read from my various posts to the list. Both approaches are
possible in general, they can even get combined. However, they are bound to
different aspects of what target should be reached. My first approach was
about having two diffent UIs (text and current GUI). The DCOP approach came
into play to allow 3rd party applications (in the style of e.g. KOrganizer)
access to some interesting functions of the KMail kernel. The problem there
is, that within the kernel you can find some GUI releated code (like
KMessageBox calls), which are a problem when you work from a text UI. This
is addressed by my second patch.

I recommend to think over the targets to be worked for before any further
technical discussion continues. Is it to have different UIs? Is it to allow
many other applications (perhaps still unknown today) to interact with
KMail's core? Is it "only" a cleanup to separate core and UI code in the
current codebase?

Of course you can discuss it in NH, but IMO it would be of a good help to at
least show the personal preferences of each core develper before the
conference in NH. There may be a lot of people out there which would like
to comment on the basic ideas (independent of what implementation will be
used) or even vote for a functionality.
I support the idea of using NH for discussion. Everyone agrees that
face-to-face meetings are that much more suited to discussing this type
of things. If Andreas is not scheduled for NH, then maybe he should
reconsider. There's definitely going to be a whole lot of KDE-PIM
discussion and hacking going on there!
If I can convince somebody to sponsor my trip (I'm working on it, but the
chances are low) then I would like to attend. (Is there any deadline to
register?)
However, the developer meeting will be successfull anyway - with or without
me. And I'm sure, that a good way will be found for the UI separation
process. If that's done carefully, why shouldn't it begin even in KDE 3.2?

Andreas
- --
~
' v '
// \\
/( )\ Powered by Penguin.
^ ' ^
Don Sanders
2003-07-17 06:02:25 UTC
Permalink
Post by Andreas Gungl
I recommend to think over the targets to be worked for before any further
technical discussion continues. Is it to have different UIs? Is it to allow
many other applications (perhaps still unknown today) to interact with
KMail's core? Is it "only" a cleanup to separate core and UI code in the
current codebase?
Probably all 3 of them should be considered. Personally speaking I
think we should start cleaning up the codebase by cleaning up UI
and processing code.
I think this is possibly a way forward.

Regardless of whether DCOP is used or not there seems to be a common
desire to partition KMail into processing and UI code spaces.

Assuming the processing code is basically the same as the
kernel/server code space idea and the UI code is basically the same
as the client code space idea, and that cleaning up includes
separating the processing/UI code then I think this is sensible
longer term goal.

Futhermore I think there is work towards this longer term goal that
can be started now. Especially if Andreas has time available to
contribute now then I think it makes sense to use this resource
rather than telling him to delay for yet another month. (Last patch
was sent on 20-June)

More specifically I'm thinking that Andreas' KMailClient class makes
sense whether or not DCOP is used. Perhaps Andreas could consider
creating a non DCOP version of his patch, and adding a global
KMailClient *kmclient variable to KMail.

This kmclient variable would act as the dual of kmkernel. kmclient
would be used within kernel space (e.g. KMFolder*) whenever gui stuff
needs to be done. There's the kmbroadcaststatus issue for instance
that I mentioned, maybe this could just be solved by replacing direct
calls to kmbroadcaststatus in the kernel with calls to
kmclient->broadCastStatus()->method(args). So in general proxy UI
class in kernel space through kmclient. (This would include stuff
like creating a composer in kmkernel). Also later on further work
could be done in investigating what else needs to be done in order to
get kmkernel and kmclient running in separate processes.

Andreas' what do you think of this idea? Does it make any sense?

Ingo, Carsten, Zack, (I've given up on Marc) do you think a workable
solution (submittable patch) could be found that initially doesn't
use DCOP? Eventually to get the client and server classes running in
different spaces we would have to use something like DCOP, and
eventually KMClient could be put in a library. Once in a library
KMClient functionality like creating a composer etc would be exposed
to other Kontact/kdepim apps like KAddressBook and then when stable
would be available for all KDE apps.

The reason why I would like to work on this now rather than waiting
for Nove Hrady is that it's easy to sit around and discuss design in
a committee. The hard part is actually trying to get the ideas
implemented and working in code, that's the frustrating part when our
mental model or the code collides with reality and the real problems
become apparent. If Andreas is able to put time into doing this hard
work now, then I think it makes sense to capitalize on this
opportunity rather than squander it, and risk discouraging him.
This can be started in 3.2.
If you look at kontact the most important thing about this
separation is probably that it makes an integration in other apps
is easier. We regularly get bug reports (wishes) to support other
guis for kmail but I consider this only a sideeffect.
I strongly vote for not mixing up things too badly for 3.2 as the
risk for bugs is too high.
Yeah, personally I want to spend some time on setting up my home
network, and a new website, reducing the load from my girl friend
process, bug fixing, starting with reproducing some hard to reproduce
problems, and then moving onto bug:4202 and bug:50997.

Don.
Marc Mutz
2003-07-18 08:44:59 UTC
Permalink
On Tuesday 15 July 2003 22:02, Andreas Gungl wrote:
<snip>
Post by Andreas Gungl
I'm just wondering why you didn't take part in the discussion of the
UI separation issue some weeks ago. And I don't understand why you
don't uncover your technical arguments pro and contra whatever
solution.
<snip>

I posted a while ago that I'm working on getting my diploma (master)
thesis out of the door and thus have internet only sporadically (to
keep me focused ;-).

First of all: I'd very much welcome work from anyone to refactor KMail
to have a decent MVC-like design and I wouldn't mind adding _some_
_general_ (_C++_) interfaces for that. In fact, I'd probably also start
with making an interface for the status bar service that the main
window provides, however, it would be KMail-internal, called StatusBar
and KMMainWindow would implement it in terms of it's KStatusBar.

So I'm mainly opposing the split of KMail into a server and a client.
That's opening a can of worms w.r.t.:

1. Compatibility of the on-the-wire format: who here can guarantee that
you keep the protocol stable for at least three years, which is the
minimum update interval for companies? Think back where KMail was
three(!) years ago. Do you want to keep interfaces that old? Now that
KMail slowly starts to get a decent internal design again? Do you want
to maintain the forwards- and backwards compatibility layers that are
necessary that KMail/Home can work with a two-years-old KMail/Work? If
not, then I'd suggest we forget this idea again. Don knows this, which
Post by Andreas Gungl
"KMailClientInterface is subject to change without notice and
not yet part of the standard KDE API."
This is to avoid the need to keep backwards compatibility with third
parties, and make sure that we are free to change this API at will.
Initially I would like to restrict the use of this interface to
kde-pim.
But that's not going to work out since the problem is not cross-module
compatibility, but cross-version compat (see above). The "third party"
here is KMail 1.8!

2. Distributed systems: I don't know about you, but I find debugging
multi-threaded and otherwise concurrent or distributed applications a
real horror. If neither you nor a decent percentage of other KMail
developers have any real-world experience with this, I'd suggest we
forget this idea again.

<rant>
All this "kmail server" - sorry - nonsense is just trying to work around
KMail's inability to have decent locking and new-mail-detection for
~/Mail and other stuff. But instead of fixing _that_, removing business
logic[1] from GUI classes and splitting off a libkmailcore.so that a
KDE and a curses front-end could link against and instead of getting
KMail's immense appetite for memory under control, the advocates of
this proprietory IMAP replacement just try to fix the symptoms.
</rant>
What I've heard as arguments of LT2k3 pro the client-server design was
exactly that:
1. Multiple KMail instances'd take too much RAM.
2. KMail wouldn't like other instances accessing it's index files.
Are there more arguments for a client-server "design" than those? Funny
thing is that everyone (correct me if I'm wrong) that I talked to there
about this now seems to agree that this would just be a a masturbation
in distributed computing.

Another thing: KMail interfaces don't help KDE as a whole a lot. Small,
task-oriented interfaces such as those that David, Danimo, et al. came
up with for Kontact do, since they stand a chance of being implemented
by other KDE MUAs (you know that there are two others? ;-)). If a
method's signature contains (..,Q_UINT32 sernum,...), then it's
completely worthless.

OTOH, linking to a kmailcore.so is cheap on the resources, forces a
business logic/UI separation just as well (if not better) and in
addition forces us to make KMail robust w.r.t. "external" accesses in
~/Mail and memory usage. Furthermore, this allows programs such a KOrg
to send emails (invitations) without a running KMail instance. KDE apps
don't need a running instance of Konq to access http or ftp URLs, now
do they? They don't need a running kaddressbook to access the
addressbook, they don't need a running KOrg to parse iCal, they don't
need a running Konq to display HTML pages. Why the heck should KMail be
so special that all these examples don't hold for it?

Marc

[1] That's the common term for what Don calls "kernel code", something
that is but a KMail-ism uncomprehensible to anyone but KMail
developers.
--
It's good fortune for the government that the masses don't think.
-- Adolf Hitler
Martin Konold
2003-07-19 07:53:53 UTC
Permalink
Am Freitag, 18. Juli 2003 10:44 schrieb Marc Mutz:

Hi,
Post by Marc Mutz
the advocates of
this proprietory IMAP replacement just try to fix the symptoms.
I think that this is the strongest argument against this c/s idea! if you want
to seperate the storage from the GUI use IMAP but dont invent another
protocol!

If you want different clients to concurrently access the mail storage use
IMAP.....

IMHO the current proposal is completeley overengineered and reinventing the
wheel.

Yours,
-- martin

Dipl.-Phys. Martin Konold
e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: ***@erfrakon.de
Don Sanders
2003-07-21 02:38:02 UTC
Permalink
Post by Martin Konold
Hi,
Post by Marc Mutz
the advocates of
this proprietory IMAP replacement just try to fix the symptoms.
I think that this is the strongest argument against this c/s idea!
if you want to seperate the storage from the GUI use IMAP but dont
invent another protocol!
If you want different clients to concurrently access the mail
storage use IMAP.....
The problem with this suggestion is that even for IMAP KMail needs a
local representation of what's on the server. eg. dimap uses a
maildir representation.

We want to allow multiple clients to access this local representation.
Thus we have a case of multiple processes (one for each client)
accessing a shared resource (the local representation) and hence we
need a mechanism to manage access to that resource.

I know of two approaches to solve this problem, the first is to use a
single server process to manage access. The second is to make every
client a server and create a peer-peer network.

The proponents of the serverless model don't seem to fully understand
the complexities of managing shared access to a resource. I covered
some of those overlooked complexities in my latest mail to Marc.
Post by Martin Konold
IMHO the current proposal is completeley overengineered and
reinventing the wheel.
It's the simplest solution I know of and in my opinion the only
feasible solution presented thus far.

Don.
Martin Konold
2003-07-21 08:38:13 UTC
Permalink
Am Montag, 21. Juli 2003 04:38 schrieb Don Sanders:

Hi Don,
Post by Don Sanders
Post by Martin Konold
If you want different clients to concurrently access the mail
storage use IMAP.....
The problem with this suggestion is that even for IMAP KMail needs a
local representation of what's on the server. eg. dimap uses a
maildir representation.
No, this is not an issue at all. By definition dimap uses a _private_ local
cache. No external program is under any circumstances allowed to make any use
(reading or writing) of this cache. The fact that the current code uses the
maildir format is completly irrelevant here. dimap could also have been
implemented using some object serialization method.
Maildir was simply chosen because it is a reliable and performant data
structure to store mail data on disk.
Post by Don Sanders
We want to allow multiple clients to access this local representation.
Why? It makes definetly _no_ sense in the dimap case.
Post by Don Sanders
Thus we have a case of multiple processes (one for each client)
accessing a shared resource (the local representation) and hence we
need a mechanism to manage access to that resource.
I am afraid that your presumptions are invalid initially.
Post by Don Sanders
Post by Martin Konold
IMHO the current proposal is completeley overengineered and
reinventing the wheel.
It's the simplest solution I know of and in my opinion the only
feasible solution presented thus far.
I think we have to go back to defining the problem before trying to find the
solution.

Yours,
--martin

Dipl.-Phys. Martin Konold
e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: ***@erfrakon.de
Don Sanders
2003-07-27 22:21:33 UTC
Permalink
Post by Martin Konold
Hi Don,
Post by Don Sanders
Post by Martin Konold
If you want different clients to concurrently access the mail
storage use IMAP.....
The problem with this suggestion is that even for IMAP KMail
needs a local representation of what's on the server. eg. dimap
uses a maildir representation.
No, this is not an issue at all. By definition dimap uses a
_private_ local cache. No external program is under any
circumstances allowed to make any use (reading or writing) of this
cache. The fact that the current code uses the maildir format is
completly irrelevant here. dimap could also have been implemented
using some object serialization method.
Maildir was simply chosen because it is a reliable and performant
data structure to store mail data on disk.
Right, it doesn't matter whether KMail uses maildir or mbox or
something else but it does matter that it uses some local
representation. That a local representation exists.
Post by Martin Konold
Post by Don Sanders
We want to allow multiple clients to access this local
representation.
Why? It makes definetly _no_ sense in the dimap case.
It makes sense even in the dimap case. We are talking about the case
of multiple clients running on the same machine. It doesn't make
sense for each client to have it's own private local cache. For me
that would mean that each new KMail client would have to copy ~2GB of
data, or re-retrieve that data from the server. I don't consider that
acceptable behavior.

Hence the private caches should be shared.
Post by Martin Konold
Post by Don Sanders
Thus we have a case of multiple processes (one for each client)
accessing a shared resource (the local representation) and hence
we need a mechanism to manage access to that resource.
I am afraid that your presumptions are invalid initially.
It's possible we disagree as to whether the local imap representation
should be shared.

Personally I think it's clearly a good idea to share the local imap
representation amongst kmail instances.
Post by Martin Konold
Post by Don Sanders
Post by Martin Konold
IMHO the current proposal is completeley overengineered and
reinventing the wheel.
It's the simplest solution I know of and in my opinion the only
feasible solution presented thus far.
I think we have to go back to defining the problem before trying to
find the solution.
Componentizing KMail so that the Composer, Mail Reader and
Folder/Messages list can be embedded in other apps. My traditional
example being a search folder and mail reader in KAddressBook.

Don.

Don Sanders
2003-07-19 09:35:04 UTC
Permalink
On Friday 18 July 2003 18:44, Marc Mutz wrote:
...
Post by Marc Mutz
So I'm mainly opposing the split of KMail into a server and a
1. Compatibility of the on-the-wire format: who here can guarantee
that you keep the protocol stable for at least three years, which
is the minimum update interval for companies? Think back where
KMail was three(!) years ago. Do you want to keep interfaces that
old? Now that KMail slowly starts to get a decent internal design
again? Do you want to maintain the forwards- and backwards
compatibility layers that are necessary that KMail/Home can work
with a two-years-old KMail/Work? If not, then I'd suggest we forget
Post by Andreas Gungl
"KMailClientInterface is subject to change without notice and
not yet part of the standard KDE API."
This is to avoid the need to keep backwards compatibility with
third parties, and make sure that we are free to change this API
at will. Initially I would like to restrict the use of this
interface to kde-pim.
But that's not going to work out since the problem is not
cross-module compatibility, but cross-version compat (see above).
The "third party" here is KMail 1.8!
KMail 1.8 (and later) will contain both the client and server source
code, so the kernel/client code will come in the same package. There
will be no need to keep compatability between server/client
interfaces in different KMail versions, just as there is no need to
freeze the interface of the c++ classes currently in the KMail source
code.

That's why I asked Andreas to mark the server interfaces as private,
so that other apps don't use them. There's no reason why a DCOP
interface can't be reserved for internal KMail (or maybe kdepim) use
only.

This is important to understand, I advocate the creation of a client
library like you, but unlike you I think the client libraries need to
talk to a server process.
Post by Marc Mutz
2. Distributed systems: I don't know about you, but I find
debugging multi-threaded and otherwise concurrent or distributed
applications a real horror. If neither you nor a decent percentage
of other KMail developers have any real-world experience with this,
I'd suggest we forget this idea again.
I've been doing it for many years, and I believe Andreas' day job is
working on a multi-tier system.
Post by Marc Mutz
<rant>
All this "kmail server" - sorry - nonsense is just trying to work
around KMail's inability to have decent locking and
new-mail-detection for ~/Mail and other stuff. But instead of
fixing _that_,
I think this is the key difference in our thinking. I think due to
KMail's index files (which I think are a good idea for performance
reasons) trying to solve this problem is similar to designing a
database where the client apps run in a different process from the
database itself. And I think solving this process requires having the
client libraries talk to a common server process.

I don't think it's a good idea for each client to have a copy of
KMFolder, KMFolderMgr etc using up memory and having to be kept in
sync with all the other clients.
Post by Marc Mutz
removing business logic[1] from GUI classes and
splitting off a libkmailcore.so that a KDE and a curses front-end
could link against and instead of getting KMail's immense appetite
for memory under control,
I've added unGetMsg so after a mail is read memory allocated for that
message could be deallocated, changed most KMail actions that act on
multiple messages so that they don't load all messages into memory at
once (I didn't manage saveAs, Till got that recently), worked with
Sam to mmap the index and make the loading of records in the index
lazy, and changed the listview of message headers so that it doesn't
duplicate data held in the folder class, and implemented zero-copy
parsing via mmaping of messages so that large messages can (in the
future) be handled efficiently. Rendering of messages is still
inefficient but I covered why (handling of attachments) in my
zero-copy parsing mail.

I think that shows I'm concerned with optimizing KMail's memory use.
Post by Marc Mutz
the advocates of this proprietory IMAP
replacement just try to fix the symptoms. </rant>
What I've heard as arguments of LT2k3 pro the client-server design
1. Multiple KMail instances'd take too much RAM.
2. KMail wouldn't like other instances accessing it's index
files. Are there more arguments for a client-server "design" than
those?
I think these are the main ones.

2 being the most difficult. Trying to allow multiple processes to
access the same (parts of a) file in a safe and efficient way without
the use of a helper process is a difficult problem. I think you're
vastly underestimating the difficulty of this problem.

If KAddressBook is showing the results of a search and being used to
delete messages, and KMail is checking mail then how are you
intending to have these different processes modify the .index files
at nearly the same time? How will the integrity of KMFolder objects
be preserved in KMail when the KAddressBook process is modifying the
.index files for that KMFolder?

Please provide a link to example software that solves this problem, or
outline a sketch of how you intend to solve it.
Post by Marc Mutz
Funny thing is that everyone (correct me if I'm wrong) that
I talked to there about this now seems to agree that this would
just be a a masturbation in distributed computing.
It's the standard database design to solve the standard database
problem. Distributed computing normally refers to distributing work
across multiple machines so it's not related to that.

Since at linuxtag there was no one present to argue for the
server/client idea it's no surprise that it did not meet with
widespread support there. I suspect most linuxtag attendees had
better things to do than argue with you about KMail design.
Post by Marc Mutz
Another thing: KMail interfaces don't help KDE as a whole a lot.
Small, task-oriented interfaces such as those that David, Danimo,
et al. came up with for Kontact do, since they stand a chance of
being implemented by other KDE MUAs (you know that there are two
others? ;-)).
If a method's signature contains (..,Q_UINT32
sernum,...), then it's completely worthless.
OTOH, linking to a kmailcore.so is cheap on the resources, forces a
business logic/UI separation just as well (if not better) and in
addition forces us to make KMail robust w.r.t. "external" accesses
in ~/Mail and memory usage. Furthermore, this allows programs such
a KOrg to send emails (invitations) without a running KMail
instance. KDE apps don't need a running instance of Konq to access
http or ftp URLs, now do they?
They don't need a running
kaddressbook to access the addressbook, they don't need a running
KOrg to parse iCal, they don't need a running Konq to display HTML
pages. Why the heck should KMail be so special that all these
examples don't hold for it?
KMail is bit more like a database than KOrg/Konq as it has index
files. But looking at Konq is still useful.

Consider Konq it's split into server/client as I would like to see
happen with KMail. The client part is the Konqueror app the server
part consists of multiple daemons, one klauncher to handle scheduling
of kioslaves and another kio_uiserver to handle UI (such as an error
dialog when a slave encounters a problem).

There's some documentation about this in kdelibs/kio/DESIGN.

I would like to change KMail so it is similar to Konq. So like KMail
being able to use the file kioslave without the Konq app running,
KOrg could send emails without requiring the GUI part of KMail to be
running. Instead only the server/kernel part would need to be
running.

I think that covers all relevant points, if I've missed one could
someone please bring any omissions to my attention.

Don.
Ingo Klöcker
2003-07-19 11:52:24 UTC
Permalink
Post by Don Sanders
Post by Marc Mutz
But that's not going to work out since the problem is not
cross-module compatibility, but cross-version compat (see above).
The "third party" here is KMail 1.8!
KMail 1.8 (and later) will contain both the client and server source
code, so the kernel/client code will come in the same package. There
will be no need to keep compatability between server/client
interfaces in different KMail versions, just as there is no need to
freeze the interface of the c++ classes currently in the KMail source
code.
That's why I asked Andreas to mark the server interfaces as private,
so that other apps don't use them. There's no reason why a DCOP
interface can't be reserved for internal KMail (or maybe kdepim) use
only.
Then you miss the whole point of the DCOP interface. The DCOP interface
must allow any application to communicate with KMail/the server. And
therefore the interface must not be changed during minor releases.
Post by Don Sanders
This is important to understand, I advocate the creation of a client
library like you, but unlike you I think the client libraries need to
talk to a server process.
But then all applications that want to communicate with KMail would have
to link against this library.
Post by Don Sanders
Post by Marc Mutz
the advocates of this proprietory IMAP
replacement just try to fix the symptoms. </rant>
What I've heard as arguments of LT2k3 pro the client-server design
1. Multiple KMail instances'd take too much RAM.
2. KMail wouldn't like other instances accessing it's index
files. Are there more arguments for a client-server "design" than
those?
I think these are the main ones.
2 being the most difficult. Trying to allow multiple processes to
access the same (parts of a) file in a safe and efficient way without
the use of a helper process is a difficult problem. I think you're
vastly underestimating the difficulty of this problem.
Strangely enough sendmail, procmail and several running mutts don't have
a problem with concurrently accessing /var/spool/mail/user.
Post by Don Sanders
If KAddressBook is showing the results of a search and being used to
delete messages, and KMail is checking mail then how are you
intending to have these different processes modify the .index files
at nearly the same time? How will the integrity of KMFolder objects
be preserved in KMail when the KAddressBook process is modifying the
.index files for that KMFolder?
Please provide a link to example software that solves this problem,
or outline a sketch of how you intend to solve it.
See above. It's as easy as
do {
while ( exists .foldername.lock.* ) wait;
if ( timeout ) return TIMEOUT;
touch .foldername.lock.hostname.PID
// avoid race condition
if ( .foldername.lock.* != .foldername.lock.hostname.PID ) {
unlink .foldername.lock.hostname.PID
wait random amount of time
}
} until ( exists .foldername.lock.hostname.PID )
do operation on folder
unlink .foldername.lock.hostname.PID
Post by Don Sanders
KMail is bit more like a database than KOrg/Konq as it has index
files. But looking at Konq is still useful.
Consider Konq it's split into server/client as I would like to see
happen with KMail. The client part is the Konqueror app the server
part consists of multiple daemons, one klauncher to handle scheduling
of kioslaves and another kio_uiserver to handle UI (such as an error
dialog when a slave encounters a problem).
There's some documentation about this in kdelibs/kio/DESIGN.
I would like to change KMail so it is similar to Konq. So like KMail
being able to use the file kioslave without the Konq app running,
KOrg could send emails without requiring the GUI part of KMail to be
running. Instead only the server/kernel part would need to be
running.
I think that covers all relevant points, if I've missed one could
someone please bring any omissions to my attention.
Hmm, so do you want to write kioslaves for accessing the directories in
~/Mail?

Regards,
Ingo
Don Sanders
2003-07-19 07:23:27 UTC
Permalink
Post by Andreas Gungl
I recommend to think over the targets to be worked for before any
further technical discussion continues. Is it to have different
UIs?
No. At least not in the foreseeable future. There are already very
capable text MUAs. IMO it doesn't make sense to even think about
writing a text KMail client.
Post by Andreas Gungl
Is it to allow many other applications (perhaps still unknown
today) to interact with KMail's core?
Definitely. The most important apps here are the KO and KAB parts
in Kontact. If Kontact is used as Kolab client then currently the
KM part has to run to allow KO and KAB to access the data which is
stored on the IMAP server (which is part of Kolab).
Post by Andreas Gungl
Is it "only" a cleanup to separate core and UI code in the
current codebase?
No. It's definitely more than a cleanup (see above).
a) We separate the backend in a (lightweight) library which will be
used by the KMail GUI and also by KO and KAB.
b) We finally make KMail work with other applications (including
several KMails) that access ~/Mail. So we need to add proper folder
locking and we need to watch the folders for changes.
c) We kpartify the Composer and the ReaderWin so that they can be
used inside other applications.
And you think that it makes sense to create this library without a
helper process?

You might like to consider why NFS and databases like mysql and
postgres use a helper process. And why a library that provides access
to messages in folders is similiar to a database that provides access
to records in tables or NFS that provides access to files in
directories.

In order to efficiently retrieve and store information databases
normally use a data structure known as a btree, a btree in some sense
indexes the information in the database, a bit like KMail index
files. It doesn't make sense for every database client to keep it's
own representation of the btree in memory, due to complexity of
keeping these representations in sync (database replication problem)
and the memory cost of having each client keep a copy of the
representation.

Likewise it doesn't make sense for every KMail client to keep a copy
of KMFolderMgr and all the KMFolders in memory and burden KMail with
further logic to keep these multiple copies in sync.

(Having said that peer to peer network technologies successfully
address these problems, but I think they are best reserved for the
case where the peers are running on different machines).

If you come to the conclusion that a helper process is necessary then
you might like to consider whether the kernel part of KMail is a good
candidate to fulfill that role.
I don't think that a complete separation of KMail into a server
- Communication via DCOP causes much overhead because every little
piece of data has to be put into a stream and has then to be read
again from the stream. With a) this isn't necessary.
DCOP being simple is a good candidate to try first. But if DCOP proves
too inefficient, KMail could be optimized or another IPC mechanism
can be considered, and I believe IPC is needed.
- We would have to add a whole lot of error handling code for each
DCOP communication. Again with a) this isn't necessary.
It's no more/less necessary to add error handling for a DCOP
interface than for a library interface. Because if DCOP calls can fail
then so can the library calls (due to failed file locking if nothing
else).
- We would have to add scheduling code to the server to make it
work with concurrent clients. With a) and b) this isn't necessary.
With DCOP you get this for free, DCOP would be a pretty useless IPC
mechanism if it didn't queue calls.
- DCOP can only be used to exchange messages on the local machine.
So if the user wants to run multiple KMails on different machines
accessing the same ~/Mail directory (e. g. via NFS) the
server/client separation doesn't help at all. b) solves this
problem.
X11 natively supports displaying applications on different machines
from that which they are being run on. No need to reinvent the wheel
here.
BTW, a simple solution for the callback stuff is the usage of
signals and slots. I don't think it's necessary to add C-style
callback functions just to be able to get some error messages from
the kernel on the screen.
I'm not advocating C-style callbacks, I'm advocating partitioning
KMail into a server and client code spaces. And I'm trying to avoid
committing to DCOP at this stage, so as in my thinking (humor me) the
client and server should be able to run in different processes
signals and slots aren't sufficient.

Don.
Ingo Klöcker
2003-07-19 11:31:08 UTC
Permalink
Post by Don Sanders
a) We separate the backend in a (lightweight) library which will be
used by the KMail GUI and also by KO and KAB.
b) We finally make KMail work with other applications (including
several KMails) that access ~/Mail. So we need to add proper folder
locking and we need to watch the folders for changes.
c) We kpartify the Composer and the ReaderWin so that they can be
used inside other applications.
And you think that it makes sense to create this library without a
helper process?
You might like to consider why NFS and databases like mysql and
postgres use a helper process. And why a library that provides access
to messages in folders is similiar to a database that provides access
to records in tables or NFS that provides access to files in
directories.
In order to efficiently retrieve and store information databases
normally use a data structure known as a btree, a btree in some sense
indexes the information in the database, a bit like KMail index
files. It doesn't make sense for every database client to keep it's
own representation of the btree in memory, due to complexity of
keeping these representations in sync (database replication problem)
and the memory cost of having each client keep a copy of the
representation.
Likewise it doesn't make sense for every KMail client to keep a copy
of KMFolderMgr and all the KMFolders in memory and burden KMail with
further logic to keep these multiple copies in sync.
That sounds as if you wanted to advertise the implementation of a
database backend for KMail (which isn't a bad idea at all).
Post by Don Sanders
I don't think that a complete separation of KMail into a server
- Communication via DCOP causes much overhead because every little
piece of data has to be put into a stream and has then to be read
again from the stream. With a) this isn't necessary.
DCOP being simple is a good candidate to try first. But if DCOP
proves too inefficient, KMail could be optimized or another IPC
mechanism can be considered, and I believe IPC is needed.
Yeah, let's use a proprietary IPC mechanism. Not. I thought you wanted
to make it possible for other KDE applications to use the daemon. Then
you will have to use DCOP unless you want to introduce another layer in
terms of a library which hides this proprietary IPC from clients who
want to access the daemon.
Post by Don Sanders
- We would have to add a whole lot of error handling code for each
DCOP communication. Again with a) this isn't necessary.
It's no more/less necessary to add error handling for a DCOP
interface than for a library interface. Because if DCOP calls can
fail then so can the library calls (due to failed file locking if
nothing else).
Wrong. We would have to add additional code to handle failed DCOP calls.
With a library that's not necessary because if the library couldn't be
loaded then the app didn't start in the first place.
Post by Don Sanders
- We would have to add scheduling code to the server to make it
work with concurrent clients. With a) and b) this isn't necessary.
With DCOP you get this for free, DCOP would be a pretty useless IPC
mechanism if it didn't queue calls.
DCOP doesn't really help if the server is performing an expensive
operation for one client while another client only wants to get a
message. In this case the server should (if possible) interrupt the
expensive operation, give the other client the message and then
continue the expensive operation.
Post by Don Sanders
- DCOP can only be used to exchange messages on the local machine.
So if the user wants to run multiple KMails on different machines
accessing the same ~/Mail directory (e. g. via NFS) the
server/client separation doesn't help at all. b) solves this
problem.
X11 natively supports displaying applications on different machines
from that which they are being run on. No need to reinvent the wheel
here.
Yeah, let's go back to the good old times where all applications ran on
a server and the local machines were mere terminals. Sorry, but people
who have an NFS mounted home directory want to run KMail on the local
machine.

BTW, you didn't say anything about b) (i. e. stop KMail's dictatorship
of ~/Mail).

Regards,
Ingo
Don Sanders
2003-07-21 04:20:57 UTC
Permalink
...
Post by Ingo Klöcker
Post by Don Sanders
I don't think that a complete separation of KMail into a server
- Communication via DCOP causes much overhead because every
little piece of data has to be put into a stream and has then
to be read again from the stream. With a) this isn't necessary.
DCOP being simple is a good candidate to try first. But if DCOP
proves too inefficient, KMail could be optimized or another IPC
mechanism can be considered, and I believe IPC is needed.
Yeah, let's use a proprietary IPC mechanism. Not. I thought you
wanted to make it possible for other KDE applications to use the
daemon. Then you will have to use DCOP unless you want to introduce
another layer in terms of a library which hides this proprietary
IPC from clients who want to access the daemon.
I would like to see the creation of a client library, I don't want
random apps accessing the KMail client/server IPC interface.
Post by Ingo Klöcker
Post by Don Sanders
- We would have to add a whole lot of error handling code for
each DCOP communication. Again with a) this isn't necessary.
It's no more/less necessary to add error handling for a DCOP
interface than for a library interface. Because if DCOP calls can
fail then so can the library calls (due to failed file locking if
nothing else).
Wrong. We would have to add additional code to handle failed DCOP
calls. With a library that's not necessary because if the library
couldn't be loaded then the app didn't start in the first place.
Even if the library successfully loads, calls into the library could
still fail, that's all I'm saying.
Post by Ingo Klöcker
Post by Don Sanders
- We would have to add scheduling code to the server to make it
work with concurrent clients. With a) and b) this isn't
necessary.
With DCOP you get this for free, DCOP would be a pretty useless
IPC mechanism if it didn't queue calls.
DCOP doesn't really help if the server is performing an expensive
operation for one client while another client only wants to get a
message. In this case the server should (if possible) interrupt the
expensive operation, give the other client the message and then
continue the expensive operation.
Sure we have problems with time consuming operations blocking the UI.
DCOP won't magically solve those problems.
Post by Ingo Klöcker
Post by Don Sanders
- DCOP can only be used to exchange messages on the local
machine. So if the user wants to run multiple KMails on
different machines accessing the same ~/Mail directory (e. g.
via NFS) the server/client separation doesn't help at all. b)
solves this problem.
X11 natively supports displaying applications on different
machines from that which they are being run on. No need to
reinvent the wheel here.
Yeah, let's go back to the good old times where all applications
ran on a server and the local machines were mere terminals. Sorry,
but people who have an NFS mounted home directory want to run KMail
on the local machine.
I think with NFS there's a reliability/speed tradeoff involved. Most
people seem to trade reliability for speed in which case I don't see
how to reliably solve this problem with either the client/server or
client only solution. Where 'this problem' is having multiple KMail
clients (not running on the NFS server) efficiently and concurrently
accessing mail on an NFS server.

You may believe that you can solve this problem using the client only
solution, but in my opinion KMail can't currently reliably handle
mail on NFS so I'd like to reserve my judgment on that matter.
Post by Ingo Klöcker
BTW, you didn't say anything about b) (i. e. stop KMail's
dictatorship of ~/Mail).
I'm not currently concerned with the dictatorship of ~/Mail. I'm
concerned with a separate problem, allowing multiples KMail clients
to run simultaneously in separate processes on the same machine. I
consider this problems separate.

I think I've addressed all criticisms of the server model. Please
inform me if I missed any.

Ok, a further problem with the client only solution has occurred to
me. To recap here is a summary of the outstanding criticisms:

Allowing other processes to modify KMail's index files while they are
open will result in invalid KMMsgInfo objects that could corrupt the
index files. I'll call this the invalid KMMsgInfo problem.

Two solutions have been suggested, I find neither acceptable.

The first suggestion was to have KMail make a private copy of any
folders it has opened. I have global searches that open most of my
folders. This would mean each client would need to copy about 2gigs
of ~/Mail data. I don't think this acceptable.

The second suggestion was to have KMail lock index files only briefly.
But this would means redesigning KMail so that index files are
read/written whenever an action takes place. Theoretically feasible
but basically requiring a rewrite of most of all the non-GUI KMail
code, and resulting in slow performance when multiple clients
interleave changes into an index file, requiring both clients to
continually reparse the index file.

Which is the second problem the updating multiple clients problem.

If a server is used then there's a central point (the server) where
changes are managed and at this point all clients can be informed of
the changes. The client only solution would require each client to
poll files for changes and possibly reparse the entire index file
each time it is changed. Too expensive for my tastes.

The third problem is the duplicate serial number problem. This is
solvable in the client model by using a shared file. But I do find
this solution very ugly and would prefer to have a single server
process give out numbers instead.

On to a new problem, a fourth problem, the lost messages problem.

Consider the case of the user opening a composer by replying to a
message, moving that message into a different folder and then sending
the reply. In this case the message has to be found and the status
changed to replied. Currently we use the kmmsgdict to find the
message.

With the server model we could continue to use the single kmmsgdict.
But with the client only model there would be multiple kmmsgdicts and
they would no longer be reliable. Because if the message was
replied-to and moved by different clients then the replied-to client
would not know what folder the message was moved into. In the client
only model keeping the kmmsgdict up to date in all the clients would
require polling the index files and reparsing them whenever they
change. So the client only model would require reparsing all folders
whenever they change. I consider this an unnecessary and
unjustifiable expense.

The serial number problem I could overlook as there is a solution to
it, albeit an ugly one. But until solutions to the other three
problems are given I'll consider the client-only model fundamentally
flawed.

Don.
Martin Konold
2003-07-21 09:07:39 UTC
Permalink
Am Samstag, 19. Juli 2003 13:31 schrieb Ingo Klöcker:

Hi,
Post by Ingo Klöcker
That sounds as if you wanted to advertise the implementation of a
database backend for KMail (which isn't a bad idea at all).
This is definetly a bad idea!

Basically databases are optimized for a totally different pattern of data
(mainly fixed size) and totally different access patterns.

Yours,
-- martin

Mit freundlichen Grüßen,
-- martin konold

Dipl.-Phys. Martin Konold
e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: ***@erfrakon.de
Don Sanders
2003-07-24 03:31:30 UTC
Permalink
Post by Martin Konold
Hi,
Post by Ingo Klöcker
That sounds as if you wanted to advertise the implementation of a
database backend for KMail (which isn't a bad idea at all).
This is definetly a bad idea!
Basically databases are optimized for a totally different pattern
of data (mainly fixed size) and totally different access patterns.
I noticed MySQL >= 3.23.23 has support for full-text indexing and
searching.

http://www.mysql.com/doc/en/Fulltext_Search.html

These leads me to think that maybe MySQL is able to handle the task of
storing/searching mail well. Do you have some evidence that
contradicts this?

Not the I'm dedicated to using MySQL to store mail but I'm trying to
keep my options open.

Don.
Ingo Klöcker
2003-07-20 12:58:57 UTC
Permalink
As I have thought a lot about the separation and as I've dealt with
DCOP, I want to make a suggestion from which I do hope that it's a
good chance to reach a consensus.
We should start without DCOP but use the current C++ and (where
usefull) Signal/Slot mechanisms. This is to
- avoid a lot of work writing the DCOP related code for streaming the
data and handling errors in the DCOP layer
- simplify debugging in this early stage of the separation process
- avoid unnecessary risk in the 3.2 release in which the separation
will certainly not be finished.
Since we have to remove all UI code from the kernel before we can think
about a separation, I think this is a good first mile stone.
Even without DCOP we'll have to decide a lot of details. Let's say
deep_in_core_function()
{
do(something);
miss_some_data;
okay = open_password_dialog(i18n("user information"));
if (okay) {
do_my_work();
Messagebox("i18n("everything okay"));
}
else
finish();
}
IMO the function has to be rewritten to have return codes (others may
prefer exceptions) to signal that an user interaction is needed
outside the kernel code. So the function should not return "true" or
"false" but "done", "failed", "password_needed_retry_possible".
Definitely.
Second, it's not a good style to keep UI relevant information like
the text for the messagebox in the core. It was easy to program it
this way in the beginning, but if you want a consequent separation,
then the text belongs in the UI code.
Agreed.
Doing this little bit of refactoring will be a lot of work even
without DCOP. You should also keep in mind, that a good separation
simplifies the introduction of DCOP when we find out that it's really
needed. It could be done in a very short time then. And it could be
done at any time.
But currently it would make sense to concentrate on code like the
pseudo code above. Resolving some of such code for 3.2 is a realistic
goal and might help to clean up the application and to prepare the
way for further changes or enhancements. OTOH this way allow us to go
with little steps. It's not something which is done better in one
step like a
"KPartification". And these little steps are why I would like to work
on it. I cannot guarantee to have time to make big changes in one
rush as sometimes needed for the Kolab related work. But I can do one
small improvement after another.
I hope, that you can accept this suggestion. In this case I would try
to provide another patch. Otherwise I'll wait for a decision in NH.
At least I think this is a very good suggestion. As this cleaning up is
completely independent from whatever we decide about the separation
Andreas can start to work on this without having to wait until we
finally come up with a decision concerning a separation of core/UI.

Regards,
Ingo
Continue reading on narkive:
Loading...