Problem with samba netdrive plugin

[piesse]

Lately (say, since Samba 3.0.33), I am experiencing serious problems of
stability of the NDCTL plugin (both version 1.0 or 1.5beta):

After a while (several times per day), while I start dowloading a file
in firefox (such as opening a pdf), both NDCTL and firefox suddendly
close without warning: the simply disappear. I need to restart Netdrive
from the netdrive control panel to get back the mounted volumes. But
after downloading a few files, it would happen again.

It may not be Firefox, since sometimes it also happened while
browsing/copying files from a CLI window.


Upgrading the Samba server up to 3.0.35 did not change anything (and it
shouldn't, since local drives are unaffected and still available from
outside, and the problem occurs even if samba server isn't running),
however with previous versions of server installed and using version 1.0
of the plugin I never had this problem (but I haven't tryed downgrading,
and might be something else).

I cannot find any trace in popupos2 or in ndctl.log.

Where also could I look?


With Theseus I see that NDCL is taking a huge amount of memory (at
least, in "linear memory usage by process", that I don't understand
much, is one of the programs with the longest column of numbers,
whatever they mean, even more than firefox. Only Clamav gets more). May
be a problem with memory?


Using thunderbird, I cannot find a newsgroup
gmane.org.netlabs.samba.user on news.gmane.org, and using the web board
at http://news.gmane.org/gmane.org.netlabs.samba.user I can't find how
to post a message there (also, the search engine on that page says that
the group probably doesn't exist): how can I contact them?



thanks,
Piersante

[Marcel Müller]

Post by piesse
Lately (say, since Samba 3.0.33), I am experiencing serious problems of
After a while (several times per day), while I start dowloading a file
in firefox (such as opening a pdf), both NDCTL and firefox suddendly
close without warning: the simply disappear. I need to restart Netdrive
from the netdrive control panel to get back the mounted volumes. But
after downloading a few files, it would happen again.
It may not be Firefox, since sometimes it also happened while
browsing/copying files from a CLI window.

All I can say is that I had the same problems. The Netdrive plugin
crashes the Netdrive worker process from time to time. This causes all
open file handles to die immediately. Probably FF dislikes this kind of
unexpected errors and crashes too.

Since I have my mail profile on the server this was unacceptable.
Finally I deleted Netdrive and returned to IBM peer, which is restricted
(to 2GB) but reliable.

Post by piesse
With Theseus I see that NDCL is taking a huge amount of memory (at
least, in "linear memory usage by process", that I don't understand
much, is one of the programs with the longest column of numbers,
whatever they mean, even more than firefox. Only Clamav gets more). May
be a problem with memory?

Hmm, I never looked at that, but it seems to be a reasonable
explanation. Almost no application handles out of memory errors
reasonable because all modern languages like C++ rely on temporary
storage on the heap (at least the use libraries do). So the default
action taken is usually to close the application abnormally. Maybe you
are right and it is simply a memory leak.


Marcel

[Sjoerd Visser]

On Sat, 1 Aug 2009 07:41:21 UTC, Marcel Müller

Post by Marcel Müller

Post by piesse
Lately (say, since Samba 3.0.33), I am experiencing serious problems of
After a while (several times per day), while I start dowloading a file
in firefox (such as opening a pdf), both NDCTL and firefox suddendly
close without warning: the simply disappear. I need to restart Netdrive
from the netdrive control panel to get back the mounted volumes. But
after downloading a few files, it would happen again.
It may not be Firefox, since sometimes it also happened while
browsing/copying files from a CLI window.

All I can say is that I had the same problems. The Netdrive plugin
crashes the Netdrive worker process from time to time. This causes all
open file handles to die immediately. Probably FF dislikes this kind of
unexpected errors and crashes too.
Since I have my mail profile on the server this was unacceptable.
Finally I deleted Netdrive and returned to IBM peer, which is restricted
(to 2GB) but reliable.

Post by piesse
With Theseus I see that NDCL is taking a huge amount of memory (at
least, in "linear memory usage by process", that I don't understand
much, is one of the programs with the longest column of numbers,
whatever they mean, even more than firefox. Only Clamav gets more). May
be a problem with memory?

Hmm, I never looked at that, but it seems to be a reasonable
explanation. Almost no application handles out of memory errors
reasonable because all modern languages like C++ rely on temporary
storage on the heap (at least the use libraries do). So the default
action taken is usually to close the application abnormally. Maybe you
are right and it is simply a memory leak.
Marcel

[Sjoerd Visser]

On Sat, 1 Aug 2009 07:41:21 UTC, Marcel Müller

Post by Marcel Müller

Post by piesse
Lately (say, since Samba 3.0.33), I am experiencing serious problems of
After a while (several times per day), while I start dowloading a file
in firefox (such as opening a pdf), both NDCTL and firefox suddendly
close without warning: the simply disappear. I need to restart Netdrive
from the netdrive control panel to get back the mounted volumes. But
after downloading a few files, it would happen again.
It may not be Firefox, since sometimes it also happened while
browsing/copying files from a CLI window.

All I can say is that I had the same problems. The Netdrive plugin
crashes the Netdrive worker process from time to time. This causes all
open file handles to die immediately. Probably FF dislikes this kind of
unexpected errors and crashes too.
Since I have my mail profile on the server this was unacceptable.
Finally I deleted Netdrive and returned to IBM peer, which is restricted
(to 2GB) but reliable.

Post by piesse
With Theseus I see that NDCL is taking a huge amount of memory (at
least, in "linear memory usage by process", that I don't understand
much, is one of the programs with the longest column of numbers,
whatever they mean, even more than firefox. Only Clamav gets more). May
be a problem with memory?

Hmm, I never looked at that, but it seems to be a reasonable
explanation. Almost no application handles out of memory errors
reasonable because all modern languages like C++ rely on temporary
storage on the heap (at least the use libraries do). So the default
action taken is usually to close the application abnormally. Maybe you
are right and it is simply a memory leak.
Marcel

[Sjoerd Visser]

On Sat, 1 Aug 2009 07:41:21 UTC, Marcel Müller

Post by Marcel Müller

Post by piesse
Lately (say, since Samba 3.0.33), I am experiencing serious problems of
After a while (several times per day), while I start dowloading a file
in firefox (such as opening a pdf), both NDCTL and firefox suddendly
close without warning: the simply disappear. I need to restart Netdrive
from the netdrive control panel to get back the mounted volumes. But
after downloading a few files, it would happen again.
It may not be Firefox, since sometimes it also happened while
browsing/copying files from a CLI window.

All I can say is that I had the same problems. The Netdrive plugin
crashes the Netdrive worker process from time to time. This causes all
open file handles to die immediately. Probably FF dislikes this kind of
unexpected errors and crashes too.
Since I have my mail profile on the server this was unacceptable.
Finally I deleted Netdrive and returned to IBM peer, which is restricted
(to 2GB) but reliable.

Post by piesse
With Theseus I see that NDCL is taking a huge amount of memory (at
least, in "linear memory usage by process", that I don't understand
much, is one of the programs with the longest column of numbers,
whatever they mean, even more than firefox. Only Clamav gets more). May
be a problem with memory?

Hmm, I never looked at that, but it seems to be a reasonable
explanation. Almost no application handles out of memory errors
reasonable because all modern languages like C++ rely on temporary
storage on the heap (at least the use libraries do). So the default
action taken is usually to close the application abnormally. Maybe you
are right and it is simply a memory leak.
Marcel

Even Pronews can get out of resources so my two earlier typed messages
did not arrive.
Sorry for this.

I constantly see problems with the netdrive smb plugin when I exchange
files at the same moment with two or more samba servers. Copying
files to or from host A to B whilst renaming files on host C works,
but exchanging files from OS/2 host A with both hosts B and C with two
instances of FC at the same time too often gives stalls.

Lack of virtual memory (VM) could be a reason (shut down Mozilla and
kill one FC/2 and then click retry often works), but running two
instances of FC/2 should not be a reason of lack of VM (certtainly not
on my 2 GiB memory system) as FC/2 uses mostly private memory (quite a
lot, but that should be okay).

Sudden lack of VM might be caused by Firefox 3 . I have at least one
eCS system where a recent FF3 3.0.10 crashed when downloading a file
, where Peters Weilbachers latest FF2 did not.

I now use OS/2 PEER (as samba client and server) and the netdrive
client (the shareware version) together. But whilst I can have many
active connections via PEER, I can only rely on one active samba
Netdrive plugin connection at the same time. And this (problems when
using more then one active connection) is not a recent bug, but
happened in my experience allready several years.

I don't think these problems were caused by a memory leak. Theseus did
not suggest so. The samb pluin uses little low shared memory. As it
is nothing new to me (as I remember) it might also play a role with
combined ftp and smb connections.

I will have to test this.

Greetings,

Sjoerd Visser

[Marcel Müller]

Hi,

Post by Sjoerd Visser
I constantly see problems with the netdrive smb plugin when I exchange
files at the same moment with two or more samba servers. Copying
files to or from host A to B whilst renaming files on host C works,
but exchanging files from OS/2 host A with both hosts B and C with two
instances of FC at the same time too often gives stalls.

I could bet that I only connected to one Server while Netdrive died. Of
course, there were concurrent activities. Applications like PMView,
T-Bird and even PMshell with XWP and Turbo-Folders turned on can cause
heavy concurrent network traffic. But copy actions in FC/2 were at least
part of the scenario.

Post by Sjoerd Visser
Sudden lack of VM might be caused by Firefox 3 . I have at least one
eCS system where a recent FF3 3.0.10 crashed when downloading a file
, where Peters Weilbachers latest FF2 did not.

FF3 is crap with respect to memory usage. FF3.5 too. They have memory
leaks when working with tabs. They do not call it leak, because closing
the window with all tabs frees the locked memory even if there are other
Windows open. But this is very inconvenient.
Simply open 10 Tabs and switch around between them. This will usually
cause the Memory to raise constantly. Maybe the leak is not in FF itself
but in some common extension like ABP. Maybe it is related to
Javascript. I don't know. I downgraded the computers with less physical
memory to FF2.x.

[...]

Post by Sjoerd Visser
I don't think these problems were caused by a memory leak. Theseus did
not suggest so. The samb pluin uses little low shared memory. As it
is nothing new to me (as I remember) it might also play a role with
combined ftp and smb connections.

What about the private memory usage?

And do we see all resources taken by the IFS component really?

The concurrency issues that you mentioned point to a race condition or
something like that. Difficult to find.


Marcel

[Sjoerd Visser]

On Sun, 9 Aug 2009 07:00:00 UTC, Marcel Müller

Post by Marcel Müller
Hi,

Post by Sjoerd Visser
I constantly see problems with the netdrive smb plugin when I exchange
files at the same moment with two or more samba servers. Copying
files to or from host A to B whilst renaming files on host C works,
but exchanging files from OS/2 host A with both hosts B and C with two
instances of FC at the same time too often gives stalls.

I could bet that I only connected to one Server while Netdrive died. Of
course, there were concurrent activities. Applications like PMView,
T-Bird and even PMshell with XWP and Turbo-Folders turned on can cause
heavy concurrent network traffic. But copy actions in FC/2 were at least
part of the scenario.

As is high (100%) CPU use in SMBCD.EXE and NDCTL.EXE. Killing one
instance of FC/2 can restore the remaining connection. CPU use will
then lower to 20-30% .

Post by Marcel Müller

Post by Sjoerd Visser
Sudden lack of VM might be caused by Firefox 3 . I have at least one
eCS system where a recent FF3 3.0.10 crashed when downloading a file
, where Peters Weilbachers latest FF2 did not.

FF3 is crap with respect to memory usage. FF3.5 too. They have memory
leaks when working with tabs. They do not call it leak, because closing
the window with all tabs frees the locked memory even if there are other
Windows open. But this is very inconvenient.
Simply open 10 Tabs and switch around between them. This will usually
cause the Memory to raise constantly. Maybe the leak is not in FF itself
but in some common extension like ABP. Maybe it is related to
Javascript. I don't know. I downgraded the computers with less physical
memory to FF2.x.

High use of physical memory is not a problem on systemns with 1-2 GIB
RAM. It should even not be a problem with 256 MiB when using lazy
commit (default). But the growing use of virtual adresss space is.

Post by Marcel Müller
[...]

Post by Sjoerd Visser
I don't think these problems were caused by a memory leak. Theseus did
not suggest so. The samb pluin uses little low shared memory. As it
is nothing new to me (as I remember) it might also play a role with
combined ftp and smb connections.

What about the private memory usage?

Just part of the running proceses:

GETRAM version is 1.0.0.
RAM use by process started on Sun Aug 9 22:52:20 2009
Doing 1 time intervals of 0 seconds each.

<Count in Megabytes><Count in Pages>
Pid Private Shared Private Shared Task Names
0000 753.082M 7.883M 192789 2018 system
0037 2.648M 2.984M 678 764 NDCTL
0099 0.145M 0.148M 37 38 PEER
0100 0.273M 0.160M 70 41 PEER
6754 4.277M 0.195M 1095 50 SMBCD
6972 1.199M 0.313M 307 80 SOMDD
14588 32.590M 55.000M 8343 14080 PRONEWS
15195 0.965M 0.355M 247 91 MPEGTV2
15916 1.070M 0.203M 274 52 FC
15980 2.984M 22.973M 764 5881 PMVIEW
16305 0.074M 0.035M 19 9 CMD
16307 0.852M 0.148M 218 38 WATCHCAT
16389 1.590M 2.605M 407 667 ZOC
16468 9.391M 80.379M 2404 20577 FIREFOX
16480 0.172M 0.035M 44 9 CMD
16481 16.230M 0.102M 4155 26 GETRAM
-------- -------- ------- -------
856.277M 211.781M 219207 54216 Total RAM used
979.254M 250689 Free RAM available
-------- -------
2047.313M 524112 Total RAM
(Prvt+Shr+Free)

Firefox has 10 tabs opened.
After Closing 9 tabs virtual memory usage was actually increased:

16468 9.566M 80.906M 2449 20712 FIREFOX

Note that SMBCD.EXE and NDCTL.EXE are modest virtual memory users.
And NDPSMB uses high memory, which is just fine.

There is 232.809M between the private and shared arenas.
Above 512m Shared arena starts at 3D0F7000, which is 976.965M.
Free memory from 3D0F7000 for 388.906M, which is equivalent to 6222
64K spaces.
555DF000 ......................................................o.
1 FLASHWIN allocated it
object is 512K
5565F000 ......................................................o.
21 FLASHWIN allocated it
object is 2.000M
5585F000 ................o............................w..........
160 NDPSMB allocated it
object is 640K
558FF000 .............................................w..........
160 NDPSMB allocated it
object is 640K
5599F000 ................o............................w..........
160 NDPSMB allocated it
object is 640K
55A3F000 ................o............................w..........
160 NDPSMB allocated it
object is 640K
55ADF000 ................o............................w..........
160 NDPSMB allocated it
object is 640K
55B7F000 ................o............................w..........
160 NDPSMB allocated it
object is 640K
55C1F000 .............................................w..........
160 NDPSMB allocated it
object is 640K
55CBF000 ................o............................w..........
32 NDPSMB allocated it
object is 128K
55CDF000 ................o............................w..........
32 NDPSMB allocated it
object is 128K
55CFF000 ................o............................w..........
32 NDPSMB allocated it
object is 128K
55D1F000 ................o............................w..........
32 NDPSMB allocated it
object is 128K
55D3F000 ................o............................w..........
160 NDPSMB allocated it
object is 640K
Free memory from 55DDF000 for 4K.
55DE0000 .................ow..ww.....www....w..w...w....wwww..ww.
1799 FT2LIB allocated it
object is 32.000M
57DE0000 ................o.....w......................w....w...w.
512 LIBC063 allocated it
object is 2.000M
57FE0000 .................ow.wwwwww.wwww..wwwwwwww.w...wwwwwwwwww
0 DRIVER #0000 (]
object is 128K
Total free space in the shared arena = 388.910M.
Return code 1 from RT2GetLinMemMap.

Post by Marcel Müller
And do we see all resources taken by the IFS component really?

No, because they belang to kernel space. But unless you have say 256
MiB videoram and a large JFS cache (never got it above 800 MiB), there
is no lack of virtual memory there.

Post by Marcel Müller
The concurrency issues that you mentioned point to a race condition or
something like that. Difficult to find.

The 100% CPU use and my problems happening with concurrent copy
actions fit better with racing conditions (and thus lack of CPU time)
then with lack of virtual memory.


Netdrive runs with server priority 400 (server priority).This gives it
priority above most normal programs (200), but will be overrruled bu
time critical (300 ) threads

prg detach 25 NDCTL.EXE
Threads
ID PRIO STATUS BLOCKID
��� ���� ������ �������
1 400 blocked fff40394
2 200 blocked fff40394
3 400 blocked 46a8000e
4 400 blocked fa65431c
5 400 blocked fa65435c
6 400 blocked fa65439c
7 400 blocked fa6543dc
8 400 blocked fa65441c

prg detach 1A62 SMBCD.EXE
ID PRIO STATUS BLOCKID
��� ���� ������ �������
1 400 blocked 8f803ae
2 400 blocked 9280014
3 400 blocked 4340000e
4 400 blocked 4680000e
5 400 blocked 4690000e
6 400 blocked 47c0000e
7 400 blocked 4610000e
8 400 blocked 47b8000e
9 400 blocked 46b8000e
10 400 blocked 4788000e

But Firefox also has a 31e thread (10) and plays YouTube video with
time-critical 300 threads (id 12 here).

ID PRIO STATUS BLOCKID
��� ���� ������ �������
1 200 ready fd85cefc
2 200 ready 24054
3 200 blocked fd9e7110
4 300 blocked fd9e6518
5 200 blocked fd9dfc44
6 200 blocked fd9dfc34
7 200 blocked fd9742b4
8 200 blocked fd9e62ec
9 200 blocked fd981394
10 31e blocked c0ac9c28
11 200 blocked fff40394
12 300 blocked fd9dbbe8


These threads have higher priority than Netdrive and could give
network time outs.
Note that PEER runs with time critical priority.

prg detach 63 PEER.EXE

ID PRIO STATUS BLOCKID
��� ���� ������ �������
1 301 blocked fff40394
2 300 blocked fffe0049
3 300 blocked a005b39
4 301 blocked fffe004d

This could explain that it keeps running when Netdrive (with priority
400 must) give up.

[piesse]

Post by Sjoerd Visser
GETRAM version is 1.0.0.

How do I get GETRAM for os/2?

thanks,
Piersante

[Herwig Bauernfeind]

Post by piesse
Lately (say, since Samba 3.0.33), I am experiencing serious problems of
After a while (several times per day), while I start dowloading a file
in firefox (such as opening a pdf), both NDCTL and firefox suddendly
close without warning: the simply disappear. I need to restart Netdrive
from the netdrive control panel to get back the mounted volumes. But
after downloading a few files, it would happen again.
It may not be Firefox, since sometimes it also happened while
browsing/copying files from a CLI window.
Upgrading the Samba server up to 3.0.35 did not change anything (and it
shouldn't, since local drives are unaffected and still available from
outside, and the problem occurs even if samba server isn't running),
however with previous versions of server installed and using version 1.0
of the plugin I never had this problem (but I haven't tryed downgrading,
and might be something else).
I cannot find any trace in popupos2 or in ndctl.log.
Where also could I look?

The Samba Plugin has a debug log feature described in the readme of the
plugin.

In case you file a ticket at http://svn.netlabs.org/samba describe your
problem and provide a debug log of your crash, we might be able to help you.

Regards,
Herwig