2GB Account storage limit. Re: mail store size recalculated, over and over...

Discussion:

(too old to reply)

Vladimir A. Butenko

2003-08-12 14:14:13 UTC

CommuniGate Pro has (and always had) a 32-bit signed counter for the total
storage used by each mailbox and by all mailboxes in the account.

When the account total storage exceeds 2GB, this counter gets the arithmetic
overflow, resulting in negative values. When CGatePro sees a negative (or
and other suspicious) value there, it puts an account into the a special
queue and recalculates all mailbox sizes in a special thread, to restore the
correct value. But this value is "correct", so it ends up with a negative
value again, and the account is placed in that queue again and again.

To avoid these things, the 4.1.x versions check if the account size would
exceed 2GB if a new message is added and report the "over quota" error even
if no quota is set in that Account settings.

Try not to exceed this limit. Remove the unused messages/mailboxes (after
archiving them). If absolutely necessary, move messages to a different
account (call it "archive", for example), and make those account mailboxes
available for you, so you can deal with them from your own account.

We are thinking on how to remove the 2GB limit, but it cannot be removed
quickly.

On Mon, 11 Aug 2003 23:02:14 -0400

recalculated: -680288658
What can I do about it?

account is full (quota exceeded)
but we don't implement quotas! Every setting I can see is that he has
unlimited quotas. What's going on?
-rw-rw---- 1 root mail 617803762 Aug 11 19:59 INBOX.mbox
---Jason Mader, FHWA/NHTSA National Crash Analysis Center,
The George Washington University, VA Campus
#############################################################
This message is sent to you because you are subscribed to

Sincerely,
Vladimir

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

Communigate Forum

2003-08-12 15:43:15 UTC

Permalink

hi,

i had same results when adding a single contact via both cgp web interface
and outlook 2002..

would appear this is a bit of a major over sight and is going to need
resolving very quickly..

rup

----- Original Message -----
From: "Vladimir A. Butenko" <***@stalker.com>
To: "CommuniGate Pro Discussions" <***@mail.stalker.com>
Sent: Tuesday, August 12, 2003 3:14 PM
Subject: 2GB Account storage limit. Re: mail store size recalculated, over
and over...

Post by Vladimir A. Butenko
CommuniGate Pro has (and always had) a 32-bit signed counter for the total
storage used by each mailbox and by all mailboxes in the account.
When the account total storage exceeds 2GB, this counter gets the arithmetic
overflow, resulting in negative values. When CGatePro sees a negative (or
and other suspicious) value there, it puts an account into the a special
queue and recalculates all mailbox sizes in a special thread, to restore the
correct value. But this value is "correct", so it ends up with a negative
value again, and the account is placed in that queue again and again.
To avoid these things, the 4.1.x versions check if the account size would
exceed 2GB if a new message is added and report the "over quota" error even
if no quota is set in that Account settings.
Try not to exceed this limit. Remove the unused messages/mailboxes (after
archiving them). If absolutely necessary, move messages to a different
account (call it "archive", for example), and make those account mailboxes
available for you, so you can deal with them from your own account.
We are thinking on how to remove the 2GB limit, but it cannot be removed
quickly.
On Mon, 11 Aug 2003 23:02:14 -0400

recalculated: -680288658
What can I do about it?

Sincerely,
Vladimir
#############################################################
This message is sent to you because you are subscribed to
**************************************************************
Scanned by MailScan Anti-Virus and Content Security Software.
Visit http://www.mwti.net for more info on eScan and MailScan.
**************************************************************

**************************************************************
Scanned by MailScan Anti-Virus and Content Security Software.
Visit http://www.mwti.net for more info on eScan and MailScan.
**************************************************************

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

Adam Sleight

2003-08-12 16:07:51 UTC

Permalink

One user has 1.5GB and another user 1.3GB....very soon they'll be over 2GB
since they have 'unlimited'. That was the whole point in buying a new mail
server was to alleviate our storage problems. In the past (I can dig it up)
all that was ever said was 2GB is the max your can limit due to 32-bit
calculations, otherwise set it to unlimited. So if setting it to 'unlimited'
requires that CGate doesn't check how much space is used at all then I suppose
that's what were going to need to do. Please figure out something to allow
users truely 'unlimited' storage.

On Tue, 12 Aug 2003 16:43:15 +0100

Post by Communigate Forum
hi,
i had same results when adding a single contact via both cgp web interface
and outlook 2002..
would appear this is a bit of a major over sight and is going to need
resolving very quickly..
rup
----- Original Message -----
Sent: Tuesday, August 12, 2003 3:14 PM
Subject: 2GB Account storage limit. Re: mail store size recalculated, over
and over...

arithmetic

Post by Vladimir A. Butenko
overflow, resulting in negative values. When CGatePro sees a negative (or
and other suspicious) value there, it puts an account into the a special
queue and recalculates all mailbox sizes in a special thread, to restore

the

Post by Vladimir A. Butenko
correct value. But this value is "correct", so it ends up with a negative
value again, and the account is placed in that queue again and again.
To avoid these things, the 4.1.x versions check if the account size would
exceed 2GB if a new message is added and report the "over quota" error

even

Post by Vladimir A. Butenko
if no quota is set in that Account settings.
Try not to exceed this limit. Remove the unused messages/mailboxes (after
archiving them). If absolutely necessary, move messages to a different
account (call it "archive", for example), and make those account mailboxes
available for you, so you can deal with them from your own account.
We are thinking on how to remove the 2GB limit, but it cannot be removed
quickly.
On Mon, 11 Aug 2003 23:02:14 -0400

recalculated: -680288658
What can I do about it?

Sincerely,
Vladimir

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

Chris Demain

2003-08-12 22:09:48 UTC

Permalink

http://www.stalker.com/WebGuide/Mailboxes.html#Formats

Post by Adam Sleight
One user has 1.5GB and another user 1.3GB....very soon they'll be over 2GB
since they have 'unlimited'. That was the whole point in buying a new mail
server was to alleviate our storage problems. In the past (I can dig it up)
all that was ever said was 2GB is the max your can limit due to 32-bit
calculations, otherwise set it to unlimited. So if setting it to 'unlimited'
requires that CGate doesn't check how much space is used at all then I suppose
that's what were going to need to do. Please figure out something to allow
users truely 'unlimited' storage.
On Tue, 12 Aug 2003 16:43:15 +0100

arithmetic

the

even

recalculated: -680288658
What can I do about it?

Sincerely,
Vladimir

#############################################################
This message is sent to you because you are subscribed to

--
Chris Demain <***@sagonet.com>
Sago Networks

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

Jason Mader

2003-08-12 20:06:43 UTC

Permalink

How should we recover from this?

15:38:17.14 2 SYSTEM account '***@ncac.gwu.edu' will be rescanned
15:38:17.14 1 ACCOUNT(***@ncac.gwu.edu) mailbox store size
recalculated: -1157455293

(deleting e-mail when this happens doesn't seem to work).

Post by Vladimir A. Butenko
When the account total storage exceeds 2GB, this counter gets the
arithmetic overflow, resulting in negative values. When CGatePro sees
a negative (or and other suspicious) value there, it puts an account
into the a special queue and recalculates all mailbox sizes in a
special thread, to restore the correct value. But this value is
"correct", so it ends up with a negative value again, and the account
is placed in that queue again and again.

---Jason Mader, FHWA/NHTSA National Crash Analysis Center,
The George Washington University, VA Campus

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

Jason Mader

2003-08-12 22:21:44 UTC

Permalink

http://www.stalker.com/WebGuide/Accounts.html

might be a good place to add documentation about the 2GB limit on
storage per Account.

Post by Chris Demain
http://www.stalker.com/WebGuide/Mailboxes.html#Formats

Jon Morby

2003-08-12 23:52:35 UTC

Permalink

Vladimir

This seems to be a major change, which is going to break a number of
long standing CGP users, let alone those who are trying to migrate in.

I was scheduling an upgrade to 4.1.1 this evening, however in light of
these changes I won't be doing so until this is cleared up.

I have a number of users (myself included) who have in excess of 2Gb of
mail in their account (using mdir format mailboxes)

Just deleting / archiving off to some random folder is not an option ...

What I fail to see is how having a large mailbox (5Gb in my personal
case) causes CGP to have any problems - or rather why going from 4.1 to
4.1.1 would cause this to happen.

Is my understanding that 4.1.1 and above will bounce any new incoming
mail to my account (and anyone else who has more than 2Gb in the entire
macnt (not just a single folder - this I understand - if it were mbox
format) with an over quota error, even though my / their accounts are
set as unlimited.

Please advise :(

I must confess I am a little concerned about the whole quota support in
CGP .. first we lose Domain quotas because they don't work, so they've
been removed (rather than fixed) .. and now this :(

Post by Vladimir A. Butenko
CommuniGate Pro has (and always had) a 32-bit signed counter for the
total storage used by each mailbox and by all mailboxes in the > account.
When the account total storage exceeds 2GB, this counter gets the
arithmetic overflow, resulting in negative values. When CGatePro sees
a negative (or and other suspicious) value there, it puts an account
into the a special queue and recalculates all mailbox sizes in a
special thread, to restore the correct value. But this value is
"correct", so it ends up with a negative value again, and the account
is placed in that queue again and again.
To avoid these things, the 4.1.x versions check if the account size
would exceed 2GB if a new message is added and report the "over quota"
error even if no quota is set in that Account settings.
Try not to exceed this limit. Remove the unused messages/mailboxes
(after archiving them). If absolutely necessary, move messages to a
different account (call it "archive", for example), and make those
account mailboxes available for you, so you can deal with them from
your own account.
We are thinking on how to remove the 2GB limit, but it cannot be
removed quickly.
On Mon, 11 Aug 2003 23:02:14 -0400

recalculated: -680288658

What can I do about it?

account is full (quota exceeded)
but we don't implement quotas! Every setting I can see is that he
has unlimited quotas. What's going on?
-rw-rw---- 1 root mail 617803762 Aug 11 19:59 INBOX.mbox
---Jason Mader, FHWA/NHTSA National Crash Analysis Center,
The George Washington University, VA Campus
#############################################################
This message is sent to you because you are subscribed to
To switch to the DIGEST mode, E-mail to
To switch to the INDEX mode, E-mail to

Sincerely,
Vladimir
#############################################################
This message is sent to you because you are subscribed to
To switch to the DIGEST mode, E-mail to
To switch to the INDEX mode, E-mail to

Communigate Forum

2003-08-13 09:23:30 UTC

Permalink

hi,

i'm in pretty much the same postition as jon, i've just bought 50 mapi seats
and want to upgarde to 4.1.1 to take advantage of a number of bug fixes but
i can't really do this and risk any interruption to the mail service. some
work round or tempory patch needs to be sorted asap to get this fixed..

cheers

rup

----- Original Message -----
From: "Jon Morby" <***@fido.net>
To: "CommuniGate Pro Discussions" <***@mail.stalker.com>
Sent: Wednesday, August 13, 2003 12:52 AM
Subject: Re: 2GB Account storage limit. Re: mail store size recalculated,
over and over...

Vladimir

This seems to be a major change, which is going to break a number of
long standing CGP users, let alone those who are trying to migrate in.

I was scheduling an upgrade to 4.1.1 this evening, however in light of
these changes I won't be doing so until this is cleared up.

I have a number of users (myself included) who have in excess of 2Gb of
mail in their account (using mdir format mailboxes)

Just deleting / archiving off to some random folder is not an option ...

What I fail to see is how having a large mailbox (5Gb in my personal
case) causes CGP to have any problems - or rather why going from 4.1 to
4.1.1 would cause this to happen.

Is my understanding that 4.1.1 and above will bounce any new incoming
mail to my account (and anyone else who has more than 2Gb in the entire
macnt (not just a single folder - this I understand - if it were mbox
format) with an over quota error, even though my / their accounts are
set as unlimited.

Please advise :(

I must confess I am a little concerned about the whole quota support in
CGP .. first we lose Domain quotas because they don't work, so they've
been removed (rather than fixed) .. and now this :(

recalculated: -680288658

What can I do about it?

account is full (quota exceeded)
but we don't implement quotas! Every setting I can see is that he
has unlimited quotas. What's going on?
-rw-rw---- 1 root mail 617803762 Aug 11 19:59 INBOX.mbox
---Jason Mader, FHWA/NHTSA National Crash Analysis Center,
The George Washington University, VA Campus
#############################################################
This message is sent to you because you are subscribed to
To switch to the DIGEST mode, E-mail to
To switch to the INDEX mode, E-mail to

--
Get your redmail.com email address for just £9.99 a year! Includes
10Mb, POP/IMAP/Webmail (with calendaring) Virus Scanning and Spam
Filtering
Sign up today - http://www.redmail.com/?tag-signup

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

**************************************************************
Scanned by MailScan Anti-Virus and Content Security Software.
Visit http://www.mwti.net for more info on eScan and MailScan.
**************************************************************

**************************************************************
Scanned by MailScan Anti-Virus and Content Security Software.
Visit http://www.mwti.net for more info on eScan and MailScan.
**************************************************************

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

Aaron

2003-08-13 00:09:46 UTC

Permalink

Post by Jon Morby
This seems to be a major change, which is going to break a
number of long standing CGP users, let alone those who are
trying to migrate in.
I was scheduling an upgrade to 4.1.1 this evening, however in
light of these changes I won't be doing so until this is cleared up.
I have a number of users (myself included) who have in excess
of 2Gb of mail in their account (using mdir format mailboxes)

Hmm... It sounded to me like the previous behaviour would cause a nasty loop
in some internal processes, and 4.1.1 just fixes that by setting a limit to
prevent any looping.

So if it works for you now with 4.1, you don't have the nasty looping
problem mentioned? So the fix may not even apply to you.

Or more to the point, if you have a working system now, why would you
upgrade anyway? :)

It just sounded less like a "major change" and more like a "major bug fix"
to me, but then I don't have any users with over 2GB in a single mailbox so
I don't know how much that "recalculating" loop would affect system
performance. Maybe it wasn't too bad and the cure here is worse than the
disease.

Perhaps Stalker should just have the recalculate thing not even bother
figuring out the size of any mailbox over 2GB, so we'd all be happy in the
short term.

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

Dmitry Valdov

2003-08-13 18:32:24 UTC

Permalink

Post by Vladimir A. Butenko
CommuniGate Pro has (and always had) a 32-bit signed counter for the total
storage used by each mailbox and by all mailboxes in the account.

Why signed?
--
Dmitry Valdov mailto:***@dv.ru
CCNP

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

Chris Ferebee

2003-08-14 08:59:38 UTC

Permalink

Vladimir,

We are running CGP 4.1b9, and my main personal account directory is
approx. 3 GB.

The Log is indeed full of "mailbox store size recalculated:" messages,
which I had previously disregarded. While they are Level 1, the wording
is pretty innocuous.

(Maybe the continual recalculation has to do with some performance
problems I have been seeing?)

--> Interestingly, mail to my account does NOT bounce, contrary to
what you wrote above.

Is the over-quota bounce for >2GB accounts a new "feature" of 4.1.0-final?

If so, I think we will have to stay with 4.1b9 until the issue is resolved.

Sure, I could put my archived email in a second account, but I bought
CGP to make my life easier, not harder. However, the real problem is
with customers.

We have several customers who want to archive all their email on our
server, and are happy to pay for unlimited storage. They are not
interested in workarounds.

So, if 4.1.x does in fact bounce mail for accounts > 2GB:

May I respectfully suggest an option to turn the entire quota system off
(until it can be fixed), if that is what it takes to avoid
inappropriate bounces?

Sincerely,
Chris Ferebee

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

Chris Ferebee

2003-08-15 12:07:48 UTC

Permalink

Post by Chris Ferebee

Post by Vladimir A. Butenko
To avoid these things, the 4.1.x versions check if the account size
would exceed 2GB if a new message is added and report the "over
quota" error even if no quota is set in that Account settings.

We are running CGP 4.1b9, and my main personal account directory is
approx. 3 GB.
--> Interestingly, mail to my account does NOT bounce, contrary to
what you wrote above.

After thinking about this, I have come to the conclusion that perhaps my
mail is not bouncing because I took my account over 2 GB with CG? 3.x,
and the whole time I have been running 4.1b9 my account size has been
"negative", thus my "account size would" NOT "exceed 2GB if a new
message is added".

Still hoping for a fix/workaround/kludge ASAP...

Chris Ferebee

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

w fm3

2003-08-14 20:23:30 UTC

Permalink

I know the recalcuating when a mailbox was > 2GB really hit server
performance for me.

Not a nice suprise.

Its 2003 business NEEDS more than 2GB of mail storage / account.

The current solutions of either:

1. getting people to accept both a normal account and an archive account
2. switching to a 64bit platform

are cost prohibitive

I have a number of users reaching towards the 2GB mark and I am surely not
the only one that will have to consider switching mail servers if there
isn't a workaround soon.

Ok end of grump.. ;)

_________________________________________________________________
Add photos to your messages with MSN 8. Get 2 months FREE*.
http://join.msn.com/?page=features/featuredemail

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>

Aaron

2003-08-14 21:15:01 UTC

Permalink

Post by w fm3
Its 2003 business NEEDS more than 2GB of mail storage / account.
1. getting people to accept both a normal account and an
archive account 2. switching to a 64bit platform

It shouldn't be as bad as all that (switching to a 64 bit platform). True,
most systems are 32-bit, but that doesn't mean you can't work with big
numbers. Intel platforms can handle 80-bit numbers natively using the FPU,
and using your own routines there's really no limit.

It does seem short-sighted on Stalker's part to not foresee a day when
people have over 2GB in a single account, but then at one time a certain
famous billionaire once thought nobody would ever need more than 640K of
RAM.

I can't figure out why Stalker's programmers couldn't just make that change
in the calc routines very easily. Or at least just put in a kludge to
ignore any current calculations that result in "negative" numbers (high bit
set) when the quota is "unlimited". Then they can implement a proper
routine at their leisure.

Doing a workaround means people get their big mailboxes, servers don't
continuously burn cycles recalculating all the time... Goodness knows I've
written my fair sure of bulky workarounds as a short term way to get
something working.

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>