Archiving email question...Important [Archive]

tarballed

October 28th, 2003, 21:16

Hello everyone.
Well, I was hit with a bombshell today at work and was not very happy about it.

I was told today that they want to archive our email. This is no problem, but the way they want to do it is very weird.

Like I said, we are running postfix and courier-imap. I had drafted up directions to teach users on how to create 'local' folders to store any email they wanted to keep. I believe that Corporate mail servers shouldn't be a place to store 'personal' emails, only Company/Important emails.

With that in mind, here is what they want to do.
Using Netscape or Mozilla, they want to be able to create a folder to holder user's email, and store it on a completely different server. They dont want any email stored locally on the users machine. Does that make sense?

So basically, if I have rack1 which is the mail server and rack6789 which is going to be a 'special' server to hold users email (all of it), that is what they want to do.

In thinking about this, this means I need to setup an additional server running samba, in order to be able to allow users to store email on that server.

Anyone care to comment on this?
Any suggestions?
Anyone wanna flame this idea? Go right ahead

What about an Open Source email archiving piece of software?

I need to think about this one, but im more pissed that they just sprung this on me than anything else.

Tarballed

elmore

October 28th, 2003, 21:41

tarballed

October 28th, 2003, 22:50

Actually all you beed to do is use imap and use Maildir/ style delivery an in imaps configuration disallow mail removal. Then just setup a second server and have your mail server nfs mount that second server on /home

Oh man, brillant!!

I will definitely take a look at this.
So I can setup, in imap, to disallow mail removal?
If im going to use NFS, I need to install and configure NFS on both servers, correct?

What about authentication and such? Possible to some how only create users on one server and use a central type place to hold all accounts?

Thanks elmore...this gives me more options.

Tarballed

tarballed

October 29th, 2003, 12:36

I'm going to be working on this today. I may stop in a bounce some questions in here...

As it stands right now, I have an idea of how to set this up, but there are a few parts im not sure about.

I'll keep updating here.

Tarballed

tarballed

October 29th, 2003, 13:52

Here is a thought:

My PDC running Samba and LDAP is setup to automatically map user's /home directories to a drive.

What about having the users email stored in their /home/$USERS partition?

Good idea? Bad idea?

Im thinking time line here and im not sure how long they are going to give me.

Tarballed

tarballed

October 29th, 2003, 14:37

Can someone fill me in and tell me what a NetApp is? I keep seeing this mentioned, but not sure exactly what it is.

This can be used in conjunction with NFS?

Thanks.

Tarballed

tarballed

October 29th, 2003, 14:40

N/M

http://www.netapp.com

:oops:

tarballed

October 29th, 2003, 17:47

Wish me Luck!! Im trying to push a netapp to management!!

Pray for me!!!

:)

Tarballed

tarballed

October 30th, 2003, 13:56

Im back everyone, looking for some feedback here.

Elmore had a good suggestion about using NFS for this particular setup.
What I need is some additional info to convince management on the best way to do it.

With that in mind, i've thought of numerous ideas that I could do to set this up, but most require awhile to setup and test fully. Problem is, I dont have that luxury.

With that in mind, I have my main mail server to be used. I'm thinking I will need a second server to be used for the NFS part. It does not have to be anything fancy, but will require a lot of disk space.

Can I get some more feedback on how to implement this? Using NFS? Setting up sorta speak.

I need to get some documentation to show to management that this is the best and stablest way to do what they are requiring.

I look forward to some feedback.

T.

elmore

October 30th, 2003, 14:34

Hmmm tarballed,

You have the same problem that I have. You over think and try to re-create the wheel. It's not that you don't get things done and do a good job because you do. You just need to slow down. Yes I know you're in a rush, I know management is breathing down your neck etc. etc. Take a step back and breathe though man. Look at what will be best, decide on one thing and do it. Don't get bogged down in every little detail. Don't get frustrated and most important be honest with the management that exists in your company.

Don't get bogged down in evey little detail -

In other words get a clear picture of what it is you need to accomplish. I see a lot of postings from you, you get bogged down. Go to your management team and get a clear scope of the projects that they want you to achieve. Prioritize each project, look for obvious solutions, then investigate the top three solutions and get into the details of each one. From there find which one best suits your needs. Take a breathe, sleep on it, re-evaluate, take another breath, sleep on it, set it up in a test environment, take another breath, sleep on it, put it into production. Next project please.

Don't get frustrated -

You can only do what you can do. I know you have a lot of things going on, in particular with the fires. With regards to your job, don;t get frustrated with one thing. If a problem or technical detail is getting to you, for god sakes man, walk away. Clear your head, take a breath, sleep on it, come back and work on it the next day. No reasonable management team will insist you know everything about everything when it comes to computers, your thought process is what counts the most. Your ability to solve problems well. Concentrate on that, quickness and efficiency come later on, the ability to solve a problem well is what counts the most.

Be honest with Management -

If the C.E.O comes to you and says I need you to do A. B. and C. by 5:00 today and you know B. and C. will take two days, then tell the C.E.O what he's asked for cannot be accomplished in the given time frame, management appreciates honesty, it sounds to me as if management at your company has not allocated enough time for these projects. You need to give them an accurate time table. Remember, Work that is rushed is very rarely done 100% right. If anything slightly overestimate the time you need. Don't go wild but if you know a project is going to take about 5 full days to implement, 2 full days to test then estimate 6 days to implement 3 days to test, this allows you more time a little more breathing room, and most importantly time to recover when something goes wrong, and something always goes wrong when working under a tight deadline with no breathing room, in other words expect the unexpected and plan for it.

Don't be afraid to speak your mind, intellectual honesty with your peers and management staff is always valued, stand up for yourself and defend your arguements in particular when it comes to time required. If done correctly you'll find yourself very well respected in your organization.

P.S. - If you can get a netapp go with it. They are badass, I've maintained several along with krusty and minion. The best NAS device on the market!

Hope that helps.

tarballed

October 30th, 2003, 15:15

You have the same problem that I have. You over think and try to re-create the wheel.

Alright, we have something in common besides computers! :)

I appreciate your kind words elmore. That was spoken like a true, veteran Sys Admin. I really appreciate your insight and wisdom.

I started to do exactly what you suggested, right before I read your post. I even smoked a cigarette...

Your right though. Management is sort all over the place and I need to pretty much be honest and say, what is it that you want. I need to know, and be honest.

Well, I did. I went in and spoke with Management and laid it out for them in simple terms so they could get the big picture. I was cool and polite, not condescending at all.

With that in mind, I found out more clarification.
Their main concern with email is this:

With IMAP, the mail stays on the server (unless copied elsewhere). Their fear is that when we start opening branch offices, if the pipe were to go down between the branch office and the corporate office, nobody would be able to read their email. So they want a way for users to be able to still read the email that they were able to copy/move/pull down before the pipe went down.

So in essence, a copy of all the email that the user can view, just in case.

I've been researching and brainstorming, gathering ideas and bouncing ideas off of other people.

I've pretty much narrowed it down to a following options: (Feel free to add other ideas as well)

1.) Since they will be using Mozilla and Netscape clients, so im wondering if there is a way to setup the client to automatically copy the emails that arrive to a local folder.

2.) A NetApp appliance (which I would like to do) but I dont think it is an option due to its price tag.

3.) Setup a second server of some sort (either NFS, second IMAP server or some other type) that can be used to map users local folders in the mail client to the server, for storing and viewing purposes.

4.) Something that was suggested was going with: http://webbasedemail.com . It's turnkey, fast to get up and running and has a lot of options that management may like.

That's what i've come up with so far.
I'm sure there are more ideas. If anyone has further suggestions, i'm all ears.

I would like to explore the idea of NFS as I think that could be a very viable solution.

Thanks elmore again and to everyone else who has helped me in the past.

Best,

Tarballed

elmore

October 30th, 2003, 15:28

tarballed

October 30th, 2003, 15:47

Well I'd do this. Setup your main mail gateway, have that mail server pass off all mail to your intnernal IMAP/SMTP computer and setup a user access list to punt mail to the branch offices. This requires a mail server at the branch offices but it stays consistent to what you initially wanted and also provisions for managements concerns. BTW, most multisite corporations set mail servers up in this fashion and I know it works well with courier-imap, postfix and FreeBSD 'cause that's what krusty, minion and myself setup when we worked together. Essentially what you end up with here is a hub and spoke configuration that scales well and easily allows for multiple hubs when it turns into the enterprise.

I like this idea. ( I love the term punt...i got a laugh out of that :) )

That should resolve the problem with redundancy, correct? Basically, everyone would have a email server at the branch offices so even if the pipe went down, they would have a copy of their emails on each mail server.

But the other thing they want me to look at is, what if the server goes down? I still think they best thing to do would be to copy messages locally to the machine. Thoughts?

This is good. I have the Blum book and read it pretty nicely. Now I need to re-read.

Did you guys setup mail gateways and mail servers at each location for your branch offices? What did you guys setup for sending wise if I might ask?

Thanks elmore.

Tarballed

elmore

October 30th, 2003, 16:32

That should resolve the problem with redundancy, correct? Basically, everyone would have a email server at the branch offices so even if the pipe went down, they would have a copy of their emails on each mail server.

Yup that's right!

But the other thing they want me to look at is, what if the server goes down? I still think they best thing to do would be to copy messages locally to the machine. Thoughts?

Yeah, setup a vpn and run amd with nfs then setup the hub as a secondary mail server. This way would allow all users to use the local mail server at whatever office they are at. Careful though if a user has a large INBOX it won't work very well. Might need to initiate mail quotas which is a good thing. Also forcing mail to be archived is good, and checkout subscriptions with IMAP. This type of setup lends itself wonderfully to a public webmail based service later on. See how careful thought and planning lead to scalable solutions.

Did you guys setup mail gateways and mail servers at each location for your branch offices? What did you guys setup for sending wise if I might ask?

Yes but we had sysadmins at each location. If I were you I'd do it differently. I'd force all outgoing mail to leave through your main hub. This allows for centralized virus scanning, content filtering etc. to take place on one machine. Which means you'll have less to update and administer.

Thanks, elmore

That's what I'm here for.

tarballed

October 30th, 2003, 16:59

Yeah, setup a vpn and run amd with nfs then setup the hub as a secondary mail server. This way would allow all users to use the local mail server at whatever office they are at. Careful though if a user has a large INBOX it won't work very well. Might need to initiate mail quotas which is a good thing. Also forcing mail to be archived is good, and checkout subscriptions with IMAP. This type of setup lends itself wonderfully to a public webmail based service later on. See how careful thought and planning lead to scalable solutions.

Ok. The company is going with Firewalls from one company (Watchguard) that will be installed at all offices. From what I know, they come with built in VPN managers that I can seutp between offices. That should solve the VPN issue.

I get the idea of what you are saying:
Setup a Main Mail server at the Corporate office. Have it 'punt' email to the branch offices as needed.
This will require a postfix setup at each branch office (Hub and spoke).
Now, users can get their email from the branch offices mail server.
If the pipe goes down or corporate mail server goes down, they can still view the email they received earlier, but will have to wait until either the pipe comes back up or the mail server is backup, correct?

As far as the VPN with NFS setup, ya lost me there. I dont follow really.

Mail quotas, definitely. Already setup the box to have user and group quotas.

I'm planning on using always_bcc to a specific user. Then, each night, compress all the email that was received and ftp it off to another server for storage. Then, if I can, i'd like to burn the data to a CD-ROM for storage. Thus, I can pop a CD in whenever I need to review a certain week's email.

Webmail is where I want to head to, so yes, great scalable solutions.

Thanks again elmore.

Tarballed

tarballed

October 30th, 2003, 21:10

You know, I did some research and I found a project that would do exactly what they want to setup here:

Project Cyrus with Murder

Reading over the web site, it seems to do everything managment has requested.

The only problem, thats a lenghty project in terms of setting up, installing, testing etc...

Just thought i'd share this.

Tarballed

tarballed

October 31st, 2003, 13:56

Yes but we had sysadmins at each location. If I were you I'd do it differently. I'd force all outgoing mail to leave through your main hub. This allows for centralized virus scanning, content filtering etc. to take place on one machine. Which means you'll have less to update and administer.

Hmm. Yes. That make sense. Once place to send out mail through. It will take some tweaking I would imagine and I will need to lock down postfix to ensure only certain people can send email through.

You said the Blum book has a pretty good explanation? I'll check it out.

Right now, I think that they only viable solution is the hub and spokes setup. I dont see any other way to do it.

It may be a tricky setup, but it can be done.

Tarballed

tarballed

December 4th, 2003, 14:07

Thought i'd give an update on this...

I recently received a promotion at my work that gave me more power, flexability and say in what is best for the company. Needless to say, there is a lot of less crap I have to go through to get things done, correctly. No more nonsense ideas of stuff that is so ass backwards, it makes you crazy...

Anyway, with that in mind, I am going to change a few things down the road here.
As it stands right now, we are using our mail server for smtp only for the moment, and will eventually switch over entirely. Eventually, I am going to replace this server with a FreeBSD box running postfix and cyrus-imap (looking forward to this one)

Anyway, I wanted to ask everyone here if they knew of a good way to archive email? What I mean by this is, I want to make a copy of all email that is sent and received through our mail server. I want to then be able to archive these emails in a way that I can archive them by date and month so it will be easy to go back and find a specific email if we have too.

Anyone know of a piece of software that can do this? Third party of open source maybe?

Thanks.

Tarballed