wouldnt a server that operates smtp mail throttling also place restrictions on how often you can run a cron?

Probably. But there're are more reasons to support a job queue than just response times. In the case of mail sending, failed jobs can be retried and presumably we could more easily wire up logging tools in the CLI environment than over web.

Things like this are exactly why the CLI console package is integrated into 4.0. It exposes the possibility for core and extension builders to build better tooling.

@brianteeman That's one reason it should be an optional configuration - for servers/hosts that aren't capable.

Other reasons include, admins who don't know how to initiate a CRON job.

For systems that throttle SMTP, this should be able to accommodate. As each message is sent, it's removed from the queue. If the cron hits a throttle, the mail isn't sent and the message can remain in queue until the next run.

@mbabker - that was one of my reasons above - temporary errors could initiate a retry delay.

I use CLI for all sorts of stuff. It's great to have that capability.

@mbabker - I'm glad you mentioned logging - because this could be used to log and report on outgoing messages and smtp performance.

With some quick research, I found that PHPMailer has a facility to output a message unsent as a complete RFC822 message.

$mailer->preSend();
$message = $mailer->getSentMIMEMessage();

Looks like the mailSend()/smtpSend()/sendmailSend() methods only require a $header and $body - which is easily obtainable via $mailer->createHeader() and $mailer->createBody() without the need to run $mailer->preSend(). So 2 database fields and we have everything we need to send a message later.

Some testing reveals that intercepting a message in Mail::Send() and storing it in the database is pretty easy to do.

I know a guy who’s written a mailbank type component/plugin and planned on releasing it but never did.

It was also something I wanted to add to the Joomla! Core. Allowing a log of all emails sent/not sent would be very useful and having a single cron/mailer script but also be useful, I.e. addtoque method instead of just send.

My thoughts would be to modify JMail to queue if configured, otherwise send normally. That way it's transparent to extension developers - they send normally and the system handles the message (queue or send)

Just adding some thoughts: my hosting provider has a limit on how many emails I can sent per hour.
Currently I run the risk of mails not being sent because that limit was reached (I learned the hard way via mailing all users via the mass mail function :O)
It would be cool if the mail queue could be configured to these provider's thresholds

I've got a CLI that implements limiting while verifying weblinks - it loads a configured maximum so as to not bog down the server. It's an integer field in a plugin config - super easy to do.

So, it stands to reason there should be a management component to hold configurations like this. I think it would be prudent to come up with a list of things the management app should do/track as that will help to define what the database should look like.

Off the top of my head, each mail record should include:

1. user id of the user sending the mail (0 for automated emails)
   extension sending the email (com_users, com_contact, plg_system_whatever, etc)
2. message priority (some core components are priority zero - most important, others can be optionally configured as 1 and higher - least important)
3. DateTime the message was queued
4. DateTime of the last send attempt (defaults to zeros)
5. DateTime the message was successfully sent
6. Number of attempts (JSON array of DateTime?)
7. header and body (it's what PHPMailer can natively output and accept as input to send later)
8. message id (checksum of the header maybe?)

Configurations for the CLI:
max messages per time period - max and time period both configurable
max messages per CLI run
max retries per message
retry interval
failed message recipient(s)

The success DateTime is used to determine the number of messages sent for the configured max/time period. Once the messages are no longer needed, they would be discarded (or not? why?)

Some messages need to be purged - like new user emails, because they contain a password in the clear. Password reset are one-time-use messages. Forgot username might become invalid if the user changes their name.... maybe a classification for sensitive messages that need to be purged.....possibly based on the sender extension? Like, anything coming from com_user should be purged upon send, to protect potentially sensitive information.

Maybe that's another configuration option, selecting which extensions emails are archived, while the rest are purged.

Just incase this has not been covered I would have a Global Joomla email cron/component where all 3rd parties can send their emails to the global que only if they choose that because some emails need to be sent out immediately.

This que could then control the send rate of all emails which and also give statistics on email volume and sources which is very much needed when running on shared hosting (i.e. most of us amateurs).

Developers would then not need to write their own crons and run the risk of us end users still overloading our servers email system. Consider you have easyblog with lots of emails being sent out and kunena with lots of emails being sent out by their seperate crons you have to work out and estimate totals from boths systems and make sure the combined lists dont go over your total or your accout will get suspended etc.., this can get tricky and requires constant revisits if you nare getting towards your max limit.

Most social components want to send emails nowadays.

Any thoughts on if this is a change in Joomla Mailer class, or something extension developers need to implement?
I use e.g. Acymailing to sent newsletters (okay to be queued), but also to sent out the Joomla notification (new user, reset password etc.) these should not be queued / sent out immediately.
When the Mailer class 'catches' and queues all mail, then there should also be a way to tell that class what to do with what emails.

@stutteringp0et It would be a great feature to add to Joomla. I have implemented a mail queue in my own component that deletes obsolete users. From my experience this works fine using settings to fine-tune to server specs.

The idea that I had regarding queuing was that the site administrator could turn it on/off globally. There would be a priority classification for emails like registration, password resets and that sort to be sent (virtually) immediately - at least, first in the next run (which would probably be once per-minute, as this is the smallest increment that CRON can handle)

Storing passwords in queue would be necessary until the email is sent - and then those messages would be deleted (or have their text replaced with a SENSITIVE DATA REMOVED message). I think that will be handled by the priority. Maybe something like 0 for sensitive, 1 for system, then everything else would be a priority 2. Sensitive emails get processed out of queue first, so the sensitive data doesn't remain in the database any longer than necessary. System goes next (lost password/username, system messages and stuff), then everything else.

I just got past a major deployment, and this is next on my list of things to do. I'll build a prototype which addresses these things and let you guys hammer on it.

Just so you understand where I'm at on this - I've got an extension which sends a half dozen emails on certain visitor generated events. It sucks that the user sits there for 5-10 seconds while the system sends emails. I thought about building a message queue into my extension, then I thought it might be more beneficial to the community if Joomla had its own queuing system. I figure, if I need a queuing system once, I'll probably need it again - and I'd rather build it only once.

@Ruud68 it would be a very minor change to the jmailer class. The modification would be to detect if the queuing system was enabled, and if so queue the message - otherwise it would send immediately. The modification would be found in the send() method (probably), because at that point - the message has been built and could be directed into the database easily.

You can't have it both ways (push email data into a queue and not store that data in some way that can be read). Either some email types are just sent without touching the queue at all (and you don't have to worry about data management) or it goes into the queue and everything going into the queue is handled the same way.

Putting something in a queue means it is scheduled to be done at a later time. If you're putting already complete jobs into the queue, then it defeats the point of a queue.

I think we're getting some wires crossed here. A job queue system doesn't really need to retain data about successful jobs (you can use other tools like logging to keep records of that), the queue system just needs to know what jobs to run and at what priority with what data at its simplest definition. Especially consider a highly active system where you wouldn't want a database table to be bloated with thousands of extra records that at that point serve as nothing but a logging system. This means a mail's content is immediately removed from storage as soon as the job for that message succeeds (if you're storing that mail content in a log, it's the API that's handling logging that's responsible for sanitizing it).

This is why for example Laravel's mailing API has two sending mechanisms, one to queue a message (which pushes the sending task into its native job queuing system) and one that immediately sends the message without touching the queue.

Someone had previously suggested an option to keep sent messages, a sort of running communication record. That's why there was going to be a method to determine sensitive messages which should not be stored.

Make it a supplemental logging system. What we are talking about here right now has IMO three distinct pieces:

• A general job queue management system
• A specific implementation surrounding email traffic
• Transaction logging for email traffic

For a general job queue, would it be better to go with something already established and implement a worker interface in a Joomla CLI? I was only interested in a mail queue, but if a general job queue has a better chance of core adoption - I'm game.... I just wonder if it's necessary to re-invent the wheel when there are existing (and mature) job queue systems that could be interfaced.

Either some email types are just sent without touching the queue at all (and you don't have to worry about data management) or it goes into the queue and everything going into the queue is handled the same way.

My hoster has an email limit per hour (siteground). Emails sent directly are counted by the hoster as are emails from the queue. So if I limit the queue to eg 400 per hour and mails are also send directly (not via queue) then these mails should also be counted for the mail queue limit.

That is why I think all emails should be 'routed' via the queue where direct emails are sent out immediately (without cron), but are counted by the queue in the configured limits.

As mails send from a domain per hour is becoming a relevant spam indicator, I think it is only a matter of time before all hosting companies apply a limit of some sort: shared hosting or not.

Again, you are talking specific implementation details for one specific problem.

The first thing Joomla needs is a job queuing system, in general. Not a solution specific for one task. Otherwise it really defeats the purpose of introducing such a system.

A job queuing system is not a transaction log. So speaking as a general job queuing system, no, it should not have a full transaction log where every action is recorded, to include things that are not even dispatched through the queue system.

So now you get into the mail specific details. This is where you get into the issue of having per-hour sending limits, transaction logging, and whatnot.

The mail system needs to be aware of the per-hour sending limit and keep track of this regardless of if a mail is sent as part of the job queuing system or directly sent without being queued. This specific feature is NOT a part of a job queue system and should not be integrated into one.

You are also looking for a transaction log which can store sent messages (and optionally sanitize their contents). This transaction log is a separate feature from the mail sending limitations AND a job queue system and should be handled separately.

Yes, all three of these things might need to be able to work with each other, but step back and look at it from a high level perspective and you will see that you have three distinctive (major) feature requests in this thread and each of them needs a proper implementation.

For a general job queue, would it be better to go with something already established and implement a worker interface in a Joomla CLI?

Yes. Let's not re-invent the wheel (I just finished this exercise for a project because the solution I needed only existed as a flawed extension and I had to re-implement it without all the flaws).

I was only interested in a mail queue, but if a general job queue has a better chance of core adoption - I'm game

I'm looking at things at a higher level. A job queue has use beyond just pushing email traffic out. If we're going to have a system that's enqueuing jobs to be run through another worker system (i.e. cron) then the high level stuff might as well be generic and usable for other tasks even if core itself isn't doing anything but queuing emails.

I think I can design something to fit this. I read about someone using redis as a job queue, and since J has a redis session store now, it seems like a likely candidate.

I'll do that research and come back with a plan for discussion. I'll frame the plan around the email implementation, but will make sure it's generic enough to handle any kind of queued task.

Depending how in depth we want to go, a few packages which are job queue systems in various contexts:

• https://github.com/illuminate/queue
• https://github.com/schmittjoh/JMSJobQueueBundle
• https://github.com/chrisboulton/php-resque

@tonypartridge - There isn't any reason for a registration email to take an hour to send out if routed through a job queue unless the queue worker can't keep up with the volume - in which case, you could launch a worker on another system. The system I envisioned for processing email would run via CRON every minute, delaying email up to one minute. If that isn't fast enough, I'm sure a clever person could write a daemon shell script that loops, listening for a queue event signal and runs jobs immediately.

@mbabker - Resque is the redis implementation I was reading about last night.

Resque might be OK, but just keep in mind we probably shouldn't lock the implementation to those having access to Redis (if you look at Laravel's system it has an abstraction layer for different connections such as Redis, a local database, and some other common system level integrations).

via CRON every minute

Hosts with restrictive mail limits are likely to have restrictive cron limits as well

@brianteeman - The plan was to make it optional.

@mbabker - illuminate it is - if queue system options are desired.

@stutteringp0et any progress on this or is it an abandoned idea

Someone did it in a 3rd party extension

@stutteringp0et can this be closed?

Yes, sorry - I was out with my family.

For anyone interested, the extension that achieves this is:
https://extensions.joomla.org/extension/jmail-queue/

This should be part of the core/framework, not via another 3rd party extension.

Yes it should and I am disgusted that @stutterinp0et came here, solicited support, advice and guidance and then released this as an extension

Thanks for coming to my defense @tonypartridge

@brianteeman - I would never do that. My name is Michael Richey - the author of the extension is Alex Chartier.

My apologies. It was just that you said you were working on it and then you posted a link to an expensive extension on the jed and closed this 2+2 !=4

Please delete above comment, it is spam.