Cope with removes overtaking writes

Merging default into ug 23142

Merging default into bug23133

fixed a small empty-set bug

deleting a queue causes all of its messages to be puback'd

confirm transient messages only after basic.return

Don't ever keep the recovery process waiting, regardless of whether the queue is going down or not

A rather crucial infinity missing

Abstract refcount == 0 to imply not_found for appropriate call sites

By this stage of the client-read, we know a GC can't start and can't be in progress. Thus even if the refcount is 0, we're still safe to read it. What's more, this area of code has never made provision for the index_lookup to return a not_found - it will crash if that happens. Thus if the implicit assumption is that the message must exist, then by the same token, we know the refcount will always be > 0

Abstraction of the inner write function

Merge in bug23145 (put plugin expand dir in db dir)

cosmetic

Because we can increment the refcount back above 0, the calculation of the contiguous top needs to be contiguousTop + totalSize, because the validtotalsize can contain data that's well beyond the old contiguous top

Eliminate RABBITMQ_PLUGINS_EXPAND_DIR

There were a number of issues with RABBITMQ_PLUGINS_EXPAND_DIR:

- It was undocumented in the context of the generic unix package, and
if unwisely set could do an effective "rm -rf" in an unintended
location.

- It did not take account of the possibility that multiple nodes could
be starting at once, and so doing plugins activation
simultanteously.

Instead, use RABBITMQ_MNESIA_DIR/plugins-scratch. This avoids the
need to extend the generic unix package documentation, the location is
node-specific, and the distinctive plugins-scratch subdirectory
reduces the risk of unintended file deletions.

Merge default into bug 23133

Backed out changeset c47670da4830 - I decided the dedup cache shouldn't be mangled with.

merge bug23142 into default

Minor tweaks

eliminate duplicate write messages completely, and simply rely on the pending write count in the cur ets file cache to indicate the delta to the refcount

told dialyzer to check for more things

As Matthew suggested, I've added {wranings, [underspecs, behaiviours,
race_conditions]} to dialyzer's options. As a result, we now have 56
warnings instead of just the 4. Most of them are underspecs (i.e. we
say a function returns any() but dialyzer's figured out that's overly
general).

small simplifying refactor

take current memory alarms status into account straight away
This does in fact not alter the behaviour at all due to the following:

- if the alarm is active the alarm registration will call the handler
straight way, which will send a {conserve_memory, true} message to the
reader.
- the reply to the alarm registration is sent after that, and from the
same process - the alarm process - so by the time the reader loops
around the mainloop again the {conserve_memory, true} message is
guaranteed to be in the mailbox
- on looping around, the reader request a frame_header from the
socket. The reader has already sent connection.open_ok to the client,
and the client may have started sending commands straight away. But
all the reader is going to see of that to start with is an {inet_async,
...} message for a frame_header. That is guaranteed to end up in the
mailbox after the {conserve_memory, true} message.

Thus the reader is guaranteed to process the {conserve_memory, true}
message before handling any more data from the socket.

With this change it is rather more obvious that the memory alarm
status gets taken into account before any more client data is
processed.

treat the dedup cache the same way as the index - permit 0 ref_counts and prune on file roll

Merging default into bug 15930

Merging default into bug 23133

The solution is very simple: In the case where the fhc sends out requests to close file handles, the clients might respond very quickly. The fhc will then gather these responses (say, just updates, not closes) and then will sit there for 2 seconds until the timer goes off. Thus the solution is just to subtract the timer period from the calculated average: i.e. the expression is to say 'close file handles that haven't been used for N seconds from NOW' rather than the previous 'close file handles that haven't been used for N seconds from NOW - 2 seconds ago'. This works very nicely and whilst the fhc can get quite busy when there are more users of file handles than there are file handles available, that is hardly surprising, and the fact is starvation is prevented and processes are promptly serviced

Track blocked pids explicitly

refactoring

publisher acks for persistent messages

Acknowledgements are sent to the producer in the following cases
(assuming the channel in confirm mode):
- a transient message has been received (i.e. it is ack'd
immediately),
- a basic return has been sent (the ack is sent immediately
afterwards),
- a persistent message has been consumed [by a consumer] or got [via
basic.get] and ack'd by the client (or the retrieval was no-ack),
- a persistent message has been written to disk (both the message
and its index).

At least in theory, every message published on a confirm channel will
at some point be ack'd. A message is only ack'd once and the reason
it was ack'd is hidden.

queue_index keeps track of which messages have been published but not written to disk

There are 2 ways queue_index writes something to disk:
- the journal is synced, and
- the journal is flushed (i.e. scattered across many files and those
files closed).

Since queue_index is the same process as variable_queue, we just keep
a list of guids that have been published and we just call an on_sync
handler whenever these are written to disk.

persistent messages are ack'd only after the consumer acks it

move msg_seq_no from #basic_message{} to #delivery{}

Correct monitoring and actions upon DOWN messages. Note this is especially subtle for obtains, which effectively implicitly allocates temporarily to the blocked caller (FromPid) whilst monitoring it, and then transfers this to the ForPid when possible. Note the ForPid can die before the obtains is processed, which which point the FromPid must be replied to immediately.

merge bug23135 into default

obtain_and_release_on_death => obtain, and minor refactor of tcp acceptor

Combine obtains and release_on_death

bug22906 merged into junk

merge bug23132 into default

Minor corrections

first stab at keeping track of when messages are synced to disk

refactoring

PubAcks are de-dupped in rabbit_channel

cosmetic

tweak

tweak

tweak

obtain_limit can be 'infinity'

cosmetic

refactor: incorporate limit checking into maybe_reduce

cosmetic

opens => open. obtains => obtain

Finally, it works. Important points: 1) remove elders from blocked clients so that we calculate average based on pids that can actually respond. 2) reduce whenever we have pending_opens to ensure we rotate the strike.

Whoops. Be consistent about comparisons

Sockets have priority up to 90%. Processes have the full 100% but at lower priority.

Make sure tests have enough file descriptors

Whoops

Split obtains in two : one for open requests (higher priority) and one for obtains requests (lower priority, and a lower limit)

Allow the fhc server to request the client closes file handles before granting an open - thus prevents deadlocks

Merging default into bug 23132

bug23129

minor refactor

cosmetic

Permit 0 ref counts and don't actually forget about messages until the file gets removed

don't log file summary recover error
the details aren't that important and get in the way of a fresh
startup

don't attempt to recover file summary when we know there isn't one

cosmetic

Obtain from the fhc sync, and obtain it before we actually open the file

more refactoring

yet more refactoring

now it compiles

minor refactor

cosmetic

minor refactor

Refactorings and improvents to msg_store startup

The underlying call to file:copy will not make use of any erlang-level read buffer, so don't bother with it.

Improve, correct and document msg store recovery

Correct and simplify the recovery from crashed compactions

When recovering, because of the possibility of finding the same msg twice, ensure that we only count and include the first copy. Other copies are then ignored.

A msg can appear > once in the same file because of multiple queues receiving the same msg. When we iterate through the msgs we find in the file, we must check that the msg corresponds to the msg in the index, by matching not just with guid but also with offset. Also just for fun, assert the total_size matches. Also, fixed a mistake in file_handle_cache where the dirty flag on the destination of copy wasn't being set.

typo + reformatting

refactoring

What's done:
- PubAck after transient messages
- PubAck after basic.returns
- PubAck after message delivered to a consumer (disregarding
consumer acks)
- PubAck after message got
- out of order ack'ing
- multiple ack'ing

Whant's not done:
- PubAck de-duplication
- PubAck after message hits disk

puback for persistent non-mandatory, non-immediate, unrouted messages

pubacks are sent right after basic.returns

simplified code a bit

puback is also sent after something basic.gets the message

partial acks for persistent messages

The publisher gets an ack in the following cases:
- the broker receives a transient message (after the exchange has
been resolved),
- the broker sends the persistent message to a consumer (regardless
of whether the consumer acks that message, rejects it, etc.).

Still to do:
- the broker writes the message to disk.

only transient messages are immediately ack'd (so far)

In addition, a message sequence number is recorded in the #delivery{}
sent to the exchange. If the message doesn't need to be ack'd, this
sequence number is undefined.

new logic

Message sequence numbers are assigned uniquely when the basic.publish
is received. If confirm is not enabled, this is undefined.

Rabbit_channel gains a new API method: confirm. This schedules an ack
for the given sequence number. The idea is that repeated acks are
allowed at this point and de-duplication is done by the channel. This
way, in order to confirm a message, another module just needs to
know 1) the channel and 2) the message sequence number.

No de-duplication is done at this point.

Remove dead code in tests

Move the linking into the start_channel function

Reduce calls to connection_sup:reader

Specs, moving things around, and removing dead code

Cosmetics

Remove writer flush and improve sync methods

bug 23118 merged into default (windows file descriptors limit)

Merging default into bug 15930

Improvements to comments

Reduce the file descriptor limit for Windows to 512

merge bug23120 into default

some instrumentation

only have one ack timer running at a time

type; all java tests pass now

minor clarification

merge default into bug15930

tiny simplifying refactor

cosmetic: friendlier type name

refactor: extract termination Reason analysis
from handle_dependent_exit and wait_for_channel_termination

revert the change from 'stop' to 'flush_stop'
and move the writer:flush call to a place where the need for it is
more obvious.

remove dead code

cosmetic

use consistent return types for start/start_link functions

refactor timer code; looks more like msg_store now

merge bug23095 into default

refactor + timer cancelled on channel terminate

When the channel terminates (for whatever reason), any outstanding
acks are simply lost. The reasoning behind this is that:
1) if the channel closed due to an exception, it should close immediately;
2) if the client wants to close the channel, but receive all of the
acks first, it should wait for the acks and only then send
channel.close.

support for multiple confirmations

Suppose the channel is in confirm multiple mode, and acks for the
following messages are outstanding:

> 3 4 5 7 10 13

The server sends one confirm-up-to 5 ack and three more acks for 7, 10
and 13.

If the client receives a basic.ack, it should interpret it like this:

basic.ack{delivery-tag=N, multiple=false} => the published message
with sequence number N has been handled

basic.ack{delivery-tag=N, multiple=true} => all the published
messages with sequence numbers up to and including N have been
handled

Merging bug 23095 into bug 15930

Merging default into bug23095

do not ignore 'normal' permanent child terminations during shutdown
If such a termination is worth reporting while the supervisor is
running, it sure is still worth reporting when it's shutting down.

This involves reverting all the changes related to ignoring normal
termination during shutdown and instead adding some logic a bit higher
up the call chain.

never ignore 'noproc' during shutdown
We have no idea why the child died. Furthermore, we will only ever get
this when a child has unlinked itself from the supervisor, which is
naughty and shouldn't go unnoticed.

we must always clear both the 'EXIT' and 'DOWN' message

cosmetic and minor refactoring

cosmetic

added support for multiple publisher acks for transient messages

When in confirm multiple mode, transient messages queue up in a
gb_set. Every 5s, these acks are sent out (individually, for now).

subsequent confirm.selects cannot change the multiple setting

rename pubAck -> confirm

merge bug23104 into default

remove redundant code

minor neatification

merge default into bug23104

cosmetic: consistency

Get the reader to link to the chan_sup instead of the framing_chan. It used to link to the framing chan, which meant that when it got the exit signal from the framing_chan, it could be sure that the channel and writer had already died. However, this is no longer the case - now the framing_chan is actually the last to start and first to exit in the chan_sup and so the reader needs to link to the chan_sup instead. This means the reader needs to track both the framing_chan and the chan_sup

make table content spot check part of schema_integrity_check
and
- rename check_table_content to check_table_integrity
- change wait_for_tables to only do what it says rather than also do
the schema integrity check. This change is necessary in order to
avoid an infinite recursion due to check_table_integrity needing to
call wait_for_tables
- add ensure_schema_integrity_check, which contains logic which
previously was part of wait_for_tables
- call wait_for_tables in all the right places, and no more

check that messages are transient before ack'ing blindly

added counter and single pubacks for transient messages

Publishes are counter starting with the first one after
pubAck.select().

If many is set to false, transient messages are ack'd as soon as the
are received, but after the exchange name is resolved.

idiot2

idiot

The table content tests need to happen *after* we've waited for the tables, otherwise they can fail. Thus now two phases - schema checks, as before, and then table content checks.

cosmetic

Improve match specs

Use match specs instead. This is a first pass - I want to be able to add some guards and assert things a little more tightly

Partial implementation of 3rd suggestion. Partial because I just wanted to see the shape of it rather than implement the whole thing and then possibly have to revoke it. IMO, testing on the deep structure of terms is likely sufficient

added pubAck handlers in rabbit_channel

A transactional channel cannot be made pubAck.
A pubAck channel cannot be made transactional.

added puback to framing codecs

merged default into bug23045

Remove dead code - collector is never explicitly shutdown: the supervisor will kill it off appropriately

I erroneously removed trap_exit on channel yesterday. It needs to be on because otherwise the gen_server won't call terminate when it's sent a shutdown signal

Abstract the heartbeaters in the same way - the reader now has no references to supervisor(2)?:.* nor does heartbeater.

Abstract out the limiter creation. The abstraction made ensures the channel never directly calls supervisor(2)?:.*, nor does it have any knowledge of the channel_sup.

Merging bug 23095 into bug 15930

Prevent supervisor2 from getting upset if a child has already vanished normally whilst the supervisor is trying to stop it

Merging an ancient outstanding head of bug 19662 into junk

Remerge bug23038 into default

Plugins directory should always exist

Make a plugin directory for the Windows package as well

merge bug23098 into default

When a supervisor is killing off its children, if its children have already exited normally, don't class it as an error - after all, the supervisor traps_exits, so the relevant exit signals will likely already be in the supervisor's mailbox

Fixes all over. Still more to do, though mainly API fixes. Functionality seems to be correct.

Given changes to intrinsic, if we do ever try to restart (abnormal exit), we should hit the max restart intensity immediately

Arbitrarily decide to make the channel_sup_sup the very first thing that gets started and thus the last that gets stopped

Flush the writer much earlier on, allowing the actual termination to be less risky

Merging bug 23098 into bug 15930

Merging default into bug 15930

Merging bug 23095 into bug 15930

Modify behaviour of intrinsic to be more like transient, except in the case of normal child exit

Reverting rabbit_sup to not use intrinsic

Merge bug21875 into default

allow erlang and rabbit to be upgraded even when plugins are present

use state_del_child instead of remove_child
remove_child removes child *specs* which is not what we want here

On reaching the max_restart_intensited, don't remove the child spec. Instead just blank out the child's pid, thus ensuring terminate works correctly even for simple_one_for_one_terminate

cosmetic

merge heads

merge bug23095 into default

merging bug23039 into default

Merging bug 23095 into bug 15930

Convert rabbit_sup to supervisor2 using intrinsic

Introduce intrinsic restart type

Merging default into bug 15930

Export gen_server2:enter_loop6 (oversight from the past)

Merging bug 23094 into bug 15930

cosmetic

cosmetic

Move parent inside of #v1 in reader

Merging default into bug 15930

Merge bug23091 (use protocol in #content rather than supplying it)

expand and rewrite tests
make it more obvious what we are testing and handle more cases

improve test coverage

handle transcoding of content when it is not available decoded

cosmetic
- we only use quoted atoms in types and when quoting is needed
- make existence check total - if it's the wrong type code further
down the chain will barf anyway

no protocol in rabbit_binary_parser:ensure_content_decoded
it can just use the protocol associated with the #content, which is
safer and presents a saner API

include method name in error if possible

oops

Fix stupid error, more consistency.

Added intrinsic restart type and used it

Removing the wrong changes to supervisor2 to suppress error reporting. Shall fix this one differently.

Merging default into bug 15930

Merge bug23035, list activated plugins

exit the framing_channel normally on unexected_frame error
and use explicit signalling to the reader instead of exiting
abnormally. This avoids cluttering the error log.

Unpack plugins into appdata on Windows.

Default to the real Unix-y scratch dir, override in the Makefile.

Merge heads

Merge bug23038, plugins dir in binary dists