merge bug23107 into default

bug23185 merged into default

Merge bug23197 (missed heartbeat may not get logged)

cosmetic

simplifying refactor

merge bug23215 into default

fixing premature generalisation

making gen:call label constant; separating calls handling

fixing unreachable function clause

Merged bug23181 into default

refactored code

shortened code a bit

in/3 -> in/2; drain/2 -> drain/1

client scope grants permission *only* for amq.gen resources

So, a client still need explicit permission to use amq.default, etc.

lists:member.

Allow byte, int, short, long.

merge bug23184 into default

Change type of expires argument from (signed)long to signedint since that's the best we can do in 0-8.

pending acks are sent out when the channel becomes idle or every 1s

fixed message leak caused by confirms

I've added variable_queue:init/4. It calls variable_queue:init/3 and
the fourth parameter is a tuple of 3 functions: the function to call
when the indices of some guids hit the disk, the function to call when
some guids hit the disk and the function to call when some guids have
been ack'd by the consumer. Rabbit_test supplies nops rather than the
usual casts, so the superfluous messages aren't generated.

better recursive delete

merge default into bug20284

The tests fail right now because queue_index sends out more messages
than expected.

Merged bug23183 into junk

remove the mnesia dir check from rabbit:prepare()

Call rabbit:prepare just before starting rabbit.

Rework delete_dir to return error tuple.

Merge bug23109 (remove reader profiling dwead code)

Increase robustness when the scratch dir cannot be deleted / written.

prevent normal heartbeater termination from tearing down the reader

All packaging makefiles can assume that "make srcdist" is already done

Merge bug23192 into default (errors in gen_server2 'terminate' callback mask termination reason)

merge bug15930 into default

Merge bug23106 (remove rabbit_hooks)

tweak

report the root cause when 'terminate' callback fails

minor cosmetic

implementing topic routing with tries; adding better test for topic routing

Merged bug23178 into default

move treating 'amq\.*' specially to inner case

The Permission argument to check_resource_access is not
a #permission{}. It is an atom naming which permission we want
(configure, read or write).

To treat 'amq.*' resources differently, the check needs to be moved to
the inner case.

Merge bug23141 into default

rabbitmq-multi should detect a bad NODENAME

removed underspecs warnings

solve a warning

delete_now -> delete

Set default paths for cygwin

Convert use of proplists:lookup to proplists:get_value

Merging default into bug 15930 and minor debitrot (heartbeater)

oops

merge bug19230 into junk

get rid of hook mechanism
It's only ever been used in the bug19230 branch, which is defunct.

remove dead code
nobody ever uses the reader profiling, and the underlying config
mechanism hasn't been used by anything else.

Merge bug 23131 into default (display plugin version during activation)

display app version during plugin activation

Ensure that non-deletion queue-termination does a call via the msg_store to ensure all messages from the queue have made it to the msg_store's mailbox (and have actually also been acted on)

Merging in default now that default has only one head

Merge heads of default

exclusive queues are deleted quickly on unclean connection termination

exclusive queues are deleted quickly on clean connection termination

Convert client_delete_and_terminate from a call to a cast

updated docs

removed all references to pcall and pcast

preemptively detect common cause of failed startup
namely when the node does actually start (if it doesn't then the
launcher process terminates, which we detect already) but we cannot
connect to it due to dns/network/epmd issues

typo

refactoring

migrated rabbit_limiter to prioritizers

refactoring

migrated amqqueue to prioritisers

migrated msg_store to prioritisers

merge default into bug23157

rabbit_channel migrated to prioritisers

added error handling wrappers for prioritisers

prioritize -> prioritise

Don't nest lists

Supply ssl verify function conditionally

Added tag rabbitmq_v2_0_0 for changeset 4f72e4d66f23

Changelog updates

Take peer verification config into account in verify function

This works around a limitation in the ssl module, which does not honour verify_none

Accept multiple SSL verification errors

merge bug23167 into default (Large msg_store mailbox can block queue.declare)

unprioritized calls, casts and infos are passed through the prioritize functions

PCalls and PCasts have the priorities assigned to them by the pcall, pcast call.

Everything else gets its priority set by prioritize_call/3,
prioritize_cast/2 or prioritize_info/2.

gen_server2: functions now use a record instead of 9 arguments

Don't ignore SSL certificate verification errors

Merge bug23155

Add priorities to two calls to ensure queue declaration remains quick even with a large msg_store workload

make 'xmlto' invocation work with stricter getopt implementations

merge bug23153 into default

Avoid crash

ensure and assert that closed handles have last_used_at=undefined
That way it is more obvious which handles should be in the age tree
and which shouldn't.

cosmetic

make it more obvious that reopen is a with_age_tree in disguise
plus some minor cosmetic shuffling

don't traverse age tree unnnecessarily on error in reopen
this also makes it clearer what's going on

cosmetic

refactor: move open1 into reopen

If we've obtained permission to open some fds and that turns out not to work out, then we should ensure that we inform the server as soon as we stop that the rest remain closed

record for pending items

Modify test so that the -1 => -2 change to OBTAIN_LIMIT does not result in the test having sufficient fds available such that the copy could be satisfied without closing

Use foldr instead of foldl to avoid having to do extra reverses

always return refs/handles in correct order
to improve clarity at the expense of a few list:reverses.

cosmetic

always leave at least two fds unobtainable
so that 'copy' can succeed
Also, zero-clamp obtain limit

Merging heads of default

Merging bug 23166 into default

refactor: make new_closed_handle create the Ref and return it

add test
this fails on default (or, rather, bug23166, until that is merged)

merge bug23166 into bug23153

add support for dynamic setting/getting of fhc limit
and use that in tests

Rejigged client side so that it asks for the correct number of fds upfront.

Modify server logic to deal with requests for multiple fds. Including nice pending_queues abstraction. Client logic remains to be done.

Add infrastructure to server to be able to handle requests for multiple fds

merge bug23152 into default

merge bug23139 into default

oops

merge heads

simplification due to the fact that we always request closing of all

cosmetic

ensure_mref => track_client andalso cosmetic

Merging heads

Convert fhc to use an ets table with record per client which amalgamates several of the previous state entries

cosmetic and some minor refactoring

fix bug that crept in

Given the clients are demanded to close all open fds when asked to, them sending a boolean is irrelvant now

If we sent an age of 0 to clients, make sure we do not send more ages of 0 to the same clients until they've actually closed all their handles. This ensures that as more requests come in once we're low on fds, we don't send hundreds of 0 ages to the same clients erroneously. It also means that we always target the correct number of *unique* clients to ask to close their fds, which avoids thrashing the same clients and improves performance markedly.

Also, if on open, we send "close" back to the client, that client *is* blocked (actually, due to have 0 opens) as we know it'll close, send us some closed msgs and then re do the open call. Thus we shouldn't be sending it any set maximum age messages.

stopped a race

Grab the msg from the cur ets file cache, thus avoiding having to send the same message many times

refactor

cosmetic

Merging default into bug 23133

Merging old head of bug 23133 to junk

Abstract refcount == 0 to imply not_found for appropriate call sites

By this stage of the client-read, we know a GC can't start and can't be in progress. Thus even if the refcount is 0, we're still safe to read it. What's more, this area of code has never made provision for the index_lookup to return a not_found - it will crash if that happens. Thus if the implicit assumption is that the message must exist, then by the same token, we know the refcount will always be > 0

Abstraction of the inner write function

cosmetic

Because we can increment the refcount back above 0, the calculation of the contiguous top needs to be contiguousTop + totalSize, because the validtotalsize can contain data that's well beyond the old contiguous top

fixed an ever-expanding dictionary

The problem with the current design is that the lowest layer that knows
about confirmations is amqqueue_process. This means that
amqqueue_process, variable_queue, msg_store and queue_index need to do
all the bookkeeping associated with confirmations even if they're not
enabled. Mind you, this doesn't slow things down noticeably.

refactoring + don't do useless work

don't do any unnecessary work when publishing in confirm mode

Some essential assertions

More meaningful variable name

Whoops

Reworked substantially

merge bug23150 into default

Remove modifications that I accidentally introduced for testing and forgot to remove prior to ci'ing the merge

Merging default into bug 23139 (substantial debitrot)

Use string tokens, not re:split

Merging bug 23138 into default

once again allow use of fhc w/o registering

refactor

rewrite

added xref check for undefined, unused and deprecated functions

Don't ever keep the recovery process waiting, regardless of whether the queue is going down or not
(transplanted from 4c99ba7eedd4b28a096d0412bbbdacb1fa91daa3)

A rather crucial infinity missing
(transplanted from aaf79aa3cacefee1742eb7f257c6a16ec5720d59)

re-merge rebased bug23145 into new head of default

Eliminate RABBITMQ_PLUGINS_EXPAND_DIR

There were a number of issues with RABBITMQ_PLUGINS_EXPAND_DIR:

- It was undocumented in the context of the generic unix package, and
if unwisely set could do an effective "rm -rf" in an unintended
location.

- It did not take account of the possibility that multiple nodes could
be starting at once, and so doing plugins activation
simultanteously.

Instead, use RABBITMQ_MNESIA_DIR/plugins-scratch. This avoids the
need to extend the generic unix package documentation, the location is
node-specific, and the distinctive plugins-scratch subdirectory
reduces the risk of unintended file deletions.
(transplanted from 064b8797493bb290156fb72a54f9e9276df0faed)

merge default into junk
in order to backout the merge of bug23142, which turns out to have
introduced some not easily fixed breakages.

remove files from junk
to aid subsequent merges

Cope with removes overtaking writes

Merging default into ug 23142

Merging default into bug23133

fixed a small empty-set bug

deleting a queue causes all of its messages to be puback'd

confirm transient messages only after basic.return

Don't ever keep the recovery process waiting, regardless of whether the queue is going down or not

A rather crucial infinity missing

Abstract refcount == 0 to imply not_found for appropriate call sites

By this stage of the client-read, we know a GC can't start and can't be in progress. Thus even if the refcount is 0, we're still safe to read it. What's more, this area of code has never made provision for the index_lookup to return a not_found - it will crash if that happens. Thus if the implicit assumption is that the message must exist, then by the same token, we know the refcount will always be > 0

Abstraction of the inner write function

Merge in bug23145 (put plugin expand dir in db dir)

cosmetic

Because we can increment the refcount back above 0, the calculation of the contiguous top needs to be contiguousTop + totalSize, because the validtotalsize can contain data that's well beyond the old contiguous top

Eliminate RABBITMQ_PLUGINS_EXPAND_DIR

There were a number of issues with RABBITMQ_PLUGINS_EXPAND_DIR:

- It was undocumented in the context of the generic unix package, and
if unwisely set could do an effective "rm -rf" in an unintended
location.

- It did not take account of the possibility that multiple nodes could
be starting at once, and so doing plugins activation
simultanteously.

Instead, use RABBITMQ_MNESIA_DIR/plugins-scratch. This avoids the
need to extend the generic unix package documentation, the location is
node-specific, and the distinctive plugins-scratch subdirectory
reduces the risk of unintended file deletions.

Merge default into bug 23133

Backed out changeset c47670da4830 - I decided the dedup cache shouldn't be mangled with.

merge bug23142 into default

Minor tweaks

eliminate duplicate write messages completely, and simply rely on the pending write count in the cur ets file cache to indicate the delta to the refcount

told dialyzer to check for more things

As Matthew suggested, I've added {wranings, [underspecs, behaiviours,
race_conditions]} to dialyzer's options. As a result, we now have 56
warnings instead of just the 4. Most of them are underspecs (i.e. we
say a function returns any() but dialyzer's figured out that's overly
general).

small simplifying refactor

take current memory alarms status into account straight away
This does in fact not alter the behaviour at all due to the following:

- if the alarm is active the alarm registration will call the handler
straight way, which will send a {conserve_memory, true} message to the
reader.
- the reply to the alarm registration is sent after that, and from the
same process - the alarm process - so by the time the reader loops
around the mainloop again the {conserve_memory, true} message is
guaranteed to be in the mailbox
- on looping around, the reader request a frame_header from the
socket. The reader has already sent connection.open_ok to the client,
and the client may have started sending commands straight away. But
all the reader is going to see of that to start with is an {inet_async,
...} message for a frame_header. That is guaranteed to end up in the
mailbox after the {conserve_memory, true} message.

Thus the reader is guaranteed to process the {conserve_memory, true}
message before handling any more data from the socket.

With this change it is rather more obvious that the memory alarm
status gets taken into account before any more client data is
processed.

treat the dedup cache the same way as the index - permit 0 ref_counts and prune on file roll

Merging default into bug 15930

Merging default into bug 23133

The solution is very simple: In the case where the fhc sends out requests to close file handles, the clients might respond very quickly. The fhc will then gather these responses (say, just updates, not closes) and then will sit there for 2 seconds until the timer goes off. Thus the solution is just to subtract the timer period from the calculated average: i.e. the expression is to say 'close file handles that haven't been used for N seconds from NOW' rather than the previous 'close file handles that haven't been used for N seconds from NOW - 2 seconds ago'. This works very nicely and whilst the fhc can get quite busy when there are more users of file handles than there are file handles available, that is hardly surprising, and the fact is starvation is prevented and processes are promptly serviced

Track blocked pids explicitly

refactoring

publisher acks for persistent messages

Acknowledgements are sent to the producer in the following cases
(assuming the channel in confirm mode):
- a transient message has been received (i.e. it is ack'd
immediately),
- a basic return has been sent (the ack is sent immediately
afterwards),
- a persistent message has been consumed [by a consumer] or got [via
basic.get] and ack'd by the client (or the retrieval was no-ack),
- a persistent message has been written to disk (both the message
and its index).

At least in theory, every message published on a confirm channel will
at some point be ack'd. A message is only ack'd once and the reason
it was ack'd is hidden.

queue_index keeps track of which messages have been published but not written to disk

There are 2 ways queue_index writes something to disk:
- the journal is synced, and
- the journal is flushed (i.e. scattered across many files and those
files closed).

Since queue_index is the same process as variable_queue, we just keep
a list of guids that have been published and we just call an on_sync
handler whenever these are written to disk.

persistent messages are ack'd only after the consumer acks it

move msg_seq_no from #basic_message{} to #delivery{}

Correct monitoring and actions upon DOWN messages. Note this is especially subtle for obtains, which effectively implicitly allocates temporarily to the blocked caller (FromPid) whilst monitoring it, and then transfers this to the ForPid when possible. Note the ForPid can die before the obtains is processed, which which point the FromPid must be replied to immediately.

merge bug23135 into default

obtain_and_release_on_death => obtain, and minor refactor of tcp acceptor

Combine obtains and release_on_death

bug22906 merged into junk

merge bug23132 into default

Minor corrections

first stab at keeping track of when messages are synced to disk

refactoring

PubAcks are de-dupped in rabbit_channel

cosmetic

tweak

tweak