Stefan Fuhrmann <stefanfuhrmann@alice-dsl.de> (stefan2)


Patch
r1030909, r1030763, r1030759, r1029381, r1029344, r1029342, r1029340, r1029335, r1029111, r1029108, r1029090, r1029078, r1029055, r1029054, r1029043, r1029042, r1029038, r1028111, r1028107, r1028104, r1028094, r1028092, r1028077, r1003430, r1002898, r1001417, r1001413, r999098, r998858, r998852, r998843, r998649, r998617, r998012, r995603, r994956, r992911, r992905, r992904, r992899, r990759, r990600, r990575, r990574, r990572, r990568, r990541, r990537, r990536, r990535, r990533, r988898, r988319, r987888, r987887, r987886, r987869, r987868, r986832, r986817, r986608, r986605, r986521, r986517, r986492, r986491, r986485, r986465, r986459, r986453, r985697, r985695, r985673, r985670, r985669, r985606, r985605, r985603, r985602, r985601, r985514, r985500, r985497, r985493, r985488, r985487, r985482, r985477, r985472, r985046, r985037, r985014, r984984, r984973, r984928, r984927, r984926, r983770, r983766, r983764, r983760, r983490, r983488, r983474, r983437, r983430, r983406, r983398, r983385, r982417, r982391, r982375, r982360, r982355, r982057, r982043, r981828, r981827, r981684, r981665, r981287, r981204, r981194, r981189, r981091, r981090, r981087, r980118, r979193, r964568, r964557, r956593, r941243, r921057

r1030909 | julianfoad | 2010-11-04 07:40:37 -0500 (Thu, 04 Nov 2010)

Make svn_tristate_t mainly compatible with svn_boolean_t by making its
numerical values match the ones used for svn_boolean_t.

* subversion/include/svn_types.h
  (svn_tristate_t): Define svn_tristate_true and svn_tristate_false in terms
    of TRUE and FALSE, respectively.

Patch by: Stefan Fuhrmann <stefanfuhrmann{_AT_}alice-dsl.de>


r1030763 | stefan2 | 2010-11-03 18:54:56 -0500 (Wed, 03 Nov 2010)

Improve doc string to mention the assumptions made 
about the parameters. Also, mention how another edge
case would be handled.

* subversion/libsvn_subr/subst.c
  (eol_unchanged): improved commentary.

Suggested by: danielsh

r1030759 | stefan2 | 2010-11-03 18:36:50 -0500 (Wed, 03 Nov 2010)

Fix yet another mistake in tristate usage.

* subversion/libsvn_fs_fs/fs_fs.c:
  (svn_fs_fs__rev_get_root): convert boolean to tristate before 
  comparing it with another tristate

r1029381 | stefan2 | 2010-10-31 10:22:31 -0500 (Sun, 31 Oct 2010)

Fix another packing issue. Due to concurrent access alone
it may happen that between determining the rev file name
and actually opening it, that very revision may get packed
(actually, get deleted after packing). 

The file handle cache adds another issue: an open file handle 
may not be suitable due to different open flags and the file 
must be re-opened. While the existing handle would have
kept the file content alive, opening a new handle to the same
file will fail if the file was deleted before. 

Since there can be only one transition from non-packed to
packed, we need to retry only once if we could not find the
revision file in question.

* subversion/libsvn_fs_fs/fs_fs.c
  (open_pack_or_rev_file): retry once because the file might have
   gotten packed.

r1029344 | stefan2 | 2010-10-31 08:52:30 -0500 (Sun, 31 Oct 2010)

On the performance branch:
* STATUS: remove r985472 after merger and checking that the issues mentioned
  have already been addressed (svn_io_open_uniquely_named is not used outside
  io.c, tests and deprecated code anymore)

r1029342 | stefan2 | 2010-10-31 08:44:19 -0500 (Sun, 31 Oct 2010)

On the performance branch:
* STATUS: add revision 1029335

r1029340 | stefan2 | 2010-10-31 08:40:12 -0500 (Sun, 31 Oct 2010)

Fix handling of cached revision root IDs. Issue discussed here:
http://svn.haxx.se/dev/archive-2010-10/0497.shtml

The /trunk code caches them for a single session (i.e. server
request) only and assumes that the respective revision files
won't get packed while the request is being processed.

But now, the cache lives as long as the server process and
therefore, we must detect whether the respective revision
got packed (or, theoretically, unpacked) after we cached its
root ID. If that state changed, the file offset needs to be
updated. Simply re-read the whole ID in that case.

* subversion/libsvn_fs_fs/id.h
  (svn_fs_fs__is_packed, svn_fs_fs__set_packed): new functions
  to store the "rev is packed" state

* subversion/libsvn_fs_fs/id.c
  (id_private_t): add new element
  (svn_fs_fs__is_packed, svn_fs_fs__set_packed): implement
  (svn_fs_fs__id_txn_create, svn_fs_fs__id_rev_create, svn_fs_fs__id_copy,
   svn_fs_fs__id_parse): handle the new element as well

* subversion/libsvn_fs_fs/fs_fs.c
  (svn_fs_fs__rev_get_root): require "is packed" flag to match 
   before returning a cached root ID; store that flag in cached ID


r1029335 | stefan2 | 2010-10-31 08:25:43 -0500 (Sun, 31 Oct 2010)

Fix testing translate_chunk for (potentially) mixed newline files
when repairing has been disabled.

* subversion/libsvn_subr/subst.c 
  (translate_chunk): properly test for tristate true

Found by: danielsh

r1029111 | stefan2 | 2010-10-30 11:31:53 -0500 (Sat, 30 Oct 2010)

On the performance branch:
* STATUS: remove merged revisions of the string capacity alignment patch set

r1029108 | stefan2 | 2010-10-30 11:17:16 -0500 (Sat, 30 Oct 2010)

Merge r1028094, r1028104, r1029038 and r1029090 from performance branch.

This change minimizes the memory wasted when allocating
strings of odd size. All memory allocated will actually
be available as string buffer capacity.

Approved by: danielsh

r1029090 | stefan2 | 2010-10-30 09:58:28 -0500 (Sat, 30 Oct 2010)

Follow-up to r1029038:
Passing '++blocksize' to a macro is a bad idea, so don't do it.
Also, fix a C89 declaration order compliance issue.

* subversion/libsvn_subr/svn_string.c
  (svn_stringbuf_create_ensure): fix as described above

Found by: danielsh

r1029078 | stefan2 | 2010-10-30 09:20:17 -0500 (Sat, 30 Oct 2010)

On the performance branch:
Bring up-to-date with trunk.


r1029055 | stefan2 | 2010-10-30 08:09:11 -0500 (Sat, 30 Oct 2010)

String-based streams are "buffered" as well, so we have
to overwrite the default "not buffered". Otherwise, line
parsing will be very slow on these streams.

* stream_readline_chunky
  (buffered_handler_string): new function
  (svn_stream_from_string): set new function in the vtable

r1029054 | stefan2 | 2010-10-30 08:02:29 -0500 (Sat, 30 Oct 2010)

Improve stream_readline_chunky performance by no longer
falling back to stream_readline_bytewise upon EOF anymore
because the fall-back would also just copy everything up to
the EOF. We can also save on a string copy that was left from
when we used a temp. pool for the string.

* subversion/libsvn_subr/stream.c
  (stream_readline_chunky): treat EOF as EOL; don't copy string
  from 'pool' to 'pool'

r1029043 | stefan2 | 2010-10-30 07:24:32 -0500 (Sat, 30 Oct 2010)

On the performance branch:
* STATUS: add revisions 1029038 and 1029042; 
  remove revisions that have been merged without comment

r1029042 | stefan2 | 2010-10-30 07:18:27 -0500 (Sat, 30 Oct 2010)

Replace iterative calls to svn_stringbuf_appendbyte copying strings
with calls to svn_stringbuf_appendbytes after the iteration. However,
there is only one such place outside the tests that used this pattern.
(Other places are reading from a stream or have complicated iteration
and copy logic.)

* subversion/libsvn_diff/parse-diff.c
  (parse_hunk_header): find copy range first and then call svn_stringbuf_appendbytes

r1029038 | stefan2 | 2010-10-30 07:09:49 -0500 (Sat, 30 Oct 2010)

Improve on r1028104:
Move string buffer size alignment logic down the call stack.
Also, extend the logic to cover string growth as well.

* subversion/libsvn_subr/svn_string.c
  (svn_stringbuf_ncreate): remove alignment logic
  (svn_stringbuf_create_ensure): add it here
  (svn_stringbuf_ensure): and here

Suggested by: danielsh


r1028111 | stefan2 | 2010-10-27 16:48:01 -0500 (Wed, 27 Oct 2010)

On the performance branch:
* STATUS: add revision 1028104

r1028107 | stefan2 | 2010-10-27 16:29:54 -0500 (Wed, 27 Oct 2010)

On the performance branch:
* STATUS: add revisions 1028092, 1028094

r1028104 | stefan2 | 2010-10-27 16:23:34 -0500 (Wed, 27 Oct 2010)

Adapt string unit test to recent behavioral changes.

* subversion/tests/libsvn_subr/string-test.c
  (test10): relax tests on string capacity

r1028094 | stefan2 | 2010-10-27 15:47:06 -0500 (Wed, 27 Oct 2010)

If we allocate stringbufs, their actual capacity allocated by APR
is often larger than requested. Take advantage of that by requesting
an enlarged buffer in the first place. Especially for short or empty 
strings, this saves some re-allocations once they grow.

* subversion/libsvn_subr/svn_string.c
  (svn_stringbuf_ncreate): allocate a buffer of aligned size

r1028092 | stefan2 | 2010-10-27 15:40:53 -0500 (Wed, 27 Oct 2010)

Incorporate feedback I got on r985606.

* subversion/libsvn_ra_svn/marshal.c
  (SUSPICIOUSLY_HUGE_STRING_SIZE_THRESHOLD): introduce symbolic name
  for an otherwise arbitrary number
  (read_long_string): fix docstring
  (read_string): use symbolic name and explain the rationale behind the special case

r1028077 | stefan2 | 2010-10-27 14:24:10 -0500 (Wed, 27 Oct 2010)

On the performance branch:
* STATUS: Remove entries from STATUS that have been merged to 
  trunk without further comments.

r1003430 | stefan2 | 2010-10-01 03:33:27 -0500 (Fri, 01 Oct 2010)

If APR_HAS_THREAD has not been defined, we got a compiler error.
Fix that.

* subversion/libsvn_subr/svn_file_handle_cache.c
  (svn_file_handle_cache__create_cache): init mutex only if APR_HAS_THREAD
   has been set to > 0

r1002898 | stefan2 | 2010-09-29 19:01:45 -0500 (Wed, 29 Sep 2010)

Merge r1001413 from the performance branch.

Improve documentation on svn_stringbuf_appendbyte.

Approved by: Hyrum K. Wright <hyrum_wright_at_mail.utexas.edu> 

r1001417 | stefan2 | 2010-09-26 06:58:58 -0500 (Sun, 26 Sep 2010)

On the performance branch:
Bring up-to-date with trunk.


r1001413 | stefan2 | 2010-09-26 06:01:03 -0500 (Sun, 26 Sep 2010)

Extensively document the benefits of svn_stringbuf_appendbyte and 
the rationals behind its implementation. To simplify the explanations,
one statement had to be moved.

* subversion/include/svn_string.h
  (svn_stringbuf_appendbyte): extend docstring to indicate that this method
  is cheaper to call and execute
* subversion/libsvn_subr/svn_string.c
  (svn_stringbuf_appendbyte): reorder statements for easier description;
  add extensive description about the optimizations done

r999098 | stefan2 | 2010-09-20 15:16:13 -0500 (Mon, 20 Sep 2010)

On the performance branch:
Bring up-to-date with trunk.


r998858 | stefan2 | 2010-09-20 04:50:27 -0500 (Mon, 20 Sep 2010)

Add STATUS file to track performance branch reviews.
So far, it only contains a list of small, independent changes.

r998852 | stefan2 | 2010-09-20 04:43:01 -0500 (Mon, 20 Sep 2010)

This patch combines all patches from the performance branch that
implement the svnadmin command line UI part of the membuffer cache 
configuration.

Revisions merged partially from /branches/performance
(excluding all changes not related to the membuffer cache):
982057, 987886

* subversion/svnserve/server.h
  (serve_params_t): add cache size parameter
* subversion/svnserve/main.c
  (svnserve__options): add "-M" parameter
  (main): initialize, parse and apply the new parameter


r998843 | stefan2 | 2010-09-20 04:27:17 -0500 (Mon, 20 Sep 2010)

This patch combines all patches from the performance branch that
implement the membuffer cache configuration but without the UI change.
A process-global membuffer cache will now be used for full-texts 
unless memcached has been enabled for the respective directory.

To minimize the number of future conflicts, the configuration structure 
has been brought up to its final state even if some options (cached file
handle count, txdelta caching etc.) will have no effect right now. 

Merged revisions from /branches/performance:
981684, 982043, 988319
Merged only partially (excluding all changes not related to the
membuffer cache):
987886

* subversion/include/private/svn_fs_private.h
  (svn_fs__get_global_membuffer_cache): 
   declare new internal API function
* subversion/include/svn_fs.h
  (svn_fs_cache_config_t): new cache config struct
  (svn_fs_get_cache_config, svn_fs_set_cache_config): new public API
   to get & modify the cache config; currently evaluated by FSFS only

* subversion/libsvn_fs_util/caching.c
  (cache_settings): new global cache settings + their defaults
  (svn_fs__get_global_membuffer_cache, svn_fs_get_cache_config, 
   svn_fs_set_cache_config): implement new public and internal API functions
* subversion/libsvn_fs_fs/caching.c
  (svn_fs_fs__initialize_caches): use membuffer cache for full-texts

r998649 | stefan2 | 2010-09-19 07:52:26 -0500 (Sun, 19 Sep 2010)

This patch combines all patches from the performance branch that
implement the membuffer cache. To minimize the number of future 
conflicts, the cache-membuffer.c has been brought up to its final
state even if some functionality (partial get, max. cachable item)
is not going to be used right now. The vtable had to shortened.

Merged revisions from /branches/performance:
979193, 980118, 981087, 984973, 990541, 990568,
990572, 990600, 990759, 992899, 992911, 994956
Merged only partially (excluding all changes to cache-membuffer.c):
986605, 986832, 992904

* subversion/include/private/svn_cache.h
  (svn_cache__partial_getter_func_t): declare new callback function;
   only to allow for cache-membuffer.c to be as close to the final version
   as possible
  (svn_membuffer_t): declare opaque structure for membuffer caches
  (svn_cache__membuffer_cache_create, svn_cache__create_membuffer_cache): 
   declare new internal API functions

* subversion/libsvn_subr/cache-membuffer.c
  (GROUP_SIZE, KEY_SIZE, ITEM_ALIGNMENT, CACHE_SEGMENTS, NO_INDEX,
   NO_OFFSET, ALLOCATIONS_PER_POOL_CLEAR): new defines
  (entry_t, entry_group_t, membuffer_cache_t, svn_membuffer_cache_t): 
   new structures
  (ALIGN_VALUE, ALIGN_POINTER): new internal alignment macros
  (lock_cache, unlock_cache, get_entry, get_index, drop_entry,
   insert_entry, get_group_index, let_entry_age, find_entry, move_entry,
   ensure_data_insertable, membuffer_cache_set, membuffer_cache_get,
   svn_membuffer_cache_get_partial):
   new internal functions implementing the membuffer cache
  (combine_key, svn_membuffer_cache_get, svn_membuffer_cache_set,
   svn_membuffer_cache_iter, serialize_svn_stringbuf, 
   deserialize_svn_stringbuf): implement the svn_cache_t interface 
  (svn_membuffer_cache_get_partial, svn_membuffer_cache_is_cachable):
   implement the future svn_cache_t extensions 
  (svn_cache__membuffer_cache_create, svn_cache__create_membuffer_cache): 
   implement API functions
  (membuffer_cache_vtable): new static variable

r998617 | stefan2 | 2010-09-19 03:55:02 -0500 (Sun, 19 Sep 2010)

Opening review and integration branch for the "membuffer cache"
part of the performance branch. Review on this integration branch
should be easier because it does not show intermediate states that
got replaced later.

r998012 | stefan2 | 2010-09-17 02:45:49 -0500 (Fri, 17 Sep 2010)

Change the svn_io_file_read_full2 API to accept an [out] boolean eof_hit
parameter instead of an [in] boolean eof_hit_ok parameter. That way,
the caller will always be informed of the EOF - either by the boolean being
set or by error return value.

* subversion/include/svn_io.h
  (svn_io_file_read_full2): replace eof_is_ok with hit_eof parameter
* subversion/libsvn_subr/io.c
  (svn_io_file_read_full2): adapt implementation to new API signature
  (contents_identical_p): use new API instead of the deprecated one

* subversion/libsvn_subr/deprecated.c
  (svn_io_file_read_full): adapt to API change
* subversion/libsvn_subr/hash.c
  (svn_hash_read): dito
* subversion/libsvn_subr/stream.c
  (read_handler_apr): dito
* subversion/libsvn_repos/reporter.c
  (read_string): dito
* subversion/libsvn_fs_fs/fs_fs.c
  (create_rep_state_body, get_contents, read_handler_recover): dito
* subversion/libsvn_diff/diff_file.c
  (read_chunk, map_or_read_file): dito

r995603 | stefan2 | 2010-09-09 17:58:06 -0500 (Thu, 09 Sep 2010)

Try really hard to help the compiler generate good code
for this frequently called function.

* subversion/libsvn_subr/svn_string.c
  (svn_stringbuf_appendbyte): reformulate

r994956 | stefan2 | 2010-09-08 04:58:07 -0500 (Wed, 08 Sep 2010)

Fix server crashes reported by Johan Corveleyn.
As it turned out, find_entry used only the first 4 or 8 bytes
of the MD5 hash as a key to identify an entry.

* subversion/libsvn_subr/cache_membuffer.c
  (find_entry): compare & copy KEY_LEN bytes when dealing with keys.


r992911 | stefan2 | 2010-09-05 19:16:51 -0500 (Sun, 05 Sep 2010)

Fix another size conversion warning that slipped under the radar. 
Once we are at it, fix formatting of code and comments in the
same function.

* subversion/libsvn_subr/cache-membuffer.c
  (svn_cache__membuffer_cache_create): fix conversion warning
   for apr_palloc call; various formatting fixes

r992905 | stefan2 | 2010-09-05 17:30:09 -0500 (Sun, 05 Sep 2010)

Format fix:
Remove spaces between function name and the opening parentheses

* subversion/libsvn_subr/svn_file_handle_cache.c
  (find_first, open_entry): fix calls to assert()

r992904 | stefan2 | 2010-09-05 17:27:48 -0500 (Sun, 05 Sep 2010)

Fix various integer size conversion warnings. 

For buffer pre-allocations it is sufficient to silence the 
warning as those buffers would be expanded automatically, 
if they had to. But the system will run OOM before that.

* subversion/libsvn_fs_fs/fs_fs.c
  (rep_read_get_baton): silence conversion warning
  (fulltext_size_is_cachable): reject caching for huge items 
   on 32 bit systems
* subversion/libsvn_ra_svn/marshal.c
  (read_string): silence conversion warning
* subversion/libsvn_subr/io.c
  (svn_io_open_uniquely_named): dito
* subversion/libsvn_subr/cache-membuffer.c
  (NO_INDEX, NO_OFFSET): introduce proper unsigned defines
  (entry_t, svn_membuffer_t): adapt commentary
  (align_entry): replace
  (ALIGN_VALUE, ALIGN_POINTER): introduce alignment macros that 
   avoid conversion issues
  (drop_entry, insert_entry, get_group_index, find_entry, move_entry,
   membuffer_cache_set, membuffer_cache_get): use new defines
  (ensure_data_insertable, svn_cache__membuffer_cache_create): 
   use new defines; fix local variable types 

r992899 | stefan2 | 2010-09-05 17:00:40 -0500 (Sun, 05 Sep 2010)

Improve scalability by segmenting the cache: the mutex is likely 
to experience high contention in case all data is cache.

So, use an array of caches ("segments"), each segment being
completely independent from all other segments. The hash key
is used to unambiguously assign items to segments.

* subversion/libsvn_subr/cache-membuffer.c
  improve documentary
  (CACHE_SEGMENTS): new define
  (get_group_index): select the cache segment, too
  (svn_cache__membuffer_cache_create): allocate and initialize 
   CACHE_SEGMENTS cache segments instead of just one.
  (membuffer_cache_set, membuffer_cache_get, 
   membuffer_cache_get_partial): adapt to signature changes
  (svn_membuffer_cache_is_cachable): max item size depends on #segments


r990759 | stefan2 | 2010-08-30 05:38:47 -0500 (Mon, 30 Aug 2010)

Non-API functions should be local / static. There was one in 
the membuffer code that violated this principle.

* subversion/libsvn_subr/cache-membuffer.c
  (membuffer_cache_get_partial): make this a static function

Found by: Lieven Govaerts <svnlgo_at_mobsol.be> 

r990600 | stefan2 | 2010-08-29 11:58:14 -0500 (Sun, 29 Aug 2010)

Minor typo / grammar fix in commentary. 

* subversion/libsvn_subr/cache-membuffer.c
  (svn_cache__membuffer_cache_create, membuffer_cache_set): typo

r990575 | stefan2 | 2010-08-29 07:37:39 -0500 (Sun, 29 Aug 2010)

Fix compiler warnings about potentially uninitialized variables.
In the current implementation, they will not be used when uninitialized
but better be safe than sorry.

[I only fix that warning in sections touched by the performance branch.]

* subversion/libsvn_fs_fs/fs_fs.c
  (svn_fs_fs__rev_get_root): initialize root_id to NULL
  (svn_fs_fs__rep_contents_dir): initialize unparsed_id to NULL

r990574 | stefan2 | 2010-08-29 07:33:39 -0500 (Sun, 29 Aug 2010)

Fix a crash when membuffer caching has been disabled (svnserve -M0):
the get_partical implementation for inprocess_cache used an outdated
callback API where the size could be 0 for non-empty entries in certain cases.

* subversion/libsvn_subr/cache-inprocess.c
  (inprocess_cache_get_partial): actually pass the size parameter to func

r990572 | stefan2 | 2010-08-29 07:00:11 -0500 (Sun, 29 Aug 2010)

Optimize cache data transfers, especially on x86/x64. Memory copies 
are most efficient when they transfer an aligned amount of data (i.e.
multiple of 16) from aligned memory locations to aligned memory
locations. 

Locations within the cache data buffer already are aligned. This patch
aligns the copy sizes and output buffer allocated in get(). The overhead
is easily justified by the gains from copying the relatively large entries
(average is 4+kB).

* subversion/libsvn_subr/cache-membuffer.c
  (move_entry): allow for copying in 16-byte chunks
  (membuffer_cache_get): dito; align copy target as well

r990568 | stefan2 | 2010-08-29 06:39:11 -0500 (Sun, 29 Aug 2010)

Harden the caching strategy. Flooding the cache with large entries (e.g. full texts)
will soon sweep the smaller, more cache-efficient entries. It also prevents earlier
large entries from survival and getting actual hits.

This is fixed using two additional rules:
(1) entries much smaller than the *current* average are kept unconditionally
(2) new entries may only push out small amounts of data and thus may get
    rejected if not enough room could be made that way

* subversion/libsvn_subr/cache-membuffer.c
  (let_entry_age): extract the entry "aging" code and give it its own function
  (find_entry, move_entry): simplify using let_entry_age
  (ensure_data_insertable): refine entry eviction code; allow for rejecting entries
  (membuffer_cache_set): insert data only if room is available / could be made

r990541 | stefan2 | 2010-08-29 06:01:02 -0500 (Sun, 29 Aug 2010)

Ensure that very large membuffers don't cause internal overflows
due to large items being added (4GB limit could be reached with
64+GB caches) and large number of entries in the dictionary (4G
entry limit would be reached with 3.2TB caches).

* subversion/libsvn_subr/cache-membuffer.c
  (get_entry): change parameter type to be consistent with the index
   type used everywhere else in this file
  (get_index): explicitly cast the result to the correct index type
  (svn_cache__membuffer_cache_create): limit the dictionary size
   such that entries can be safely addressed using the index type
  (svn_membuffer_cache_is_cachable): also, entry sizes must be
   representable in 32 bits

r990537 | stefan2 | 2010-08-29 05:32:08 -0500 (Sun, 29 Aug 2010)

Looking for the cause of Johan Corveleyn's crash (see 
http://svn.haxx.se/dev/archive-2010-08/0652.shtml), it 
seems that wrong / corrupted data contains backward
pointers, i.e. negative offsets. That cannot happen if
everything works as intended.

* subversion/libsvn_subr/svn_temp_serializer.c
  (svn_temp_deserializer__resolve): add assertion

r990536 | stefan2 | 2010-08-29 05:27:49 -0500 (Sun, 29 Aug 2010)

Minor improvement of the serialization API implementation: slightly faster
and with fixed comments.

* subversion/libsvn_subr/svn_temp_serializer.c
  (store_current_end_pointer): calculate the target offset only if we actually need it;
  improve commentary
  (svn_temp_serializer__set_null): improve commentary
  

r990535 | stefan2 | 2010-08-29 05:07:25 -0500 (Sun, 29 Aug 2010)

Although this is not likely to actually change the tool behavior
on any platform, it is technically correct to check NULL pointers
only after de-serializing ("resolving") them. Their serialized
representation may be (or become in future) different from (int)0.

* subversion/libsvn_fs_fs/temp_serializer.c
  (deserialize_svn_string, deserialize_checksum,
   deserialize_representation, svn_fs_fs__noderev_deserialize):
  check for NULL pointers only after they got resolved from their 
  serialized representation

* subversion/libsvn_fs_fs/id.c
  (svn_fs_fs__id_deserialize): dito

r990533 | stefan2 | 2010-08-29 04:42:42 -0500 (Sun, 29 Aug 2010)

Since the data buffer passed to de-serialization callbacks is already a 
private copy in pool, we don't need to create another copy in the DAG
de-serialization code. For all other objects, this has already been done
in r986832.

* subversion/libsvn_fs_fs/dag.c
  (svn_fs_fs__dag_deserialize): directly fix up the content in DATA

r988898 | stefan2 | 2010-08-25 03:47:41 -0500 (Wed, 25 Aug 2010)

Consulting the C++ ARM and various C online resources, I verified 
that structure size and array stride are guaranteed to be identical.
I.e. sizeof(T) is equivalent to sizeof(T[1]) -> simplify the the code.

* subversion/libsvn_fs_fs/temp_serializer.c
  (serialize_dir, serialize_txdelta_ops):
   use sizeof(T) instead of sizeof(T[1]); improve commentary

r988319 | stefan2 | 2010-08-23 16:36:07 -0500 (Mon, 23 Aug 2010)

Fix gcc / LINUX linkage: libsvn_fs functionality used by the individual FS modules
must reside in libsvn_fs_util. I verified that this works for VisualStudio as well.

* subversion/libsvn_fs/caching.c
  remove here
* subversion/libsvn_fs_util/caching.c
  copy to here

r987888 | stefan2 | 2010-08-22 07:19:57 -0500 (Sun, 22 Aug 2010)

svn:ignore file generated by VisualStudio 2010

* /
  ignore ipch folder , subversion_vcnet.opensdf, subversion_vcnet.sdf
* build/win32
  ignore *.log

r987887 | stefan2 | 2010-08-22 07:15:59 -0500 (Sun, 22 Aug 2010)

Remove unused local declaration.

* subversion/libsvn_subr/subst.c
  (translated_stream_skip): b was unused

r987886 | stefan2 | 2010-08-22 07:14:23 -0500 (Sun, 22 Aug 2010)

Fix a library dependency issue with the FSFS caching mechanism:
Main EXEs may only depend on generic libs (like FS) but not on the 
specific implementation modules (FSbase, FSFS). 

Thus, make the cache configuration interface public and FS generic
since it would also be suitable for BDB - even if it is not used today.
Likewise, the global caches can be shared throughout all SVN code.
Make the access to them an internal API.

* subversion/include/private/svn_fs_private.h
  (svn_fs__get_global_membuffer_cache,
   svn_fs__get_global_file_handle_cache): declare former local functions
   without the svn_fs__ part as private FS API
* subversion/include/svn_fs.h
  (svn_fs_cache_config_t, svn_fs_get_cache_config, svn_fs_set_cache_config):
   declare former FSFS-local declarations as part of the public FS API;
   rename them from svn_fs_fs__ form
* subversion/libsvn_fs/caching.c
  (cache_settings, svn_fs_get_cache_config, svn_fs_set_cache_config,
   svn_fs__get_global_membuffer_cache, 
   svn_fs__get_global_file_handle_cache):
   moved here from FSFS; rename; adapt to renames

* subversion/libsvn_fs_fs/fs_fs.h
  (svn_fs_fs__cache_config_t, svn_fs_fs__get_cache_config,
   svn_fs_fs__set_cache_config): remove
* subversion/libsvn_fs_fs/caching.c
  (cache_settings, svn_fs_fs__get_cache_config, get_global_membuffer_cache,
   get_global_file_handle_cache, svn_fs_fs__set_cache_config): remove
  (svn_fs_fs__initialize_caches): adapt to renames

* subversion/svnadmin/main.c
  remove SVN_LIBSVN_FS_LINKS_FS_FS checks
  (main): adapt to renames; silence an integer cast warning 
* subversion/svnserve/main.c
  remove SVN_LIBSVN_FS_LINKS_FS_FS checks
  (main): adapt to renames; silence some integer cast warning 

r987869 | stefan2 | 2010-08-22 06:37:38 -0500 (Sun, 22 Aug 2010)

Fix VisualStudio build: memory size calculation for variable size 
arrays is not portable.

* subversion/libsvn_fs_fs/temp_serializer.c
  (serialize_dir, serialize_txdelta_ops): fix array size calculation

Suggested by: Johan Corveleyn <jcorvel@gmail.com>

r987868 | stefan2 | 2010-08-22 06:32:10 -0500 (Sun, 22 Aug 2010)

Fix usage of qsort() to make it more portable. The comparison function
type name is not "standardized", so we can't use it for casting.

* subversion/libsvn_fs_fs/temp_serializer.c
  (compare_dirent_id_names):
   change signature such that qsort does not require casting it
  (serialize_dir): remove the cast operator

Found by: Johan Corveleyn <jcorvel@gmail.com>

r986832 | stefan2 | 2010-08-18 13:17:52 -0500 (Wed, 18 Aug 2010)

Reduce the (process globally) mutex-ed cache getter code by only
copying the serialized buffer in that critical section and performing
the de-serialization of the buffer content after the end of that section.

As a result, the server scalability increases slightly and de-serializer
functions get somewhat simplified as the data duplication has already
been done by the cache object.

* subversion/libsvn_subr/cache-memcache.c
  (memcache_get): already no local sync. but buffer must survive the
   end of the function
* subversion/libsvn_subr/cache-membuffer.c
  (membuffer_cache_set): fix compiler warning
  (membuffer_cache_get): duplicate cached data in CS, serialize outside
* subversion/libsvn_subr/cache-inprocess.c
  (inprocess_cache_get): dito
* subversion/libsvn_fs_fs/temp_serializer.c
  (svn_fs_fs__deserialize_txdelta_window, svn_fs_fs__deserialize_manifest,
   svn_fs_fs__deserialize_id, svn_fs_fs__deserialize_node_revision,
   svn_fs_fs__deserialize_dir_entries): remove the buffer duplication code


r986817 | stefan2 | 2010-08-18 12:44:19 -0500 (Wed, 18 Aug 2010)

Very often, a container is only accessed to process / use a single element
of that container. Copying these containers from cache as a whole can
be very expensive and result in O(N^2) runtime.

Therefore, provide partial getters for the two container types we cache
(dirs and manifests) and use them.

* subversion/libsvn_fs_fs/temp_serializer.h
  (svn_fs_fs__get_sharded_offset, svn_fs_fs__extract_dir_entry):
   declare partial / selective de-serialization functions
* subversion/libsvn_fs_fs/temp_serializer.c
  (svn_fs_fs__get_sharded_offset, svn_fs_fs__extract_dir_entry):
   implement them

* subversion/libsvn_fs_fs/fs_fs.h
  (svn_fs_fs__rep_contents_dir_partial): declare a new private getter function
   that returns the data more selectively than svn_fs_fs__rep_contents_dir
* subversion/libsvn_fs_fs/fs_fs.c
  (svn_fs_fs__rep_contents_dir_partial): implement that getter
  (get_packed_offset): simplify and speed up by using a partial getter

* subversion/libsvn_fs_fs/dag.h
  (svn_fs_fs__dag_dir_entry): declare a new private getter function
* subversion/libsvn_fs_fs/dag.c
  (svn_fs_fs__dag_dir_entry): implement the new getter using the new FSFS API
  (dir_entry_id_from_node): simplify and speed up by using the new API


r986608 | stefan2 | 2010-08-18 04:29:41 -0500 (Wed, 18 Aug 2010)

Extend the temp. serialization API with a function that de-serializes a given
pointer but returns the de-serialized value instead of writing it back into the
buffer.

* subversion/include/private/svn_temp_serialization.h
  (svn_temp_deserializer__ptr): declare new private API function
* subversion/libsvn_subr/svn_temp_serialization.c
  (svn_temp_deserializer__ptr): implement new private API function

r986605 | stefan2 | 2010-08-18 04:21:18 -0500 (Wed, 18 Aug 2010)

Extend cache API with a function that effectively allows partial de-
serialization as an alternative to the mandatory full de-serialization
in svn_cache__get. This can be used to read a single element from
a cached container, for instance.

* subversion/include/private/svn_cache.h
  (svn_cache__partial_getter_func_t): declare new callback type
  (svn_cache__get_partial): declare new API function

* subversion/libsvn_subr/cache.h
  (svn_cache__vtable_t): extend vtable
* subversion/libsvn_subr/cache.c
  (svn_cache__get_partial): implement new API function

* subversion/libsvn_subr/cache-memcache.c
  (memcache_get_partial): implement new API for memcached caches
  (memcache_vtable): extend vtable
* subversion/libsvn_subr/cache-membuffer.c
  (membuffer_cache_get_partial, svn_membuffer_cache_get_partial):
   implement new API for membuffer caches
  (membuffer_cache_vtable): extend vtable
* subversion/libsvn_subr/cache-inprocess.c
  (inprocess_cache_get_partial): implement new API for in-process caches
  (inprocess_cache_vtable): extend vtable


r986521 | stefan2 | 2010-08-17 18:11:13 -0500 (Tue, 17 Aug 2010)

Change serialized representation of pointers: instead of storing offsets
relative to the whole buffer, store the offset relative to the (local) structure
that immediately contains that pointer as an element.

Also, store directory content hashes in lexicographical order of their names.
This will soon be used to implement sub-entry cache access.

* subversion/libsvn_subr/svn_temp_serializer.c
  (store_current_end_pointer): store the pointer target as offset relative to the
   current struct's start offset (instead of the global position in the buffer)
* subversion/libsvn_fs_fs/id.c
  (deserialize_id_private, svn_fs_fs__id_deserialize): 
   resolve ptrs local to their containing struct
* subversion/libsvn_fs_fs/temp_serializer.c
  (deserialize_svn_string, deserialize_checksum, deserialize_representation,
   deserialize_dir, svn_fs_fs__noderev_deserialize,
   svn_fs_fs__deserialize_txdelta_window): dito
  (hash_data_t, compare_dirent_id_names, serialize_dir):
   serialize hash entries sorted by the hash key, i.e. the name


r986517 | stefan2 | 2010-08-17 17:55:30 -0500 (Tue, 17 Aug 2010)

Final move to deprecate svn_io_file_read_full: Fix documentation
and move the old function to deprecated.c

* subversion/include/svn_io.h
  (svn_io_file_read_full2): reformat commentary
  (svn_io_file_read_full): document as diff to svn_io_file_read_full2
* subversion/libsvn_subr/io.c
  (svn_io_file_read_full): remove
* subversion/libsvn_subr/deprecated.c
  (svn_io_file_read_full): insert here

r986492 | stefan2 | 2010-08-17 16:28:04 -0500 (Tue, 17 Aug 2010)

Fix signedness and integer size mismatches.

* subversion/libsvn_subr/stream.c
  (stream_readline_chunky, skip_handler_apr): fix integer usage

r986491 | stefan2 | 2010-08-17 16:25:28 -0500 (Tue, 17 Aug 2010)

Implement the deprecated svn_io_file_read_full as a wrapper around
svn_io_file_read_full2.

* subversion/libsvn_subr/io.c
  (svn_io_file_read_full): re-implement as simply calling svn_io_file_read_full2

r986485 | stefan2 | 2010-08-17 16:12:12 -0500 (Tue, 17 Aug 2010)

Replace the "move mark" concept by one that is strictly compatible with 
the stream semantics: skipping a number of chars from the stream. This
can always be emulated by reading that amount of data and discarding it.

* subversion/include/svn_io.h
  (svn_skip_fn_t): introduce as replacement for svn_io_move_mark_fn_t
  (svn_stream_set_skip): introduce as replacement for svn_io_skip_fn_t
  (svn_stream_skip): introduce as replacement for svn_stream_move_mark

* subversion/libsvn_subr/stream.c
  (svn_stream_t, svn_stream_create): adapt
  (svn_stream_set_skip, svn_stream_skip): implement
  (svn_stream_set_move_mark, svn_stream_move_mark): drop
  (stream_readline_chunky): adapt
  (skip_default_handler): new utility function implementing "skip" in terms of "read"
  (skip_handler_empty, skip_handler_disown, skip_handler_apr,
   skip_range_handler_apr, skip_handler_gz, skip_handler_checksum, 
   skip_handler_md5, skip_handler_stringbuf):
   implement "skip" for various types of stream
  (move_mark_handler_empty, move_mark_handler_disown,
   move_mark_handler_apr, move_mark_handler_stringbuf, skip_handler_string): drop 
  (svn_stream_empty, svn_stream_disown, stream_from_aprfile,
   svn_stream_from_aprfile_range_readonly, svn_stream_compressed, 
   svn_stream_checksummed2, svn_stream_checksummed,
   svn_stream_from_stringbuf, svn_stream_from_string):
   adapt vtable initialization
  (read_handler_apr, write_handler_apr): fix formatting

* subversion/libsvn_subr/subst.c
  (translated_stream_skip): implement "skip" for translated streams
  (translated_stream_move_mark): drop
  (svn_subst_stream_translated): adapt vtable initialization

r986465 | stefan2 | 2010-08-17 14:44:56 -0500 (Tue, 17 Aug 2010)

Deprecate svn_io_file_read_full and replace it with svn_io_file_read_full2
in (hopefully) all places it was used.

* subversion/include/svn_io.h
  (svn_io_file_read_full2): move in front of svn_io_file_read_full
  (svn_io_file_read_full): deprecate
* subversion/libsvn_diff/diff_file.c
  (read_chunk, map_or_read_file): use the newer API
* subversion/libsvn_fs_fs/fs_fs.c
  (create_rep_state_body, get_contents, read_handler_recover): dito
* subversion/libsvn_repos/reporter.c
  (read_string): dito
* subversion/libsvn_subr/hash.c
  (svn_hash_read): dito

r986459 | stefan2 | 2010-08-17 14:31:15 -0500 (Tue, 17 Aug 2010)

Introduce a new API function for creating txdelta streams that don't
calculate checksums. The actual support for that feature is already
available in txdelta_next_window.

Use that new API to skip the calculation of MD5 checkums for the MD5
on the server-side txdelta stream after the data has already been checked
in rep_read_contents. The data will usually be checksummed again by 
the client. Doing it once on both side should do.

* subversion/include/svn_delta.h
  (svn_txdelta_unchecked): declare new API function
* subversion/libsvn_delta/text_delta.c
  (txdelta_baton): clarify that checksum context may be NULL
  (txdelta_no_digest): new svn_txdelta_md5_digest_fn_t function returning NULL
  (svn_txdelta_unchecked): implement new API function
* subversion/libsvn_fs_fs/fs_fs.c
  (svn_fs_fs__get_file_delta_stream): use the new API

r986453 | stefan2 | 2010-08-17 14:04:21 -0500 (Tue, 17 Aug 2010)

Speed up EOF and keyword translation in two ways.

First, don't translate EOFs if the "is" equals the "ought". 
This reduces the number of calls to translate_newline as 
well as translate_write and ultimately apr_file_write.

Second, optimize the memcspn() replacement implementation
by checking 4 bytes at once. That results in about 3 CPU 
ticks / byte instead of roughly 8 to 10 ticks before.

* subversion/libsvn_delta/subst.c
  (translation_baton): add optimization info field
  (create_translation_baton): initialize the new member
  (eol_unchanged): new utiltiy function to compare EOLs
  (translate_chunk): faster scanning for interesting chars;
   don't translate unchaning EOLs

r985697 | stefan2 | 2010-08-15 10:58:54 -0500 (Sun, 15 Aug 2010)

Reduce the decompression overhead by eliminating unneeded allocation
and copy operations.

* subversion/libsvn_delta/svndiff.c
  (window_handler): use the new svn_stringbuf_create_empty function
  (zlib_decode): pass the input data buffer through to the output 
   if there was no compression
  (decode_window): use the new svn_stringbuf_create_empty function;
   adapt to zlib_decode signature change.

r985695 | stefan2 | 2010-08-15 10:47:38 -0500 (Sun, 15 Aug 2010)

Follow-up to r985601: fix a rare error condition that does not hurt functionality
but performance (massively, so). stream_readline_chunky failed to detect EOL
for longer lines until the end of the stream was encountered and would eventually
fall back to byte-wise parsing the data.

* subversion/libsvn_subr/stream.c
  (stream_readline_chunky): fix EOL detection for long lines

r985673 | stefan2 | 2010-08-15 08:20:42 -0500 (Sun, 15 Aug 2010)

Introduce a minimal svn_stringbuf_t creation function that does not
attempt to allocate any string buffer at all. So, it can be used to
reference existing string data.

* subversion/include/svn_string.h
  (svn_stringbuf_create_empty): declare new API function
* subversion/libsvn_subr/svn_string.c
  (svn_stringbuf_create_empty): implement new API function


r985670 | stefan2 | 2010-08-15 08:14:20 -0500 (Sun, 15 Aug 2010)

Fix / improve commentary.

* subversion/libsvn_fs_fs/fs_fs.c
  (cache_rep): add a comment
* subversion/libsvn_subr/stream.c
  (stream_from_aprfile): fix function name mentioned in comment

r985669 | stefan2 | 2010-08-15 08:09:33 -0500 (Sun, 15 Aug 2010)

Follow-up to r985014: replace apr_is_* in even more places, in particular in tests

Use svn_ctype_is* functions instead of the corresponding apr_is* wrappers
around the locale-dependent CRT implementations. Not all places are actually
performance critical but we should prefer consequent use of svn_* functions 
over a mixture of functions with different implementations.

* subversion/libsvn_fs_fs/fs_fs.c
  (check_format_file_buffer_numeric): use svn_ctype_is* instead of apr_is*
* subversion/libsvn_ra_svn/marshal.c
  (read_item): dito
* subversion/libsvn_subr/validate.c
  (svn_mime_type_validate): dito
* subversion/mod_dav_svn/repos.c
  (get_entry): dito
* subversion/svnserve/log-escape.c
  (escape_errorlog_item): dito
* subversion/tests/libsvn_delta/delta-window-test.h
  (delta_window_print): dito
* subversion/tests/svn_test_main.c
  (do_test_num): dito


r985606 | stefan2 | 2010-08-14 19:20:34 -0500 (Sat, 14 Aug 2010)

Increase the RA_SVN throughput by reducing the overhead in read_item for
packing received data into various structures.

* subversion/libsvn_ra_svn/marshal.c
  (readbuf_getchar): suggest inlining to the compiler
  (read_long_string): new fallback function containing the data receiving part
   of the former read_string
  (read_string): use special code that eliminates re-alloc operations for non-huge strings
  (read_item): ensure reasonable container sizes to minimize the number of re-allocs

r985605 | stefan2 | 2010-08-14 18:55:00 -0500 (Sat, 14 Aug 2010)

Minor optimization: don't use apr_psprintf for combining a number and
a string into a single string.

* subversion/libsvn_fs_fs/tree.c
  (locate_cache): use a faster key construction function

r985603 | stefan2 | 2010-08-14 18:51:38 -0500 (Sat, 14 Aug 2010)

Rename svn_fs_fs__dir_entries_(de)serialize to svn_fs_fs__(de)serialize_dir_entries
to match the naming scheme of all other (de-)serialization functions.

Note, that *both* declarations were accidentally present previously, so we just 
delete the obsolete one.

* subversion/libsvn_fs_fs/fs_fs.h
  (svn_fs_fs__dir_entries_serialize, svn_fs_fs__dir_entries_deserialize):
   drop obsolete function declarations; temp_serializer.h already contains the new ones
* subversion/libsvn_fs_fs/temp_serializer.c
  (svn_fs_fs__serialize_dir_entries, svn_fs_fs__deserialize_dir_entries): renamed
* subversion/libsvn_fs_fs/caching.c
  (svn_fs_fs__initialize_caches): adapt to the rename

r985602 | stefan2 | 2010-08-14 18:40:25 -0500 (Sat, 14 Aug 2010)

Fix a compiler warning ("ISO C90 forbids mixed declarations and code").

* subversion/libsvn_subr/stream.c
  (svn_stream_from_aprfile2, svn_stream__from_cached_file_handle):
   fix declaration order compiler warning.


r985601 | stefan2 | 2010-08-14 18:37:46 -0500 (Sat, 14 Aug 2010)

Management data in FSFS is stored as text lines. This patch speeds up
stream_readline by fetching larger chunks from the stream instead of
individual bytes. If the string does not support the features necessary
for that, fall back to the standard (=previously existing) implementation.

* subversion/libsvn_subr/stream.c
  (stream_readline_bytewise): renamed from the former stream_readline
  (stream_readline_chunky): new, faster alternative implementation
  (stream_readline): invoke the fastest suitable implementation

r985514 | stefan2 | 2010-08-14 10:46:13 -0500 (Sat, 14 Aug 2010)

Extend the stream API by three functions:
svn_stream_move_mark() to move an existing mark by some delta
svn_stream_supports_mark() tells whether getting, setting and moving marks 
                           is supported by this stream
svn_stream_buffered() tells whether a stream is buffered (and thus, efficient
                      local random access is possible)

* subversion/include/svn_io.h
  (svn_io_move_mark_fn_t, svn_io_buffered_fn_t):
   declare new vtable function pointers
  (svn_stream_set_move_mark, svn_stream_set_buffered):
   declare functions to set these vtable pointers
  (svn_stream_supports_mark, svn_stream_move_mark, svn_stream_buffered):
   declare new stream API functions

* subversion/libsvn_subr/stream.c
  (svn_stream_t): extend the vtable part by the new functions
  (svn_stream_create): add initialization code for the new vtable entries
  (svn_stream_set_move_mark, svn_stream_set_buffered):
   implement new vtable modifiers
  (svn_stream_supports_mark, svn_stream_buffered, svn_stream_buffered):
   implement new stream generic API functions
  (move_mark_handler_empty, buffered_handler_empty, svn_stream_empty):
   implement support for the new stream API in empty streams
  (move_mark_handler_disown, buffered_handler_disown, svn_stream_disown):
   implement support for the new stream API in disowned streams
  (move_mark_handler_apr, buffered_handler_apr, stream_from_aprfile,
   svn_stream_from_aprfile_range_readonly):
   implement support for the new stream API in APR file based streams
  (move_mark_handler_stringbuf, buffered_handler_stringbuf,
   svn_stream_from_stringbuf):
   implement support for the new stream API in stringbuf streams

* subversion/libsvn_subr/subst.c
  (translated_stream_move_mark, translated_stream_buffered,
   svn_subst_stream_translated):
   implement support for the new stream API in translated streams


r985500 | stefan2 | 2010-08-14 09:38:58 -0500 (Sat, 14 Aug 2010)

Speed up svn_io_open_unique_file3() on UNIX systems: There is no need to convert
the file path to locale encoding. Instead, just get the APR-compatible file name directly
from the APR.

* subversion/libsvn_subr/io.c
  (file_perms_set2): new local variant of file_perms_set() with different signature
  (svn_io_open_unique_file3): use the cheaper file_perms_set2

r985497 | stefan2 | 2010-08-14 09:26:42 -0500 (Sat, 14 Aug 2010)

Follow-up to r985487: There is still a case where we have to actively suppress EOF.
(This one got lost by me reformatting the code)

* subversion/libsvn_subr/stream.c
  (read_handler_apr): we still need to filter EOF errors for the getc() case

r985493 | stefan2 | 2010-08-14 08:59:40 -0500 (Sat, 14 Aug 2010)

Reimplement svn_io_file_write_full() for Windows in terms of apr_file_write() instead
of *_write_full() since we already implement the "_full" part here and don't need that
overhead on the APR layer.

* subversion/libsvn_subr/io.c
  (svn_io_file_write_full): reimplement using basic apr_file_write()

r985488 | stefan2 | 2010-08-14 08:28:42 -0500 (Sat, 14 Aug 2010)

Use svn_io_file_read_full2 to simplify and speed up file comparison.

* subversion/libsvn_subr/io.c
  (contents_identical_p): use *_read_full2() to minimize EOF detection overhead

r985487 | stefan2 | 2010-08-14 08:20:22 -0500 (Sat, 14 Aug 2010)

APR file I/O is a high frequency operation as the respective streams directly map their requests
onto the file layer. Thus, introduce svn_io_file_putc() as symmetrical counterpart to *_getc()
and use both for single char read requests because their APR implementation tends to involve
far less overhead.

Also, introduce and use svn_io_file_read_full2 that optionally doesn't create an error object
upon EOF that may be discarded by the caller anyways. This actually translates into significant
runtime savings because constructing the svn_error_t for file errors involves expensive locale
dependent functions.

* subversion/include/svn_io.h
  (svn_io_file_putc, svn_io_file_read_full2): declare new API functions
* subversion/libsvn_subr/svn_io.c
  (svn_io_file_putc, svn_io_file_read_full2): implement new API functions
* subversion/libsvn_subr/stream.c
  (read_handler_apr, write_handler_apr): use the I/O function best suitable for the request


r985482 | stefan2 | 2010-08-14 07:43:55 -0500 (Sat, 14 Aug 2010)

File I/O functions tend to be called frequently and many of them call the
do_io_file_wrapper_cleanup() function. Most of the times, the function code
will be almost a no-op making the call itself expensive. Thus, we suggest
to the compiler to inline this function.

* subversion/libsvn_subr/io.c
  (do_io_file_wrapper_cleanup): inline

r985477 | stefan2 | 2010-08-14 07:32:04 -0500 (Sat, 14 Aug 2010)

Eliminate all file system access in get_default_file_perms() for all but the first execution.

The only downside is that we don't detect FS permission changes made while the 
(client) process runs. Because such changes would cause race conditions and file I/O
errors anyway, we are not make things worse by omitting those tests.

* subversion/libsvn_subr/io.c
  (get_default_file_perms): store result in a singleton in the first run and bypass
  the FS access in all later runs

r985472 | stefan2 | 2010-08-14 07:03:54 -0500 (Sat, 14 Aug 2010)

Reduce the average number of iterations required by svn_io_open_uniquely_named()
and reduce the runtime of each iteration. 

Often, the initial temp. path suggested to the function is very generic one. If as a 
result of a crash, other failure or simple parallel execution, many temporary files
share the same prefix causing the algorithm to take many iterations to find an unused
(prefix, iteration, suffix) combination. Just throw an additional random element in
there and we are usually good after the second iteration.

* subversion/libsvn_subr/io.c
  (svn_io_open_uniquely_named): add a random element to the path once the first
  test failed; don't call UTF8 conversion  unnecessarily

r985046 | stefan2 | 2010-08-12 19:09:33 -0500 (Thu, 12 Aug 2010)

Follow-up to r985037: fix broken build (this change somehow got stuck in the editor).

* subversion/tests/libsvn_subr/stream-test.c
  (generate_test_bytes): fix function name to call

r985037 | stefan2 | 2010-08-12 18:27:40 -0500 (Thu, 12 Aug 2010)

The second (and probably last) mass change: svn_stringbuf_appendbytes has
a relatively large runtime overhead if we only add single bytes - which happens
frequently in svn's parser codes. 

Therefore, add a specialized and simpler variant that takes exactly one character
and use it in any suitable place.

* subversion/include/svn_string.h
  (svn_stringbuf_appendbyte): declare this new API function
* subversion/libsvn_subr/svn_string.c
  (svn_stringbuf_appendbyte): implement the new API function

* subversion/libsvn_client/deprecated.c
  (wrapped_receiver): use new API
* subversion/libsvn_diff/diff_file.c
  (output_unified_line): dito
* subversion/libsvn_diff/parse-diff.c
  (parse_hunk_header, hunk_readline): dito
* subversion/libsvn_fs_fs/lock.c
  (write_digest_file): dito
* subversion/libsvn_ra_svn/marshal.c
  (read_item): dito
* subversion/libsvn_repos/dump.c
  (write_hash_to_stringbuf): dito
* subversion/libsvn_subr/config_file.c
  (parse_value, parse_option, parse_section_name): dito
* subversion/libsvn_subr/prompt.c
  (prompt): dito
* subversion/libsvn_subr/quoprint.c
  (encode_bytes, decode_bytes): dito
* subversion/libsvn_subr/skel.c
  (unparse): dito
* subversion/libsvn_subr/stream.c
  (stream_readline): dito
* subversion/libsvn_subr/subst.c
  (keyword_printf): dito
* subversion/libsvn_wc/old-and-busted.c
  (read_str): dito
* subversion/libsvn_wc/props.c
  (svn_wc_canonicalize_svn_prop): dito
* subversion/svn/util.c
  (svn_cl__get_log_message): dito
* subversion/svndumpfilter/main.c
  (write_prop_to_stringbuf, output_revision, output_node): dito
* subversion/tests/libsvn_subr/skel-test.c
  (put_implicit_length_byte, put_implicit_length_all_chars, put_list_start, put_list_end): dito
* subversion/tests/libsvn_subr/stream-test.c
  (generate_test_bytes): dito

r985014 | stefan2 | 2010-08-12 17:13:56 -0500 (Thu, 12 Aug 2010)

Use svn_ctype_is* functions instead of the corresponding apr_is* wrappers
around the locale-dependent CRT implementations. Not all places are actually
performance critical but we should prefer consequent use of svn_* functions 
over a mixture of functions with different implementations.

* subversion/svn/util.c
  (svn_cl__get_log_message): use svn_ctype_is* instead of apr_is*
* subversion/libsvn_subr/utf.c
  (check_non_ascii): dito
* subversion/libsvn_subr/svn_string.c
  (string_first_non_whitespace, svn_stringbuf_strip_whitespace,
   svn_cstring_split_append): dito
* subversion/libsvn_subr/path.c
  (svn_path_is_uri_safe, svn_path_uri_decode): dito
* subversion/libsvn_subr/opt.c
  (parse_one_rev): dito
* subversion/libsvn_subr/io.c
  (svn_io_read_version_file): dito
* subversion/libsvn_subr/dirent_uri.c
  (canonicalize, svn_uri_is_canonical): dito
* subversion/libsvn_subr/config_file.c
  (skip_whitespace): dito
* subversion/libsvn_repos/load.c
  (svn_repos_parse_dumpstream2): dito
* subversion/libsvn_ra_svn/marshal.c
  (read_item): dito
* subversion/libsvn_client/add.c
  (trim_string): dito


r984984 | stefan2 | 2010-08-12 16:25:11 -0500 (Thu, 12 Aug 2010)

Eliminate redundant revprop lookups: Exports / checkouts often
contain multiple nodes from the same revision. Therefore, we
cache essential revision info in the report baton for as long as
the report is running.

As a neat side-effect, this will also fix inconsistencies created by
changing revprops (in a parallel request) while a report is running.

* subversion/libsvn_repos/reporter.c
  (revision_info_t): new structure containing essential revprops
  (report_baton_t): add revision info cache
  (get_revision_info): new function to transparently read the rev 
   info from cache including the fallback to a full FSFS revprop read.
  (delta_proplists): simplify by calling the new function
  (svn_repos_begin_report2): extend baton init code

r984973 | stefan2 | 2010-08-12 16:04:59 -0500 (Thu, 12 Aug 2010)

Cache (almost) all representations as full text. To handle contents longer
than one MB, let the caches decide upon their limits. Also, reps with
rb->len == 0 may have actual content but we can only cache them once
the stream gets closed because the content end won't be detected before
that.

* subversion/include/private/svn_cache.h
  (svn_cache__is_cachable): declare new API function

* subversion/libsvn_subr/cache.h
  (svn_cache__vtable_t): add method pointer for is_cachable
* subversion/libsvn_subr/cache.c
  (svn_cache__is_cachable): implement by forwarding to the actual cache object
* subversion/libsvn_subr/cache-memcache.c
  (memcache_is_cachable): implement is_cachable method
  (memcache_vtable): adapt vtable
* subversion/libsvn_subr/cache-membuffer.c
  (svn_membuffer_cache_is_cachable): implement is_cachable method
  (membuffer_cache_vtable): adapt vtable
* subversion/libsvn_subr/cache-inprocess.c
  (inprocess_cache_is_cachable): implement is_cachable method
  (inprocess_cache_vtable): adapt vtable

* subversion/libsvn_fs_fs/fs_fs.c
  (fulltext_size_is_cachable): reimplement, add fs_fs_data_t parameter for cache access
  (cache_rep): extract this utility function from rep_read_contents
  (rep_read_contents_close): make sure the fulltext gets cached at latest when the stream gets closed
  (rep_read_contents): call cache_rep
  (read_representation): adapt to called function signature change



r984928 | stefan2 | 2010-08-12 14:42:43 -0500 (Thu, 12 Aug 2010)

Merge r983766,r984927 from branches/performance. Approved by: danielsh
http://svn.haxx.se/dev/archive-2010-08/0219.shtml

Fix the root cause of an assertion triggered by exporting KDE /trunk:
Relative file paths need to be canonicalized when building URLs.

* subversion/libsvn_client/export.c
  (add_file): properly escape the file's URL

r984927 | stefan2 | 2010-08-12 14:38:11 -0500 (Thu, 12 Aug 2010)

As suggested in http://svn.haxx.se/dev/archive-2010-08/0219.shtm,
use svn_path_url_add_component2 instead of svn_uri_canonicalize
and svn_uri_join because the latter might double-escape the root url.

I tested the new code with  a repository url containing a space and
a relative path containing a space. It worked as suggested by the
comment.

(Fixing that on this branch to minimize the conflicts when merging
 to trunk and back again.) 

* subversion/libsvn_client/export.c:
  (add_file): use svn_path_url_add_component2

r984926 | stefan2 | 2010-08-12 14:29:23 -0500 (Thu, 12 Aug 2010)

Merge r983764 from branches/performance. Approved by: danielsh
http://svn.haxx.se/dev/archive-2010-08/0217.shtml

Fix an obvious typo in the path validation code.
It produces false negatives, i.e. certain malformed URIs won't be detected.

* subversion/libsvn_subr/dirent_uri.c
  (svn_uri_is_canonical): actually compare the chars following '%' instead
   of comparing '%'+1 and '%'+2.

r983770 | stefan2 | 2010-08-09 13:47:41 -0500 (Mon, 09 Aug 2010)

When reading cache-able full text from the DB, pre-allocate the
respective buffer to the expected size of the full text.

* subversion/libsvn_fs_fs/fs_fs.c
  (rep_read_get_baton): allocate just as much space for the fulltext
  buffer as will probably be needed to hold it.

r983766 | stefan2 | 2010-08-09 13:33:53 -0500 (Mon, 09 Aug 2010)

Fix the root cause of an assertion triggered by exporting KDE /trunk:
File names need to be canonicalized when forming URLs.

* subversion/libsvn_client/export.c
  (add_file): properly escape the file's URL

r983764 | stefan2 | 2010-08-09 13:27:49 -0500 (Mon, 09 Aug 2010)

Fix an obvious typo in the path validation code that is also present at /trunk.
It produces false negatives, i.e. certain malformed URIs won't be detected.

* subversion/libsvn_subr/dirent_uri.c
  (svn_uri_is_canonical): actually compare the chars following '%' instead
   of comparing '%'+1 and '%'+2.

r983760 | stefan2 | 2010-08-09 13:14:01 -0500 (Mon, 09 Aug 2010)

Add a cache for node_revision_t objects.

* subversion/libsvn_fs_fs/fs.h
  (fs_fs_data_t): add node_revision_cache member
* subversion/libsvn_fs_fs/fs_fs.c
  (get_noderev_cache_key, get_cached_node_revision_body,
   set_cached_node_revision_body): new functions to get / set noderevs 
from / to cache
  (get_node_revision_body): speed up using the noderev cache
* subversion/libsvn_fs_fs/caching.c
  (svn_fs_fs__initialize_caches): initialize the revnode cache; fix a comment


r983490 | stefan2 | 2010-08-08 17:00:32 -0500 (Sun, 08 Aug 2010)

Unify cache interfaces: the in-process cache now uses the generally more efficient
(de-)serialization methods to copy data from / to the cache.

* subversion/include/svn_cache.h
  (svn_cache__create_inprocess): switch from duplication to (de-)serialization functions
* subversion/libsvn_subr/cache-inprocess.c
  (inprocess_cache_t): replace dup_func member with (de-)serialization function pointers
  (cache_entry): add size member (to be provided for the callbacks)
  (duplicate_value): remove now unused function
  (inprocess_cache_get, inprocess_cache_set): call (de-)serialization functions instead of duplication
  (svn_cache__create_inprocess): implement API changes
* subversion/tests/libsvn_subr/cache-test.c
  (dup_revnum): drop now unused function
  (test_inprocess_cache_basic): adapt to new API

* subversion/libsvn_fs_fs/tree.c
  (make_txn_root): adapt to new cache API
* subversion/libsvn_fs_fs/dag.h
  (svn_fs_fs__dag_dup_for_cache): remove now unused declaration
* subversion/libsvn_fs_fs/dag.c
  (svn_fs_fs__dag_dup_for_cache): remove now unused implementation
* subversion/libsvn_fs_fs/caching.c
  (dup_id, serialize_id, deserialize_id, dup_dir_listing, manifest_serialize, manifest_deserialize,
   dup_pack_manifest): remove unused functions
  (svn_fs_fs__initialize_caches): adapt to new cache API

* subversion/libsvn_fs_fs/dag.c


r983488 | stefan2 | 2010-08-08 16:46:10 -0500 (Sun, 08 Aug 2010)

Switch all cached structure (de-)serialization code to the serializer framework.
IDs and DAG nodes cannot be serialized outside the .c files defining their private
data structures.

* subversion/libsvn_fs_fs/temp_serializer.h
  (svn_fs_fs__noderev_serialize, svn_fs_fs__noderev_deserialize, svn_fs_fs__serialize_manifest,
   svn_fs_fs__deserialize_manifest, svn_fs_fs__serialize_id, svn_fs_fs__deserialize_id,
   svn_fs_fs__serialize_node_revision, svn_fs_fs__deserialize_node_revision, 
   svn_fs_fs__serialize_dir_entries, svn_fs_fs__deserialize_dir_entries): 
  declare new (de)serialization functions
* subversion/libsvn_fs_fs/temp_serializer.c:
  (serialize_checksum, deserialize_checksum, serialize_representation, deserialize_representation,
   serialize_dir, deserialize_dir): new utility functions
  (svn_fs_fs__noderev_serialize, svn_fs_fs__noderev_deserialize, svn_fs_fs__serialize_manifest,
   svn_fs_fs__deserialize_manifest, svn_fs_fs__serialize_id, svn_fs_fs__deserialize_id,
   svn_fs_fs__serialize_node_revision, svn_fs_fs__deserialize_node_revision):
  implement new (de)serialization functions
  (svn_fs_fs__serialize_dir_entries, svn_fs_fs__deserialize_dir_entries): reimplement

* subversion/libsvn_fs_fs/id.h
  (svn_fs_fs__id_serialize, svn_fs_fs__id_deserialize): declare new (de)serialization functions
* subversion/libsvn_fs_fs/id.c
  (serialize_id_private, deserialize_id_private): new utility functions
  (svn_fs_fs__id_serialize, svn_fs_fs__id_deserialize): implement new (de)serialization functions

* subversion/libsvn_fs_fs/fs_fs.c
  (svn_fs_fs__dir_entries_serialize, svn_fs_fs__dir_entries_deserialize): 
  remove (got reimplemented in temp_serializer.*)

* subversion/libsvn_fs_fs/dag.c
  (svn_fs_fs__dag_serialize, svn_fs_fs__dag_deserialize): reimplement

r983474 | stefan2 | 2010-08-08 14:41:11 -0500 (Sun, 08 Aug 2010)

Memcached is often slower than a single file access. Thus, it is not inefficient
for most FSFS data structures: they can be read in a single request. Use the
membuffer cache for them instead.

* subversion/libsvn_fs_fs/caching.c
  (svn_fs_fs__initialize_caches): replace use of memcached with membuffer cache
  except for fulltext caches

r983437 | stefan2 | 2010-08-08 10:39:01 -0500 (Sun, 08 Aug 2010)

Add memory-cache-size and open-file-count command line options to svnadmin.
Enable txdelta window caching instead of fulltext caching (FSFS only).

* subversion/svnadmin/main.c
  (options_table): add new command line options
  (cmd_table): make them available for deltify, dump, load and verify sub-commands
  (svnadmin_opt_state): add members to hold the new options
  (main): default to larger caches then the FSFS default; parse new CL options; 
  configure caches

r983430 | stefan2 | 2010-08-08 10:34:48 -0500 (Sun, 08 Aug 2010)

Implement txdelta window caching. This will be used by svnadmin only
since fulltext caches are mainly ineffective during dump etc. The opposite
is true for server functions (get / set fulltext data).

The cache processing for txdelta windows is relatively straightforward 
but requires some auxiliary information. Thus, we wrap them in the
svn_fs_fs__txdelta_cached_window_t structure.

* subversion/libsvn_fs_fs/temp_serializer.h
  (svn_fs_fs__combine_number_and_string, svn_fs_fs__combine_two_numbers):
  declare key construction utility functions
  (svn_fs_fs__txdelta_cached_window_t): auxilliary struct used to represent cached txdelta windows
  (svn_fs_fs__serialize_txdelta_window, svn_fs_fs__deserialize_txdelta_window):
  declare API to (de)serialize txdelta windows for caching
* subversion/libsvn_fs_fs/temp_serializer.c
* subversion/libsvn_fs_fs/fs_fs.c
  (rep_state): add a reference to the txdelta window cache to use
  (create_rep_state_body): init that reference
  (get_window_key, get_cached_window, set_cached_window):
  utility functions to get / set txdelta window objects from / to the cache
  (read_window): speed up using the window cache
* subversion/libsvn_fs_fs/fs.h
  (fs_fs_data_t): add txdelta_window_cache window
* subversion/libsvn_fs_fs/caching.c
  (svn_fs_fs__initialize_caches): add initialization of the txdelta window cache

r983406 | stefan2 | 2010-08-08 07:46:53 -0500 (Sun, 08 Aug 2010)

Introduce svn_temp_serializer__set_NULL to allow resetting pointers
in the serialized data without modifying the source structure.

* subversion/include/private/svn_temp_serializer.h
  (svn_temp_serializer__set_NULL): declare new function
* subversion/libsvn_subr/svn_temp_serializer.c
  (svn_temp_serializer__set_NULL): implement


r983398 | stefan2 | 2010-08-08 07:20:40 -0500 (Sun, 08 Aug 2010)

Simplify serialization of structures that may be a root structure as well
as being embedded into others. So, no longer require the initialization
routine to be provided with a root structure.

* subversion/libsvn_subr/svn_temp_serializer.c
  (svn_temp_serializer__init): don't try to store the root struct if none was given
  (store_current_end_pointer): don't try to update the parent reference, if there is none
* subversion/include/private/svn_temp_serializer.h
  (svn_temp_serializer__init, svn_temp_serializer__push, 
   svn_temp_serializer__add_string): explain the new options

r983385 | stefan2 | 2010-08-08 06:29:23 -0500 (Sun, 08 Aug 2010)

Since svn_file_handle_cache__handle_t is not part of the public API,
svn_stream_from_cached_file_handle shouldn't be as well.

Therefore, move the declaration to a private header and rename the
function to indicate that it is private, too. Because the streams created
are performance critical and to minimize copy'n'paste code, the stream
implementation remains in stream.c .

* subversion/libsvn_subr/stream.c
  (svn_stream__from_cached_file_handle): rename from 
svn_stream_from_cached_file_handle
* subversion/libsvn_fs_fs/fs_fs.c
  (get_node_revision_body, read_window): adapt to renamed API
* subversion/include/svn_io.h
  (svn_stream_from_cached_file_handle, svn_file_handle_cache__handle_t): 
remove here
* subversion/include/private/svn_file_handle_cache.h
  (svn_stream__from_cached_file_handle): moved function declaration 


r982417 | stefan2 | 2010-08-04 17:23:34 -0500 (Wed, 04 Aug 2010)

Instead of using plain APR files, use cached file handles for FSFS
read operations.

* subversion/libsvn_fs_fs/fs_fs.c
  (DEFAULT_FILE_COOKIE, REF_FILE_COOKIE): define file cookies
  (svn_fs_fs__path_rev_absolute): use cache lookup to increase performance
  (sync_file_handle_cache): new utility function
  (open_pack_or_rev_file, open_and_seek_revision, open_and_seek_transaction,
  open_and_seek_representation):
  use cached file handles instead of APR file handles; add / change parameters
  (get_node_revision_body, svn_fs_fs__rev_get_root, svn_fs_fs__paths_changed,
   recover_get_largest_revision, recover_body): implement using file handle cache
  (svn_fs_fs__put_node_revision, write_next_ids, svn_fs_fs__set_entry,
   svn_fs_fs__add_change, rep_write_contents_close, svn_fs_fs__set_proplist,
   commit_body, commit_obliteration_body): clear cache after DB modification
  (rep_state): add  cached file handle member
  (create_rep_state_body, read_window, get_contents, 
   svn_fs_fs__get_file_delta_stream): read reps from cached file handles


r982391 | stefan2 | 2010-08-04 15:22:10 -0500 (Wed, 04 Aug 2010)

Upon second thought, svn_stream_from_aprfile3 should be renamed to 
svn_stream_from_cached_file_handle.

* subversion/include/svn_io.h
  (svn_stream_from_cached_file_handle): renamed from svn_stream_from_aprfile3
* subversion/libvn_subr/stream.c
  (svn_stream_from_cached_file_handle): renamed from svn_stream_from_aprfile3

r982375 | stefan2 | 2010-08-04 14:27:41 -0500 (Wed, 04 Aug 2010)

Introduce a variant of svn_stream_from_aprfile2, that takes cached file handles
instead of APR file handles.

* subversion/include/svn_io.h
  (svn_stream_from_aprfile3): declare new API function
* subversion/libsvn_subr/stream.c
  (baton_apr): add cached_handle member
  (close_handler_cached_handle): new close() function
  (stream_from_aprfile): extract generic part from svn_stream_from_aprfile2
  (svn_stream_from_aprfile2): call stream_from_aprfile for most of the init code
  (svn_stream_from_aprfile3): implement new API function

r982360 | stefan2 | 2010-08-04 13:56:24 -0500 (Wed, 04 Aug 2010)

Introduce a process-global open file handle cache and make it available to FSFS code.
A reference in fs_fs_data_t is used instead of a global "svn_get_fh_cache" function
to allow for more fine-grained configuration in future.

* subversion/libsvn_fs_fs/fs.h
  (fs_fs_data_t): add file_handle_cache member; fix indentation
* subversion/libsvn_fs_fs/caching.c
  (get_global_file_handle_cache): new function managing the cache singleton
  (svn_fs_fs__set_cache_config): auto-init the file handle cache as well
  (svn_fs_fs__initialize_caches): assign the file handle cache to the FSFS context

r982355 | stefan2 | 2010-08-04 13:43:29 -0500 (Wed, 04 Aug 2010)

Fix an issue with fulltext caching already present in production SVN:
APR pools often won't reuse memory fragments if they are larger
than 80kB. Using string buffers while reconstructing fulltexts does 
exactly The Bad Thing: request large buffers of various sizes that
APR pools will often not reuse due to their differing and often just
a tad bit too small size.

Use an allocator to limit the amount of unused memory fragments
held by the root pools.

* subversion/svnserve/main.c
  (main): limit the idle memory allocated per request root pool to 4 MB.


r982057 | stefan2 | 2010-08-03 17:46:10 -0500 (Tue, 03 Aug 2010)

Add "compression", "memory-cache-size" and "open-file-count"
command line parameters to svnserve. The latter two are only
available (on the CL) if FSFS is supported.

Currently, the file handle cache is not being used but will be soon.

* subversion/svnserve/server.h
  (serve_params_t): add FSFS data and file handle cache sizes
* subversion/svnserve/main.c
  (svnserve__options): add command line option definitions
  (main): parse the new parameters; update the FSFS cache settings

r982043 | stefan2 | 2010-08-03 16:56:24 -0500 (Tue, 03 Aug 2010)

Incorporate Blair's feedback from http://svn.haxx.se/dev/archive-2010-08/0071.shtml
These are mainly commentary fixes and style issues.

* subversion/libsvn_fs_fs/fs_fs.h
  (svn_fs_fs__cache_config_t): fix commentary
  (svn_fs_fs__set_cache_config): make parameter const; extend commentary
* subversion/libsvn_fs_fs/caching.c
  (internal_get_cache_config): remove
  (cache_settings): introduce this static variable instead
  (svn_fs_fs__get_cache_config, svn_fs_fs__set_cache_config): adapt
  (get_global_membuffer_cache): adapt; extend commentary

r981828 | stefan2 | 2010-08-03 06:41:16 -0500 (Tue, 03 Aug 2010)

Revert r981665 because there is no such naming convention.
I guess it is my fault when I blindly follow advise .. *sigh*

r981827 | stefan2 | 2010-08-03 06:38:01 -0500 (Tue, 03 Aug 2010)

Make the compression level used over RA_SVN selectable by serve_params_t.
Disable wire compression entirely for level 0, i.e. skip zlib in that case.
Because level 0 is only useful for maximizing throughput in broadband LANs
(>= 1GB/s), we won't advertise the the SVNDIFF1 capability in that case, either.
Thus, clients will send plaintext data as well during commits, etc.

* subversion/svnserve/server.h
  (serve_params_t): add compression_level parameter
* subversion/svnserve/serve.c
  (file_rev_handler): switch to new diff API; disable compression for level 0
  (serve): don't advertise SVN_RA_SVN_CAP_SVNDIFF1 capability if we
  want uncompressed communication
* subversion/svnserve/main.c
  (main): init new compression level parameter

* subversion/include/svn_ra_svn.h
  (svn_ra_svn_create_conn2): declare new API function taking the compression level
  as additional parameter

* subversion/libsvn_ra_svn/ra_svn.h
  (svn_ra_svn_conn_st): add compression_level parameter
* subversion/libsvn_ra_svn/marshal.c
  (svn_ra_svn_create_conn2, svn_ra_svn_compression_level): implement new API function
  (svn_ra_svn_create_conn): simply wrap the new API
* subversion/libsvn_ra_svn/editorp.c
  (ra_svn_apply_textdelta): switch to new diff API; disable compression for level 0

r981684 | stefan2 | 2010-08-02 15:51:35 -0500 (Mon, 02 Aug 2010)

Bring the membuffer cache to its first use for the full text cache.
Also, provide functions to get / set the FSFS cache configuration
although not all of it is supported, yet.

* subversion/libsvn_fs_fs/fs_fs.h
  (svn_fs_fs__cache_config_t): introduce this new config struct
  (svn_fs_fs__get_cache_config, svn_fs_fs__set_cache_config): 
  declare new functions to get/set the FSFS cache configuration
* subversion/libsvn_fs_fs/caching.c
  (internal_get_cache_config, get_global_membuffer_cache):
  new singleton management functions
  (svn_fs_fs__get_cache_config, svn_fs_fs__set_cache_config): 
  implement new functions to get/set the FSFS cache configuration
  (svn_fs_fs__initialize_caches): use the membuffer cache for
  full texts if memcached has not been configured


r981665 | stefan2 | 2010-08-02 14:36:59 -0500 (Mon, 02 Aug 2010)

Rename all svn_* in include/private to svn__* as requested in 
http://svn.haxx.se/dev/archive-2010-08/0043.shtml

* subversion/include/private/svn_file_handle_cache.h
  (svn_*): rename to svn__*
* subversion/include/private/svn_temp_serializer.h
  (svn_*): rename to svn__*

* subversion/libsvn_subr/svn_file_handle_cache.c
  (svn_*): rename to svn__*
  (lock_cache, unlock_cache, find_first, internal_file_open, internal_close_file,
   open_entry, close_oldest_idle, auto_close_oldest): adapt signatures
  (close_handle_before_cleanup): adapt implementation
* subversion/libsvn_subr/svn_temp_serializer.c
  (svn_*): rename to svn__*


r981287 | stefan2 | 2010-08-01 14:36:07 -0500 (Sun, 01 Aug 2010)

Add compression_level argument to svn_txdelta_to_svndiff.

* subversion/include/svn_delta.h
  (SVNDIFF1_COMPRESS_LEVEL): make define public; moved from svndiff.c to here
  (svn_txdelta_to_svndiff3): new API function; adds compression_level parameter

* subversion/libsvn_delta/svndiff.c
  (SVNDIFF1_COMPRESS_LEVEL): drop; moved to header
  (NORMAL_BITS, LENGTH_BITS): drop unused defines
  (encoder_baton): add compression_level member
  (zlib_encode): add compression_level parameter and pass it to zlib
  (window_handler): pass compression level from encoder baton to zlib_encode
  (svn_txdelta_to_svndiff3): implement
  (svn_txdelta_to_svndiff2): implement as wrapper around svn_txdelta_to_svndiff3
  (svn_txdelta_to_svndiff): call svn_txdelta_to_svndiff3

* subversion/tests/libsvn_delta/random-test.c
  (random_test, do_random_combine_test): use newest API variant with
  different compression levels

* subversion/tests/libsvn_delta/svndiff-test.c
  (main): use newest API variant

r981204 | stefan2 | 2010-08-01 06:13:46 -0500 (Sun, 01 Aug 2010)

Rename all svn_file_cache_* to svn_file_handle_cache_*, including the
file names as requested in http://svn.haxx.se/dev/archive-2010-08/0005.shtml

* rename subversion/include/private/svn_file_cache.h to
  subversion/include/private/svn_file_handle_cache.h
* rename subversion/libsvn_subr/svn_file_cache.c to
  subversion/libsvn_subr/svn_file_handle_cache.c

* subversion/include/private/svn_file_handle_cache.h
  (svn_file_cache_*): rename to svn_file_handle_cache_*

* subversion/libsvn_subr/svn_file_handle_cache.c
  (svn_file_cache_*): rename to svn_file_handle_cache_*
  (lock_cache, unlock_cache, find_first, internal_file_open, internal_close_file,
   open_entry, close_oldest_idle, auto_close_oldest): adapt signatures
  (close_handle_before_cleanup): adapt implementation


r981194 | stefan2 | 2010-08-01 05:02:00 -0500 (Sun, 01 Aug 2010)

Remove typedefs for various generic pointer types as requested in
http://svn.haxx.se/dev/archive-2010-08/0007.shtml .

* subversion/include/private/svn_temp_serializer.h
  (PCPCSTR, PCPCVOID, PPVOID): drop typedefs
  (svn_temp_serializer__push, svn_temp_serializer__add_string,
   svn_temp_deserializer__resolve): replace typedef with verbatim types

* subversion/libsvn_subr/svn_temp_serializer.c
  (svn_temp_serializer__push, svn_temp_serializer__add_string,
   svn_temp_deserializer__resolve): replace typedef with verbatim types
  (svn_temp_serializer__init): fix formatting


r981189 | stefan2 | 2010-08-01 04:35:10 -0500 (Sun, 01 Aug 2010)

In reaction to http://svn.haxx.se/dev/archive-2010-08/0000.shtml and
http://svn.haxx.se/dev/archive-2010-08/0007.shtml , make crystal clear
that this is a serialization utility for temporary data, ONLY.

Basically, rename all svn_serializer_* to svn_temp_serializer_*,
including the file names. Also, put a Doxygen note at the beginning 
of the header file.

* rename subversion/include/private/svn_serializer.h to
  subversion/include/private/svn_temp_serializer.h
* rename subversion/libsvn_subr/svn_serializer.c to
  subversion/libsvn_subr/svn_temp_serializer.c

* subversion/include/private/svn_temp_serializer.h
  enhance Doxygen commentary
  (svn_serializer_*): rename to svn_temp_serializer_*
  (svn_deserializer_*): rename to svn_temp_deserializer_*
* subversion/libsvn_subr/svn_temp_serializer.c
  (svn_serializer_*): rename to svn_temp_serializer_*
  (svn_deserializer_*): rename to svn_temp_deserializer_*


r981091 | stefan2 | 2010-07-31 15:46:59 -0500 (Sat, 31 Jul 2010)

Introduce a private file handle cache API and provide an implementation.

* subversion/include/private/svn_file_cache.h
  (svn_file_cache_t, svn_file_cache__handle_t): introduce new opaque data types
  (svn_file_cache__open, svn_file_cache__has_file, 
   svn_file_cache__get_apr_handle, svn_file_cache__get_name,
   svn_file_cache__close, svn_file_cache__flush, svn_file_cache__create_cache):
  declare new private API functions
* subversion/libsvn_subr/svn_file_cache.c
  (cache_entry_t, entry_link_t, entry_list_t): define new internal data structures
  (svn_file_cache_t, svn_file_cache__handle_t): define data types used in the API
  (lock_cache, unlock_cache, init_list, init_link, link_link, unlink_link,
   get_previous_entry, get_next_entry, append_to_list, remove_from_list,
   find_first, auto_close_cached_handle, internal_file_open, internal_close_file,
   close_handle_before_cleanup, open_entry, close_oldest_idle, auto_close_oldest,
   pointer_is_closer): new utility functions
  (svn_file_cache__open, svn_file_cache__has_file, 
   svn_file_cache__get_apr_handle, svn_file_cache__get_name,
   svn_file_cache__close, svn_file_cache__flush, svn_file_cache__create_cache):
  implement new private API functions


r981090 | stefan2 | 2010-07-31 15:28:49 -0500 (Sat, 31 Jul 2010)

Introduce a simple serialization framework that simplifies and
speeds up serialization of most data structures used by FSFS,
for instance.

* subversion/include/private/svn_serializer.h
  (PCPCSTR, PCPCVOID, PPVOID): introduce useful typedefs
  (svn_serializer__context_t): introduce new data type
  (svn_serializer__init, svn_serializer__push, svn_serializer__pop, 
   svn_serializer__add_string, svn_serializer__get): declare new
  serialization functions
  (svn_deserializer__resolve): declare new deserialization function
* subversion/libsvn_subr/svn_serializer.c
  (source_stack_t, svn_serializer__context_t): define structs
  (align_buffer_end, store_current_end_pointer): new utility functions
  (svn_serializer__init, svn_serializer__push, svn_serializer__pop, 
   svn_serializer__add_string, svn_serializer__get,
   svn_deserializer__resolve): implement functions


r981087 | stefan2 | 2010-07-31 15:22:50 -0500 (Sat, 31 Jul 2010)

Code formatting only: Inserting a few spaces where they belong.

* subversion/libsvn_subr/cache-membuffer.c
  (membuffer_cache_set, membuffer_cache_get, 
   svn_cache__create_membuffer_cache): minor formatting fixes

r980118 | stefan2 | 2010-07-28 11:42:10 -0500 (Wed, 28 Jul 2010)

Rename membuffer_cache_t to svn_membuffer_t as requested in 
http://svn.haxx.se/dev/archive-2010-07/0625.shtml It is not using
the svn_cache__ prefix, though, because it doesn't have anything
to do with the svn_cache__* interface. It is more akin to 
svn_memcache_t.

Once we are at it, we also fix some pointer formatting "t* x" to "t *x".

* subversion/include/private/svn_cache.h
  (svn_membuffer_t): renamed from membuffer_cache_t
  (svn_cache__membuffer_cache_create, svn_cache__create_membuffer_cache): 
  adapt signature to rename
* subversion/libsvn_subr/cache-membuffer.c
  (svn_membuffer_t): renamed from membuffer_cache_t
  (lock_cache, unlock_cache, get_entry, move_entry, ensure_data_insertable): 
  adapt signature to rename and fix pointer formatting
  (get_index, drop_entry, insert_entry, get_group_index, find_entry, 
   membuffer_cache_set, membuffer_cache_get, svn_membuffer_cache_t, 
   svn_cache__create_membuffer_cache): adapt signature to rename 
  (svn_cache__membuffer_cache_create, svn_cache__membuffer_cache_create): 
  adapt signature as well as implementation to rename and fix pointer formatting


r979193 | stefan2 | 2010-07-26 03:30:08 -0500 (Mon, 26 Jul 2010)

Provide a memcached-like implementation of svn_cache_t that does not have
the same latency and reliability issues. Detailed descriptions can be found in
the .c file.

* subversion/include/private/svn_cache.h
  (membuffer_cache_t): new opaque type
  (svn_cache__membuffer_cache_create, svn_cache__create_membuffer_cache): 
   declare new functions
* subversion/libsvn_subr/cache-membuffer.c
  (GROUP_SIZE, KEY_SIZE, ITEM_ALIGNMENT, ALLOCATIONS_PER_POOL_CLEAR):
   new defines
  (entry_t, entry_group_t, membuffer_cache_t, svn_membuffer_cache_t): 
   new structures
  (align_entry, lock_cache, unlock_cache, get_entry, get_index, drop_entry,
   insert_entry, get_group_index, find_entry, move_entry, ensure_data_insertable,
   membuffer_cache_set, membuffer_cache_get):
  new internal functions implementing the membuffer cache
  (combine_key, svn_membuffer_cache_get, svn_membuffer_cache_set,
   svn_membuffer_cache_iter, serialize_svn_stringbuf, deserialize_svn_stringbuf):
  implement the svn_cache_t interface 
  (svn_cache__membuffer_cache_create, svn_cache__create_membuffer_cache): 
   implement public functions
  (membuffer_cache_vtable): new static variable

r964568 | stefan2 | 2010-07-15 15:20:40 -0500 (Thu, 15 Jul 2010)

Open performance improvement integration branch.
The core changes will come from https://tortoiseanalyze.svn.sourceforge.net/svnroot/tortoiseanalyze/OptimizedSVN/branches/prototype.
Those changes will be regrouped for the sake of review,
extensive review is to be added and tests to be written
where they are due.

r964557 | stefan2 | 2010-07-15 15:05:55 -0500 (Thu, 15 Jul 2010)

add stefan2 to COMMITTERS

r956593 | julianfoad | 2010-06-21 08:56:30 -0500 (Mon, 21 Jun 2010)

Optimize the copies performed by svn_txdelta_apply_instructions().

svn_txdelta_apply_instructions() is relatively slow for long instruction
sequences copying small pieces of data.  This is particularly visible in
"format 2" FSFS repositories.

Two kinds of copy are involved.  For simple copies, the system's memcpy()
is used, which is fast for long lengths; this patch bypasses it for very
short lengths.  For intentionally overlapping copies, a custom loop is used,
which was already fast for very short lengths; this patch adds a code path
that is fast for longer lengths.

* subversion/libsvn_delta/text_delta.c
  (fast_memcpy, patterning_copy): New functions, optimized for our specific
    workload.
  (svn_txdelta_apply_instructions): Use fast_memcpy() and patterning_copy().

Patch by: Stefan Fuhrmann <stefanfuhrmann{_AT_}alice-dsl.de>
(I tweaked the comments and split Stefan's patch into two parts, of which
this is the first.)


r941243 | julianfoad | 2010-05-05 06:01:40 -0500 (Wed, 05 May 2010)

Various local optimizations.  These opportunities became visible after
significantly optimizing other parts of svn.  The total savings for a 'svn
export' is almost 3 percent.

The largest savings can be attributed to the svndiff.c
changes (~1.5%) and the checksum parser (~1%).

* subversion/include/svn_delta.h
  (enum svn_delta_action): Document that these enum values must match the
    encoding values.

* subversion/libsvn_delta/compose_delta.c
  (search_offset_index, copy_source_ops): Use size_t to index memory. This
    is mainly a consistency fix but may actually result in slightly higher
    performance due to fewer conversions.

* subversion/libsvn_delta/svndiff.c
  (decode_file_offset, decode_size): Use slightly more efficient
    formulations.
  (decode_instruction): Directly map action codes, avoiding a 'switch'.

* subversion/libsvn_subr/checksum.c
  (svn_checksum_parse_hex): Eliminate calls to locale-aware CRT functions.
    At least with MS compilers, these are very expensive.

* subversion/libsvn_subr/stream.c
  (stream_readline): Hoist 'numbytes' from the loop: it is invariant until
    EOF.

Patch by: Stefan Fuhrmann <stefanfuhrmann{_AT_}alice-dsl.de>


r921057 | julianfoad | 2010-03-09 13:04:00 -0600 (Tue, 09 Mar 2010)

Speed up input stream processing in config parser and others that read
single bytes from a stream.

* subversion/libsvn_subr/subst.c
  (translated_stream_read): Add an optimized code path for single byte read
    requests.

Patch by: Stefan Fuhrmann <stefanfuhrmann{_AT_}alice-dsl.de>