<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>kafka0102的边城客栈</title>
	<atom:link href="http://www.kafka0102.com/feed" rel="self" type="application/rss+xml" />
	<link>http://www.kafka0102.com</link>
	<description>要有最朴素的生活与最遥远的梦想，即使明日天寒地冻、路远马亡。</description>
	<lastBuildDate>Tue, 24 Apr 2012 21:29:14 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.9.1</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>总结获取html页面遇到的几个case</title>
		<link>http://www.kafka0102.com/2012/04/457.html</link>
		<comments>http://www.kafka0102.com/2012/04/457.html#comments</comments>
		<pubDate>Tue, 24 Apr 2012 21:29:14 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[framework]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=457</guid>
		<description><![CDATA[写段html页面获取的代码很简单了，其实也不需我多说。不过，要能正确的获取html内容似乎也不是很容易。自tuicool.com内测以来，不断发现需要改进的细节问题，以致之前的新功能开发计划只能往后推。昨天就发现网站推荐的文章中有乱码的情况，站点来源竟是大名鼎鼎的mashable.com，而且收录的该站点文章多数都是乱码。所以，赶紧的在新文章抓取时对乱码情况做了检查，至少不能暴露给用户啊。接下来就分析和实验xhttpproxy的代码，结果发现，当不给httpclient传“user-agent"头时，返回的结果是ok的，这种情况实际上httpclient加了自己的“user-agent"头，而我试了其他多种“user-agent"都不行，不知道mashable.com对请求头做了什么样的校验策略啊，竟然对httpclient这么友好（如果使用java自带的Httpconnection也是可以的，它会默认传个带有java标识的一串字符）。前一阵子还遇到一个case，需要设置请求头”Accept“为"*/*"才能正常返回结果，否则连接就会被重置。如果我深入的检查抓取来的数据，可能还有不少bad case。这其实就有些让人头疼，因为很多网站对http头做了检查但策略又各不相同，加之我这方面的经验也不丰富，只能遇到一个分析一个。说起乱码，又想起charset，也是不让人省心，还是有个别charset解析错误的情况难解决，使用的icu不是很给力啊。]]></description>
			<content:encoded><![CDATA[<p>写段html页面获取的代码很简单了，其实也不需我多说。不过，要能正确的获取html内容似乎也不是很容易。自tuicool.com内测以来，不断发现需要改进的细节问题，以致之前的新功能开发计划只能往后推。昨天就发现网站推荐的文章中有乱码的情况，站点来源竟是大名鼎鼎的mashable.com，而且收录的该站点文章多数都是乱码。所以，赶紧的在新文章抓取时对乱码情况做了检查，至少不能暴露给用户啊。接下来就分析和实验xhttpproxy的代码，结果发现，当不给httpclient传“user-agent&#8221;头时，返回的结果是ok的，这种情况实际上httpclient加了自己的“user-agent&#8221;头，而我试了其他多种“user-agent&#8221;都不行，不知道mashable.com对请求头做了什么样的校验策略啊，竟然对httpclient这么友好（如果使用java自带的Httpconnection也是可以的，它会默认传个带有java标识的一串字符）。前一阵子还遇到一个case，需要设置请求头”Accept“为&#8221;*/*&#8221;才能正常返回结果，否则连接就会被重置。如果我深入的检查抓取来的数据，可能还有不少bad case。这其实就有些让人头疼，因为很多网站对http头做了检查但策略又各不相同，加之我这方面的经验也不丰富，只能遇到一个分析一个。说起乱码，又想起charset，也是不让人省心，还是有个别charset解析错误的情况难解决，使用的icu不是很给力啊。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2012/04/457.html/feed</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>开源分词程序ki-analyzer启动</title>
		<link>http://www.kafka0102.com/2011/06/453.html</link>
		<comments>http://www.kafka0102.com/2011/06/453.html#comments</comments>
		<pubDate>Sat, 18 Jun 2011 04:20:42 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[framework]]></category>
		<category><![CDATA[ki]]></category>
		<category><![CDATA[ki-analyzer]]></category>
		<category><![CDATA[中文分词]]></category>
		<category><![CDATA[分词程序]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=453</guid>
		<description><![CDATA[ki-analyzer 是基于ik-analyzer 修改而来的分词程序，所以名字上只是简单的变了一下，源码方面还是沿用了ik的包名等。之所以在ik-analyzer之上山寨另一个轮子，也实在是因为我的需求ik-analyzer不能很好满足，并且功能、设计、改动方面较ik-analyzer有很大出入，所以另起山寨。ik-analyzer项目现在还活着，前不久发布了新版，貌似只是源码实现细节的调整，项目也不是很活跃。当然，国内的几个开源分词程序基本都停滞了。ki-analyzer程序是上周修改的，这周实际测试了一下，只能说bug方面基本稳定，但尚需进一步检验。因为一些想要的功能暂时还用不上，所以也没着急做。关于ki-analyzer的详细情况可以参考项目首页信息，对该项目感兴趣或有问题的朋友可以联系我。]]></description>
			<content:encoded><![CDATA[<p><a href="http://code.google.com/p/ki-analyzer/" target="_blank">ki-analyzer</a> 是基于<a href="http://code.google.com/p/ik-analyzer/" target="_blank">ik-analyzer</a> 修改而来的分词程序，所以名字上只是简单的变了一下，源码方面还是沿用了ik的包名等。之所以在ik-analyzer之上山寨另一个轮子，也实在是因为我的需求ik-analyzer不能很好满足，并且功能、设计、改动方面较ik-analyzer有很大出入，所以另起山寨。ik-analyzer项目现在还活着，前不久发布了新版，貌似只是源码实现细节的调整，项目也不是很活跃。当然，国内的几个开源分词程序基本都停滞了。ki-analyzer程序是上周修改的，这周实际测试了一下，只能说bug方面基本稳定，但尚需进一步检验。因为一些想要的功能暂时还用不上，所以也没着急做。关于ki-analyzer的详细情况可以参考项目首页信息，对该项目感兴趣或有问题的朋友可以联系我。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2011/06/453.html/feed</wfw:commentRss>
		<slash:comments>8</slash:comments>
		</item>
		<item>
		<title>Double.NaN != Double.NaN</title>
		<link>http://www.kafka0102.com/2011/05/439.html</link>
		<comments>http://www.kafka0102.com/2011/05/439.html#comments</comments>
		<pubDate>Fri, 20 May 2011 09:43:11 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[java]]></category>
		<category><![CDATA[double.nan]]></category>
		<category><![CDATA[nan]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=439</guid>
		<description><![CDATA[昨天做数据时发现个很诡异的情况，当然，再诡异的技术现象也有起发生原因。抛开我的问题来说，我可以再提出个问题：是否在某种编程语言中，存在着那么一个变量i，使得i!=i成立，也使得while(i==i);可以成功的退出？如果你想到并理解了，你可以终止阅读这篇可能冗长的文章，尽管为了填补这个月的博客空白我刻意小题大作一番。在java语言中，Double.NaN就能使得上面两个表达式成立。]]></description>
			<content:encoded><![CDATA[<p>昨天做数据时发现个很诡异的情况，当然，再诡异的技术现象也有发生原因。抛开我的问题来说，我可以再提出个问题：是否在某种编程语言中，存在着那么一个变量i，使得i!=i成立，也使得while(i==i);可以成功的退出？如果你想到并理解了，你可以终止阅读这篇可能冗长的文章，尽管为了填补这个月的博客空白我刻意小题大作一番。在java语言中，Double.NaN就能使得上面两个表达式成立。</p>
<p>说起浮点型中的NaN，之前的少有印象就是来自javascript，而考虑到没怎么搞过js，这点印象也就仅限于曾经alert出来的令人惊异的NaN（以及undefined）了。先说说浮点型吧，它的标准出自<a href="http://en.wikipedia.org/wiki/IEEE_754-2008" target="blank">IEEE 754</a>，Java规范中的章节是<a href="http://java.sun.com/docs/books/jls/third_edition/html/typesValues.html#4.2.3">http://java.sun.com/docs/books/jls/third_edition/html/typesValues.html#4.2.3</a>，两篇也挺冗长的英文内容。可以重点提及的是，在内部实现上，32位的float和64位的double是使用int和long表示的。具体的表示方法可以从java api文档上看到更好的解释。以Double为例，它提供了下面3个方法：</p>
<p style="text-align: center;"><a href="http://www.kafka0102.com/wp-content/uploads/2011/05/Screenshot.png"><img class="aligncenter size-full wp-image-445" title=" doubletolong" src="http://www.kafka0102.com/wp-content/uploads/2011/05/Screenshot.png" alt="" width="576" height="370" /></a></p>
<p style="text-align: center;"><a href="http://www.kafka0102.com/wp-content/uploads/2011/05/Screenshot-1.png"><img class="aligncenter size-full wp-image-446" title="doubletorawlong" src="http://www.kafka0102.com/wp-content/uploads/2011/05/Screenshot-1.png" alt="" width="577" height="428" /></a></p>
<p style="text-align: center;"><a href="http://www.kafka0102.com/wp-content/uploads/2011/05/Screenshot-2.png"><img class="aligncenter size-full wp-image-447" title="longtodouble" src="http://www.kafka0102.com/wp-content/uploads/2011/05/Screenshot-2.png" alt="" width="631" height="446" /></a></p>
<p>分析发现，除了常规的double取值，NaN（not a number）并不是对应着唯一的表示，尽管它并不是一个数值，比如除0或者对负数取平方根的结果都是NaN，但它们的发生原因是不一样的，所以如果是signaling NaN，可以通过比对位来得到更多的信息，尽管这在多数语言（包括Java）中是不可见的。那么，我们来看看Double.NaN会带来什么特别之处吧：</p>
<p>1）Double.NaN != Double.NaN。既非数值，何来相等。和Double.NaN相关的比较操作结果都是false，所以，Double.NaN &gt; 1.0为false，Double.NaN &lt; 1.0为false，Double.NaN == 1.0也为false。不过，Double a = new Double(Double.NaN);a.equals(a)==true;是成立的，因为对象a比较的是引用而不是数值。</p>
<p>2）Double.NaN + Double.NaN  -&gt;  Double.NaN<br />
3）Double.NaN + 1.0  -&gt;  Double.NaN<br />
4）(int) (Double.NaN) -&gt; 0。我的问题也是因为这个转换，以致我不可思议的盯着那些本不应该为0的结果。</p>
<p>从上可以看出，在Double.NaN存在时，既有的逻辑判断和数值运算可能走向未料想的方向。从语言设计角度来说，Java接纳了浮点型规范中的NaN，不像整形那样抛出异常，但这却带来潜伏的bug陷阱。看看其它语言的做法吧，C/C++会毫不留情的出core，ruby、python会报错，即便是一塌糊涂的PHP，至少还会报个warning，是不是Java的表弟C#有可能接纳了NaN？要我说啊，还是抛个异常好些，既非数值，要你添个毛乱阿。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2011/05/439.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>理解mongodb的ObjectId</title>
		<link>http://www.kafka0102.com/2011/03/435.html</link>
		<comments>http://www.kafka0102.com/2011/03/435.html#comments</comments>
		<pubDate>Tue, 08 Mar 2011 09:33:45 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[nosql]]></category>
		<category><![CDATA[mongodb]]></category>
		<category><![CDATA[objectid]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=435</guid>
		<description><![CDATA[	mongodb支持的数据类型中，ObjectId是其自有产物，本文对其做些简单的介绍。
	存储在mongodb集合中的每个文档（document）都有一个默认的主键_id，这个主键名称是固定的，它可以是mongodb支持的任何数据类型，默认是ObjectId。在关系数据库schema设计中，主键大多是数值型的，比如常用的int和long，并且更通常的，主键的取值由数据库自增获得，这种主键数值的有序性有时也表明了某种逻辑。反观mongodb，它在设计之初就定位于分布式存储系统，所以它原生的不支持自增主键。而现实的世界是，大量应用在可预见的时空里并不需要分布式的mongodb，所以网上就出现了大量的实现mongodb自增主键方法的文章。恩，我之前也干过这种事情。]]></description>
			<content:encoded><![CDATA[<p>	mongodb支持的数据类型中，ObjectId是其自有产物，本文对其做些简单的介绍。</p>
<p>	存储在mongodb集合中的每个文档（document）都有一个默认的主键_id，这个主键名称是固定的，它可以是mongodb支持的任何数据类型，默认是ObjectId。在关系数据库schema设计中，主键大多是数值型的，比如常用的int和long，并且更通常的，主键的取值由数据库自增获得，这种主键数值的有序性有时也表明了某种逻辑。反观mongodb，它在设计之初就定位于分布式存储系统，所以它原生的不支持自增主键。而现实的世界是，大量应用在可预见的时空里并不需要分布式的mongodb，所以网上就出现了大量的实现mongodb自增主键方法的文章。恩，我之前也干过这种事情。</p>
<p>	还是看看ObjectId的底细吧。ObjectId被设计成跨机器的分布式环境中全局唯一的类型，长度是12个字节。有朋友可能嘀咕了，这可比int大了两倍，比long也多了一个int，很不经济嘛，但在现在的硬件配置中，多出的这些字节很难有理由成为系统的瓶颈所在，所以尽可能放心使用之。ObjectId的12字节是如此构成的：0-3这4个字节是时间戳（timestamp）、4-6这3个字节是机器码（machine）、7-8两个字节是进程id（pid）、9-11是程序自增id（increment）。可以看下java driver中ObjectId的实现代码：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">	<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> ObjectId <span style="color: #000000; font-weight: bold;">implements</span> Comparable<span style="color: #339933;">&lt;</span>ObjectId<span style="color: #339933;">&gt;</span> , java.<span style="color: #006633;">io</span>.<span style="color: #003399;">Serializable</span> <span style="color: #009900;">&#123;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">boolean</span> D <span style="color: #339933;">=</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
&nbsp;
    <span style="color: #008000; font-style: italic; font-weight: bold;">/** Gets a new object id.
     * @return the new id
     */</span>
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">static</span> ObjectId get<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000000; font-weight: bold;">new</span> ObjectId<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #008000; font-style: italic; font-weight: bold;">/** Checks if a string could be an &lt;code&gt;ObjectId&lt;/code&gt;.
     * @return whether the string could be an object id
     */</span>
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000066; font-weight: bold;">boolean</span> isValid<span style="color: #009900;">&#40;</span> <span style="color: #003399;">String</span> s <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> s <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> s.<span style="color: #006633;">length</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">&lt;</span> <span style="color: #cc66cc;">18</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span> i<span style="color: #339933;">&lt;</span>s.<span style="color: #006633;">length</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span> i<span style="color: #339933;">++</span> <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
            <span style="color: #000066; font-weight: bold;">char</span> c <span style="color: #339933;">=</span> s.<span style="color: #006633;">charAt</span><span style="color: #009900;">&#40;</span> i <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> c <span style="color: #339933;">&gt;=</span> <span style="color: #0000ff;">'0'</span> <span style="color: #339933;">&amp;&amp;</span> c <span style="color: #339933;">&lt;=</span> <span style="color: #0000ff;">'9'</span> <span style="color: #009900;">&#41;</span>
                <span style="color: #000000; font-weight: bold;">continue</span><span style="color: #339933;">;</span>
            <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> c <span style="color: #339933;">&gt;=</span> <span style="color: #0000ff;">'a'</span> <span style="color: #339933;">&amp;&amp;</span> c <span style="color: #339933;">&lt;=</span> <span style="color: #0000ff;">'f'</span> <span style="color: #009900;">&#41;</span>
                <span style="color: #000000; font-weight: bold;">continue</span><span style="color: #339933;">;</span>
            <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> c <span style="color: #339933;">&gt;=</span> <span style="color: #0000ff;">'A'</span> <span style="color: #339933;">&amp;&amp;</span> c <span style="color: #339933;">&lt;=</span> <span style="color: #0000ff;">'F'</span> <span style="color: #009900;">&#41;</span>
                <span style="color: #000000; font-weight: bold;">continue</span><span style="color: #339933;">;</span>
&nbsp;
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>        
&nbsp;
        <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">true</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #008000; font-style: italic; font-weight: bold;">/** Turn an object into an &lt;code&gt;ObjectId&lt;/code&gt;, if possible.
     * Strings will be converted into &lt;code&gt;ObjectId&lt;/code&gt;s, if possible, and &lt;code&gt;ObjectId&lt;/code&gt;s will
     * be cast and returned.  Passing in &lt;code&gt;null&lt;/code&gt; returns &lt;code&gt;null&lt;/code&gt;.
     * @param o the object to convert 
     * @return an &lt;code&gt;ObjectId&lt;/code&gt; if it can be massaged, null otherwise 
     */</span>
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">static</span> ObjectId massageToObjectId<span style="color: #009900;">&#40;</span> <span style="color: #003399;">Object</span> o <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> o <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> o <span style="color: #000000; font-weight: bold;">instanceof</span> ObjectId <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #009900;">&#40;</span>ObjectId<span style="color: #009900;">&#41;</span>o<span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> o <span style="color: #000000; font-weight: bold;">instanceof</span> <span style="color: #003399;">String</span> <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
            <span style="color: #003399;">String</span> s <span style="color: #339933;">=</span> o.<span style="color: #006633;">toString</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> isValid<span style="color: #009900;">&#40;</span> s <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span>
                <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000000; font-weight: bold;">new</span> ObjectId<span style="color: #009900;">&#40;</span> s <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> ObjectId<span style="color: #009900;">&#40;</span> <span style="color: #003399;">Date</span> time <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        _time <span style="color: #339933;">=</span> _flip<span style="color: #009900;">&#40;</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#40;</span>time.<span style="color: #006633;">getTime</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">/</span> <span style="color: #cc66cc;">1000</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        _machine <span style="color: #339933;">=</span> _genmachine<span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">synchronized</span> <span style="color: #009900;">&#40;</span> _incLock <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
            _inc <span style="color: #339933;">=</span> _nextInc<span style="color: #339933;">++;</span>
        <span style="color: #009900;">&#125;</span>     
        _new <span style="color: #339933;">=</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> ObjectId<span style="color: #009900;">&#40;</span> <span style="color: #003399;">Date</span> time , <span style="color: #000066; font-weight: bold;">int</span> inc <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">this</span><span style="color: #009900;">&#40;</span> time , _genmachine , inc <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> ObjectId<span style="color: #009900;">&#40;</span> <span style="color: #003399;">Date</span> time , <span style="color: #000066; font-weight: bold;">int</span> machine , <span style="color: #000066; font-weight: bold;">int</span> inc <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        _time <span style="color: #339933;">=</span> _flip<span style="color: #009900;">&#40;</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#40;</span>time.<span style="color: #006633;">getTime</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">/</span> <span style="color: #cc66cc;">1000</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        _machine <span style="color: #339933;">=</span> machine<span style="color: #339933;">;</span>
        _inc <span style="color: #339933;">=</span> inc<span style="color: #339933;">;</span>
        _new <span style="color: #339933;">=</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
&nbsp;
    <span style="color: #008000; font-style: italic; font-weight: bold;">/** Creates a new instance from a string.
     * @param s the string to convert
     * @throws IllegalArgumentException if the string is not a valid id
     */</span>
    <span style="color: #000000; font-weight: bold;">public</span> ObjectId<span style="color: #009900;">&#40;</span> <span style="color: #003399;">String</span> s <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">this</span><span style="color: #009900;">&#40;</span> s , <span style="color: #000066; font-weight: bold;">false</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> ObjectId<span style="color: #009900;">&#40;</span> <span style="color: #003399;">String</span> s , <span style="color: #000066; font-weight: bold;">boolean</span> babble <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> <span style="color: #339933;">!</span> isValid<span style="color: #009900;">&#40;</span> s <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">throw</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">IllegalArgumentException</span><span style="color: #009900;">&#40;</span> <span style="color: #0000ff;">&quot;invalid ObjectId [&quot;</span> <span style="color: #339933;">+</span> s <span style="color: #339933;">+</span> <span style="color: #0000ff;">&quot;]&quot;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> babble <span style="color: #009900;">&#41;</span>
            s <span style="color: #339933;">=</span> babbleToMongod<span style="color: #009900;">&#40;</span> s <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000066; font-weight: bold;">byte</span> b<span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #000066; font-weight: bold;">byte</span><span style="color: #009900;">&#91;</span><span style="color: #cc66cc;">12</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span> i<span style="color: #339933;">&lt;</span>b.<span style="color: #006633;">length</span><span style="color: #339933;">;</span> i<span style="color: #339933;">++</span> <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
            b<span style="color: #009900;">&#91;</span>b.<span style="color: #006633;">length</span><span style="color: #339933;">-</span><span style="color: #009900;">&#40;</span>i<span style="color: #339933;">+</span><span style="color: #cc66cc;">1</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">byte</span><span style="color: #009900;">&#41;</span><span style="color: #003399;">Integer</span>.<span style="color: #006633;">parseInt</span><span style="color: #009900;">&#40;</span> s.<span style="color: #006633;">substring</span><span style="color: #009900;">&#40;</span> i<span style="color: #339933;">*</span><span style="color: #cc66cc;">2</span> , i<span style="color: #339933;">*</span><span style="color: #cc66cc;">2</span> <span style="color: #339933;">+</span> <span style="color: #cc66cc;">2</span><span style="color: #009900;">&#41;</span> , <span style="color: #cc66cc;">16</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
        ByteBuffer bb <span style="color: #339933;">=</span> ByteBuffer.<span style="color: #006633;">wrap</span><span style="color: #009900;">&#40;</span> b <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
        _inc <span style="color: #339933;">=</span> bb.<span style="color: #006633;">getInt</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span> 
        _machine <span style="color: #339933;">=</span> bb.<span style="color: #006633;">getInt</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        _time <span style="color: #339933;">=</span> bb.<span style="color: #006633;">getInt</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
        _new <span style="color: #339933;">=</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> ObjectId<span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">byte</span><span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> b <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> b.<span style="color: #006633;">length</span> <span style="color: #339933;">!=</span> <span style="color: #cc66cc;">12</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">throw</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">IllegalArgumentException</span><span style="color: #009900;">&#40;</span> <span style="color: #0000ff;">&quot;need 12 bytes&quot;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        reverse<span style="color: #009900;">&#40;</span> b <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        ByteBuffer bb <span style="color: #339933;">=</span> ByteBuffer.<span style="color: #006633;">wrap</span><span style="color: #009900;">&#40;</span> b <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
        _inc <span style="color: #339933;">=</span> bb.<span style="color: #006633;">getInt</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>            
        _machine <span style="color: #339933;">=</span> bb.<span style="color: #006633;">getInt</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        _time <span style="color: #339933;">=</span> bb.<span style="color: #006633;">getInt</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> ObjectId<span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">int</span> time , <span style="color: #000066; font-weight: bold;">int</span> machine , <span style="color: #000066; font-weight: bold;">int</span> inc <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        _time <span style="color: #339933;">=</span> time<span style="color: #339933;">;</span>
        _machine <span style="color: #339933;">=</span> machine<span style="color: #339933;">;</span>
        _inc <span style="color: #339933;">=</span> inc<span style="color: #339933;">;</span>
&nbsp;
        _new <span style="color: #339933;">=</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #008000; font-style: italic; font-weight: bold;">/** Create a new object id.
     */</span>
    <span style="color: #000000; font-weight: bold;">public</span> ObjectId<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        _time <span style="color: #339933;">=</span> _gentime<span style="color: #339933;">;</span>
        _machine <span style="color: #339933;">=</span> _genmachine<span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">synchronized</span> <span style="color: #009900;">&#40;</span> _incLock <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
            _inc <span style="color: #339933;">=</span> _nextInc<span style="color: #339933;">++;</span>
        <span style="color: #009900;">&#125;</span>
&nbsp;
        _new <span style="color: #339933;">=</span> <span style="color: #000066; font-weight: bold;">true</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">int</span> hashCode<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> _inc<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">boolean</span> equals<span style="color: #009900;">&#40;</span> <span style="color: #003399;">Object</span> o <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> <span style="color: #000000; font-weight: bold;">this</span> <span style="color: #339933;">==</span> o <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">true</span><span style="color: #339933;">;</span>
&nbsp;
        ObjectId other <span style="color: #339933;">=</span> massageToObjectId<span style="color: #009900;">&#40;</span> o <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> other <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">return</span> 
            _time <span style="color: #339933;">==</span> other._time <span style="color: #339933;">&amp;&amp;</span> 
            _machine <span style="color: #339933;">==</span> other._machine <span style="color: #339933;">&amp;&amp;</span> 
            _inc <span style="color: #339933;">==</span> other._inc<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">String</span> toStringBabble<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> babbleToMongod<span style="color: #009900;">&#40;</span> toStringMongod<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">String</span> toStringMongod<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000066; font-weight: bold;">byte</span> b<span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> toByteArray<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
        StringBuilder buf <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> StringBuilder<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">24</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span> i<span style="color: #339933;">&lt;</span>b.<span style="color: #006633;">length</span><span style="color: #339933;">;</span> i<span style="color: #339933;">++</span> <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
            <span style="color: #000066; font-weight: bold;">int</span> x <span style="color: #339933;">=</span> b<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span> <span style="color: #339933;">&amp;</span> 0xFF<span style="color: #339933;">;</span>
            <span style="color: #003399;">String</span> s <span style="color: #339933;">=</span> <span style="color: #003399;">Integer</span>.<span style="color: #006633;">toHexString</span><span style="color: #009900;">&#40;</span> x <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> s.<span style="color: #006633;">length</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">==</span> <span style="color: #cc66cc;">1</span> <span style="color: #009900;">&#41;</span>
                buf.<span style="color: #006633;">append</span><span style="color: #009900;">&#40;</span> <span style="color: #0000ff;">&quot;0&quot;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            buf.<span style="color: #006633;">append</span><span style="color: #009900;">&#40;</span> s <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">return</span> buf.<span style="color: #006633;">toString</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">byte</span><span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> toByteArray<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000066; font-weight: bold;">byte</span> b<span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #000066; font-weight: bold;">byte</span><span style="color: #009900;">&#91;</span><span style="color: #cc66cc;">12</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
        ByteBuffer bb <span style="color: #339933;">=</span> ByteBuffer.<span style="color: #006633;">wrap</span><span style="color: #009900;">&#40;</span> b <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        bb.<span style="color: #006633;">putInt</span><span style="color: #009900;">&#40;</span> _inc <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        bb.<span style="color: #006633;">putInt</span><span style="color: #009900;">&#40;</span> _machine <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        bb.<span style="color: #006633;">putInt</span><span style="color: #009900;">&#40;</span> _time <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        reverse<span style="color: #009900;">&#40;</span> b <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">return</span> b<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000066; font-weight: bold;">void</span> reverse<span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">byte</span><span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> b <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span> i<span style="color: #339933;">&lt;</span>b.<span style="color: #006633;">length</span><span style="color: #339933;">/</span><span style="color: #cc66cc;">2</span><span style="color: #339933;">;</span> i<span style="color: #339933;">++</span> <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
            <span style="color: #000066; font-weight: bold;">byte</span> t <span style="color: #339933;">=</span> b<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
            b<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> b<span style="color: #009900;">&#91;</span> b.<span style="color: #006633;">length</span><span style="color: #339933;">-</span><span style="color: #009900;">&#40;</span>i<span style="color: #339933;">+</span><span style="color: #cc66cc;">1</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
            b<span style="color: #009900;">&#91;</span>b.<span style="color: #006633;">length</span><span style="color: #339933;">-</span><span style="color: #009900;">&#40;</span>i<span style="color: #339933;">+</span><span style="color: #cc66cc;">1</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> t<span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #003399;">String</span> _pos<span style="color: #009900;">&#40;</span> <span style="color: #003399;">String</span> s , <span style="color: #000066; font-weight: bold;">int</span> p <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> s.<span style="color: #006633;">substring</span><span style="color: #009900;">&#40;</span> p <span style="color: #339933;">*</span> <span style="color: #cc66cc;">2</span> , <span style="color: #009900;">&#40;</span> p <span style="color: #339933;">*</span> <span style="color: #cc66cc;">2</span> <span style="color: #009900;">&#41;</span> <span style="color: #339933;">+</span> <span style="color: #cc66cc;">2</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #003399;">String</span> babbleToMongod<span style="color: #009900;">&#40;</span> <span style="color: #003399;">String</span> b <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> <span style="color: #339933;">!</span> isValid<span style="color: #009900;">&#40;</span> b <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">throw</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">IllegalArgumentException</span><span style="color: #009900;">&#40;</span> <span style="color: #0000ff;">&quot;invalid object id: &quot;</span> <span style="color: #339933;">+</span> b <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
        StringBuilder buf <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> StringBuilder<span style="color: #009900;">&#40;</span> <span style="color: #cc66cc;">24</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">7</span><span style="color: #339933;">;</span> i<span style="color: #339933;">&gt;=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span> i<span style="color: #339933;">--</span> <span style="color: #009900;">&#41;</span>
            buf.<span style="color: #006633;">append</span><span style="color: #009900;">&#40;</span> _pos<span style="color: #009900;">&#40;</span> b , i <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">11</span><span style="color: #339933;">;</span> i<span style="color: #339933;">&gt;=</span><span style="color: #cc66cc;">8</span><span style="color: #339933;">;</span> i<span style="color: #339933;">--</span> <span style="color: #009900;">&#41;</span>
            buf.<span style="color: #006633;">append</span><span style="color: #009900;">&#40;</span> _pos<span style="color: #009900;">&#40;</span> b , i <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">return</span> buf.<span style="color: #006633;">toString</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">String</span> toString<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> toStringMongod<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">int</span> compareTo<span style="color: #009900;">&#40;</span> ObjectId id <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> id <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #339933;">-</span><span style="color: #cc66cc;">1</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000066; font-weight: bold;">long</span> xx <span style="color: #339933;">=</span> id.<span style="color: #006633;">getTime</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">-</span> getTime<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> xx <span style="color: #339933;">&gt;</span> <span style="color: #cc66cc;">0</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #339933;">-</span><span style="color: #cc66cc;">1</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">else</span> <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> xx <span style="color: #339933;">&lt;</span> <span style="color: #cc66cc;">0</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #cc66cc;">1</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000066; font-weight: bold;">int</span> x <span style="color: #339933;">=</span> id._machine <span style="color: #339933;">-</span> _machine<span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> x <span style="color: #339933;">!=</span> <span style="color: #cc66cc;">0</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #339933;">-</span>x<span style="color: #339933;">;</span>
&nbsp;
        x <span style="color: #339933;">=</span> id._inc <span style="color: #339933;">-</span> _inc<span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> x <span style="color: #339933;">!=</span> <span style="color: #cc66cc;">0</span> <span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #339933;">-</span>x<span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">int</span> getMachine<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> _machine<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">long</span> getTime<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000066; font-weight: bold;">long</span> z <span style="color: #339933;">=</span> _flip<span style="color: #009900;">&#40;</span> _time <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">return</span> z <span style="color: #339933;">*</span> <span style="color: #cc66cc;">1000</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">int</span> getInc<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> _inc<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">int</span> _time<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> _time<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">int</span> _machine<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> _machine<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">int</span> _inc<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> _inc<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">boolean</span> isNew<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> _new<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> notNew<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        _new <span style="color: #339933;">=</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> _time<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> _machine<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> _inc<span style="color: #339933;">;</span>
&nbsp;
    <span style="color: #000066; font-weight: bold;">boolean</span> _new<span style="color: #339933;">;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000066; font-weight: bold;">int</span> _flip<span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">int</span> x <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">true</span> <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
            <span style="color: #000066; font-weight: bold;">byte</span> b<span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #000066; font-weight: bold;">byte</span><span style="color: #009900;">&#91;</span><span style="color: #cc66cc;">4</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
            ByteBuffer bb <span style="color: #339933;">=</span> ByteBuffer.<span style="color: #006633;">wrap</span><span style="color: #009900;">&#40;</span> b <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            bb.<span style="color: #006633;">order</span><span style="color: #009900;">&#40;</span> ByteOrder.<span style="color: #006633;">LITTLE_ENDIAN</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            bb.<span style="color: #006633;">putInt</span><span style="color: #009900;">&#40;</span> x <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            bb.<span style="color: #006633;">flip</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            bb.<span style="color: #006633;">order</span><span style="color: #009900;">&#40;</span> ByteOrder.<span style="color: #006633;">BIG_ENDIAN</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #000000; font-weight: bold;">return</span> bb.<span style="color: #006633;">getInt</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #000066; font-weight: bold;">int</span> z <span style="color: #339933;">=</span> <span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>
        z <span style="color: #339933;">|=</span> <span style="color: #009900;">&#40;</span> x <span style="color: #339933;">&amp;</span> 0xFF <span style="color: #009900;">&#41;</span> <span style="color: #339933;">&lt;&lt;</span> <span style="color: #cc66cc;">24</span><span style="color: #339933;">;</span>
        z <span style="color: #339933;">|=</span> <span style="color: #009900;">&#40;</span> x <span style="color: #339933;">&amp;</span> 0xFF00 <span style="color: #009900;">&#41;</span> <span style="color: #339933;">&lt;&lt;</span> <span style="color: #cc66cc;">8</span><span style="color: #339933;">;</span>
        z <span style="color: #339933;">|=</span> <span style="color: #009900;">&#40;</span> x <span style="color: #339933;">&amp;</span> 0xFF00000 <span style="color: #009900;">&#41;</span> <span style="color: #339933;">&gt;&gt;</span> <span style="color: #cc66cc;">8</span><span style="color: #339933;">;</span>
        z <span style="color: #339933;">|=</span> <span style="color: #009900;">&#40;</span> x <span style="color: #339933;">&amp;</span> 0xFF000000 <span style="color: #009900;">&#41;</span> <span style="color: #339933;">&gt;&gt;</span> <span style="color: #cc66cc;">24</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">return</span> z<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000066; font-weight: bold;">int</span> _nextInc <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">new</span> java.<span style="color: #006633;">util</span>.<span style="color: #003399;">Random</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">nextInt</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">String</span> _incLock <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">String</span><span style="color: #009900;">&#40;</span> <span style="color: #0000ff;">&quot;ObjectId._incLock&quot;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000066; font-weight: bold;">int</span> _gentime <span style="color: #339933;">=</span> _flip<span style="color: #009900;">&#40;</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#40;</span><span style="color: #003399;">System</span>.<span style="color: #006633;">currentTimeMillis</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">/</span><span style="color: #cc66cc;">1000</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Thread</span> _timeFixer<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> _genmachine<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #009900;">&#123;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
&nbsp;
            <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> machinePiece<span style="color: #339933;">;</span>
            <span style="color: #009900;">&#123;</span>
                StringBuilder sb <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> StringBuilder<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                Enumeration<span style="color: #339933;">&lt;</span>NetworkInterface<span style="color: #339933;">&gt;</span> e <span style="color: #339933;">=</span> NetworkInterface.<span style="color: #006633;">getNetworkInterfaces</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #000000; font-weight: bold;">while</span> <span style="color: #009900;">&#40;</span> e.<span style="color: #006633;">hasMoreElements</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
                    NetworkInterface ni <span style="color: #339933;">=</span> e.<span style="color: #006633;">nextElement</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                    sb.<span style="color: #006633;">append</span><span style="color: #009900;">&#40;</span> ni.<span style="color: #006633;">toString</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #009900;">&#125;</span>
                machinePiece <span style="color: #339933;">=</span> sb.<span style="color: #006633;">toString</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">hashCode</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">&lt;&lt;</span> <span style="color: #cc66cc;">16</span><span style="color: #339933;">;</span>
                <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> D <span style="color: #009900;">&#41;</span> <span style="color: #003399;">System</span>.<span style="color: #006633;">out</span>.<span style="color: #006633;">println</span><span style="color: #009900;">&#40;</span> <span style="color: #0000ff;">&quot;machine piece post: &quot;</span> <span style="color: #339933;">+</span> <span style="color: #003399;">Integer</span>.<span style="color: #006633;">toHexString</span><span style="color: #009900;">&#40;</span> machinePiece <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #009900;">&#125;</span>
&nbsp;
            <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> processPiece <span style="color: #339933;">=</span> java.<span style="color: #006633;">lang</span>.<span style="color: #006633;">management</span>.<span style="color: #006633;">ManagementFactory</span>.<span style="color: #006633;">getRuntimeMXBean</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">getName</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">hashCode</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">&amp;</span> 0xFFFF<span style="color: #339933;">;</span>
            <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> D <span style="color: #009900;">&#41;</span> <span style="color: #003399;">System</span>.<span style="color: #006633;">out</span>.<span style="color: #006633;">println</span><span style="color: #009900;">&#40;</span> <span style="color: #0000ff;">&quot;process piece: &quot;</span> <span style="color: #339933;">+</span> <span style="color: #003399;">Integer</span>.<span style="color: #006633;">toHexString</span><span style="color: #009900;">&#40;</span> processPiece <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
            _genmachine <span style="color: #339933;">=</span> machinePiece <span style="color: #339933;">|</span> processPiece<span style="color: #339933;">;</span>
            <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span> D <span style="color: #009900;">&#41;</span> <span style="color: #003399;">System</span>.<span style="color: #006633;">out</span>.<span style="color: #006633;">println</span><span style="color: #009900;">&#40;</span> <span style="color: #0000ff;">&quot;machine : &quot;</span> <span style="color: #339933;">+</span> <span style="color: #003399;">Integer</span>.<span style="color: #006633;">toHexString</span><span style="color: #009900;">&#40;</span> _genmachine <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span> java.<span style="color: #006633;">io</span>.<span style="color: #003399;">IOException</span> ioe <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
            <span style="color: #000000; font-weight: bold;">throw</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">RuntimeException</span><span style="color: #009900;">&#40;</span> ioe <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
&nbsp;
        _timeFixer <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">Thread</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;ObjectId-TimeFixer&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
                <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> run<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
                    <span style="color: #000000; font-weight: bold;">while</span> <span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">true</span> <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
                        <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
                            <span style="color: #003399;">Thread</span>.<span style="color: #006633;">sleep</span><span style="color: #009900;">&#40;</span> <span style="color: #cc66cc;">499</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                        <span style="color: #009900;">&#125;</span>
                        <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span> <span style="color: #003399;">Exception</span> e <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span><span style="color: #009900;">&#125;</span>
                        _gentime <span style="color: #339933;">=</span> _flip<span style="color: #009900;">&#40;</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#40;</span><span style="color: #003399;">System</span>.<span style="color: #006633;">currentTimeMillis</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">/</span><span style="color: #cc66cc;">1000</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                    <span style="color: #009900;">&#125;</span>
                <span style="color: #009900;">&#125;</span>
            <span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
        _timeFixer.<span style="color: #006633;">setDaemon</span><span style="color: #009900;">&#40;</span> <span style="color: #000066; font-weight: bold;">true</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        _timeFixer.<span style="color: #006633;">start</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>	对于ObjectId的组成，有一些值得说道的地方：<br />
	1、因为ObjectId以时间戳打头，所以它是近似有序的，使得_id的索引插入效率相比普通索引高很多。<br />
	2、ObjectId的前9个字节（timestamp+machine+pid）可以保证不同进程生成的ObjectId不会重复，而后3个字节increment又可以保证同一进程内生成的ObjectId不会重复，所以无需怀疑ObjectId的全局唯一性。<br />
	3、ObjectId存储是12个字节，但如果应用有需要以可读的方式表现它，就需要将它转成字符串，这需要24字节（每字节转成2字节的16进制表示），这个长度的字符串看起来就有些不让人舒服了，如果是追踪某个_id引发的bug，就需要配上copy+paste的杀招。<br />
	4、初涉ObjectId的朋友很容易犯的两个错误：1）是查询时直接使用类似db.collection.find({_id:&#8221;xx&#8221;})式的代码，结果怎么也查不到明明存在的文档，而正确的写法应该是：db.collection.find({_id:new ObjectId(&#8220;xx&#8221;)})。2）是集合间有外键关联时，也需要将外键置为ObjectId类型，而不要直接使用上24字节的string。在写与mongodb打交道的CRUD代码时，需要多留意ObjectId和string的转换代码。<br />
	5、ObjectId的产生既可以在应用程序端也可以在mongodb端，各种语言的driver都提供了程序端生成ObjectId的方法，不过大多数人徒省事直接交给mongodb做了。但从mongodb的设计哲学来说，ObjectId更应该由客户端生成，毕竟应用层比存储层更容易扩展，并会提高mongodb的插入速度。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2011/03/435.html/feed</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>夜说mongodb</title>
		<link>http://www.kafka0102.com/2011/02/430.html</link>
		<comments>http://www.kafka0102.com/2011/02/430.html#comments</comments>
		<pubDate>Wed, 16 Feb 2011 20:28:49 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[nosql]]></category>
		<category><![CDATA[mongodb]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=430</guid>
		<description><![CDATA[赋闲以后很长没有更新博客了，说忙完全是借口，多半因为没有兴致所致。今天凌晨比赛多多，趁着比赛的前奏和间隙，遂浏览些技术文章。发现了 highscalability.com整理出了wordnik使用mongodb和scala的使用经 验：http://highscalability.com/blog/2011/2/15/wordnik-10-million-api- requests-a-day-on-mongodb-and-scala.html。这个wordnik也算是mongodb的先驱应用者之一，并贡献了一些管理工具给社区，去年的mongoSF大会也有它的参与：http://www.slideshare.net/fehguy /migrating-from-mysql-to-mongodb-at-wordnik。]]></description>
			<content:encoded><![CDATA[<p>赋闲以后很长没有更新博客了，说忙完全是借口，多半因为没有兴致所致。今天凌晨比赛多多，趁着比赛的前奏和间隙，遂浏览些技术文章。发现了 highscalability.com整理出了wordnik使用mongodb和scala的使用经 验：<a href="http://highscalability.com/blog/2011/2/15/wordnik-10-million-api- requests-a-day-on-mongodb-and-scala.html" target="_blank">http://highscalability.com/blog/2011/2/15/wordnik-10-million-api- requests-a-day-on-mongodb-and-scala.html</a>。这个wordnik也算是mongodb的先驱应用者之一，并贡献了一些管理工具给社区，去年的mongoSF大会也有它的参与：<a href="http://www.slideshare.net/fehguy /migrating-from-mysql-to-mongodb-at-wordnik" target="_blank">http://www.slideshare.net/fehguy /migrating-from-mysql-to-mongodb-at-wordnik</a>。</p>
<p>说起mongodb，从初识到现在已经有些时间，却也没有大场面使用和深入研究。nosql风吹了两年多势头愈发趋于冷静，能真正被大家认可使用的也就那 么几个，但又都离如mysql般成熟有些距离。而说起wordnik，它最初使用的就是mysql，后来主要因为性能问题转投mongodb，效果似乎还不 错。不过，单看wordnik的数据规模和访问情况，精心的使用mysql完全可以胜任，只是mongodb可能能更简单的胜任他们的应用场景。 wordnik给出的一些mongodb使用经验值得分享一下。</p>
<p>mongodb is so  fast！！！就像“flash”一样！！！这是很多mongodb使用者对它的响应速度的赞叹。在一些场景下，它确实可能如此。mongodb不像其他 db自己管理内存，而是使用操作系统的内存映射文件(memory-mapped  file)，这使得mongodb省去了复杂的内存管理，也能取得良好的文件cache效果。当内存够大（比如wordnik为mongodb配备了 72GB的内存，是不是可以根据请求数据的分布来度量出多大内存能满足cache的命中率呢？），读请求的数据相对集中，那么mongodb的读速度要卓越的多。如果mongodb恰好能满足你对速度的要求，那么你可以放弃那一层 cache胶水代码了。</p>
<p>wordnik在更新数据时使用了一个trick，它不是直接更新，而是先把要更新的数据读出来（到了内存），然后再更新。就单条数据来说，这种并没有提 高更新速度。不过，因为mongodb的更新使用的是全局锁，这种方式可以减少实际更新时的锁时间。而且，wordnik的更新是异步的，它会控制更新频 率以避免阻塞读请求，而mongodb在读写锁的效率上应该还可以提高。另一方面，如果更新的文档数据超过原来的大小，mongodb会标记删除旧数据追 加新数据，而在无效碎片利用上，mongodb做得似乎不是很好。之前我在使用时，发现经过几次全数据的批量更新后，文件大小会翻番，而大量的碎片又会影响到 cache效果使得读写速度都有下降。因为mongodb的每个colleciton文件都有一些空间留作padding，如果colleciton很 多，相应的总文件大小也会变大。</p>
<p>mongodb的文档模型很给力，编程接口很简洁。所以从关系数据库迁移到文档数据库，相比什么列型数据库，要容易得多。而从编程的对象模型到存储模型的 映射来说，文档模型也要比关系模型更友好。在我使用mongodb过程中，非常喜欢它的文档嵌套性和value的数组类型，一些一对多关系就省去了关系数 据库中新建一个表存储的需要，简化schema的设计。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2011/02/430.html/feed</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>searchblox&#8211;一个基于lucene的搜索产品</title>
		<link>http://www.kafka0102.com/2010/12/425.html</link>
		<comments>http://www.kafka0102.com/2010/12/425.html#comments</comments>
		<pubDate>Wed, 08 Dec 2010 15:04:40 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[search]]></category>
		<category><![CDATA[lucene]]></category>
		<category><![CDATA[searchblox]]></category>
		<category><![CDATA[搜索]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=425</guid>
		<description><![CDATA[前两天在solr邮件组看到一封广告帖，一个叫searchblox的搜索产品可免费使用，好奇心驱使我简单了解并使用了一下。searchblox是基于lucene的搜索解决方案，现在的版本已经是6.1,看来也有些年头了。searchblox不是个开源产品，有免费的版本，也有收费的版本，看文档介绍，收费版本除了提供服务支持还多了复制功能。]]></description>
			<content:encoded><![CDATA[<p>前两天在solr邮件组看到一封广告帖，一个叫<a href="http://www.searchblox.com" target="_blank">searchblox</a>的搜索产品可免费使用，好奇心驱使我简单了解并使用了一下。searchblox是基于lucene的搜索解决方案，现在的版本已经是6.1,看来也有些年头了。searchblox不是个开源产品，有免费的版本，也有收费的版本，看文档介绍，收费版本除了提供服务支持还多了复制功能。</p>
<p>功能上看，searchblox集成了爬虫和搜索功能，也提供Http API接口供索引和查询，可以在<a href="http://www.searchblox.com/developers/documentation" target="_blank">http://www.searchblox.com/developers/documentation</a>上更多的了解searchblox的功能。但搜索功能方面，它也就是solr的子集。试用它很简单，下载解压缩后可直接java -jar start.jar通过jetty启动它，访问它的web admin console观察它的功能。就数据源来说，可以通过配置抓取的网站、RSS feed、本地文件夹数据，供searchblox自动索引和查询。如果索引数据来源自数据库等，searchblox没有提供直接支持，但可以使用它的Http API操作（或许它更应该提供多语言的客户端API）。</p>
<p>下面看下它的架构图，可以更好的睽睽它提供的功能：<br />
<a href="http://www.kafka0102.com/wp-content/uploads/2010/12/Drawing1.png"><img class="aligncenter size-full wp-image-426" title="searchblox arch" src="http://www.kafka0102.com/wp-content/uploads/2010/12/Drawing1.png" alt="" width="507" height="374" /></a><br />
在可扩展性方面，它提供了复制的支持，但没有提供shard支持：<br />
<a href="http://www.kafka0102.com/wp-content/uploads/2010/12/ha.png"><img class="aligncenter size-full wp-image-427" title="searchblox ha" src="http://www.kafka0102.com/wp-content/uploads/2010/12/ha.png" alt="" width="500" height="461" /></a></p>
<p>写到这里，其实我都不想对searchblox再评头论足了，它看起来实在不是很吸引人。尽管有免费版本，但免费版本功能残缺、文档匮乏，不交钱想用好searchblox也是件难事。看它网站上列出的客户列表，用户群还不少，只是不知道其中水分怎么样。国外基于lucene等开源库的收费搜索解决方案还是很多的，毕竟使用lucene开发一套好的搜索产品成本还是很可观的。但搜索解决方案无可避免的，会和开源的solr做比较，像searchblox的搜索功能，就不如solr的强大。如果使用solr和某个开源爬虫，搭起一个类searchblox的产品需要做的也就是admin console。当然，即便是开源的solr，用好它用对它也是需要功夫的，尤其是其在中文圈的资料较少，经验积累方面也较欠缺。</p>
<p>searchblox，再见！！！</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/12/425.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Solr复制bug一例：Unable to move index file from tempfile to indexfile</title>
		<link>http://www.kafka0102.com/2010/11/414.html</link>
		<comments>http://www.kafka0102.com/2010/11/414.html#comments</comments>
		<pubDate>Wed, 24 Nov 2010 15:37:28 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[solr]]></category>
		<category><![CDATA[solr replication]]></category>
		<category><![CDATA[solr 复制]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=414</guid>
		<description><![CDATA[22日下午3时多，收到搜索系统的报警邮件，错误日志如下：
[2010-11-22 15:16:14][ERROR][pool-6-thread-1][SnapPuller.java(650)]Unable to move index file from: /indexpath/index.20101122031500/_21.frq to: /indexpath/index.20101122031000/_21.frq
	SnapPuller是Solr复制用到的一个类，我对它做了一些修改，所以把它挪到我的代码里了。报错的代码片断如下：]]></description>
			<content:encoded><![CDATA[<p>22日下午3时多，收到搜索系统的报警邮件，错误日志如下：<br />
<font color="blue"><br />
[2010-11-22 15:16:14][ERROR][pool-6-thread-1][SnapPuller.java(650)]Unable to move index file from: /indexpath/index.20101122031500/_21.frq to: /indexpath/index.20101122031000/_21.frq<br />
</font><br />
	SnapPuller是Solr复制用到的一个类，我对它做了一些修改，所以把它挪到我的代码里了。报错的代码片断如下：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000066; font-weight: bold;">boolean</span> copyAFile<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">File</span> tmpIdxDir, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">File</span> indexDir, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">String</span> fname, <span style="color: #000000; font-weight: bold;">final</span> List<span style="color: #339933;">&lt;</span>String<span style="color: #339933;">&gt;</span> copiedfiles<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">File</span> indexFileInTmpDir <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">File</span><span style="color: #009900;">&#40;</span>tmpIdxDir, fname<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">File</span> indexFileInIndex <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">File</span><span style="color: #009900;">&#40;</span>indexDir, fname<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">boolean</span> success <span style="color: #339933;">=</span> indexFileInTmpDir.<span style="color: #006633;">renameTo</span><span style="color: #009900;">&#40;</span>indexFileInIndex<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span><span style="color: #339933;">!</span>success<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      LOG.<span style="color: #006633;">error</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;Unable to move index file from: &quot;</span> <span style="color: #339933;">+</span> indexFileInTmpDir
          <span style="color: #339933;">+</span> <span style="color: #0000ff;">&quot; to: &quot;</span> <span style="color: #339933;">+</span> indexFileInIndex<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">String</span> f <span style="color: #339933;">:</span> copiedfiles<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">File</span> indexFile <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">File</span><span style="color: #009900;">&#40;</span>indexDir, f<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>indexFile.<span style="color: #006633;">exists</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
          indexFile.<span style="color: #006633;">delete</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
      <span style="color: #009900;">&#125;</span>
      delTree<span style="color: #009900;">&#40;</span>tmpIdxDir<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">true</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span></pre></div></div>

<p>  	错误现象表明，indexFileInTmpDir.renameTo(indexFileInIndex)失败了。在这个复制过程中，这种失败最大的可能性就是indexFileInIndex所在的indexDir不存在。查看索引数据目录，确实没有找到index.20101122031000文件夹。不过，从库的index.properties文件却记录“index=index.20101122031000”，也就是说，当前提供查询服务的索引目录就是index.20101122031000。查看进程打开的句柄，进程确实保持着index.20101122031000目录下的索引文件句柄，但它们也确实被删除了。</p>
<p>  	这里补充一下背景，搜索系统使用的是最新solr1.4.1版本，从库每5分钟同步一次主库。所以，当上面的错误出现以后，后续每5分钟就又会出现类似的错误，使得复制总是失败。Solr复制的部分原理是：从库默认的索引目录是index，在正常复制时，从库只是对该目录里的索引文件做修改（添加新的，删除旧的），但当从库索引提交点（IndexCommit）的generation大于主库时，会发生索引文件的全copy，也就是抛弃当前使用的索引目录，而是使用新生成的临时索引目录作为新的索引目录，并修改index.properties，临时目录的命名格式就是取当前的一时间戳（也就是bug所在，但到这里我还没发现这个bug）。Solr wiki也有提到，从库的IndexCommit.generation大于的bad case就是外界对从库做了不适当的提交。而在我的系统里，这种现象是由于重建索引造成的，系统重建索引没有使用DIH，而是在不影响更新时单独线程重建，建完后覆盖旧的索引，这造成重建后的索引的generation会比从库小，所以每次重建从库都会使用新的索引目录。</p>
<p>  	回到这个问题上来，临时的解决办法就是重启从库，因为重启后从库读index.properties发现index.20101122031000并不存在，默认就会使用index目录的索引，而index目录的generation会和主库不一致，就产生全复制，短时间内恢复成最新的索引，但这样可能后续还会触发到bug。但问题并没有解决，我继续查看主从库的日志，再没有发现其他错误信息。而这个问题之前是没有出现的，所以可能是最近的什么操作引起的。而最近的操作就是老高22号凌晨重建了索引，我对照了重建完成时间，是凌晨3点多，而从库的index.20101122031000应该就是之后复制生成的。</p>
<p>  	google之错误信息，找到https://issues.apache.org/jira/browse/SOLR-1822，报bug的人出现和我一样的错误！！不过他也不清楚索引目录怎么没了，他是建议发现目录没了就新建一个，这也是可以曲线解决问题的（我想了一下，这种方法应该不会产生句柄泄漏）。可问题是，好端端的一个索引目录怎么就没了。我继续查看从库复制相关的log，发现在下午3点10分之前都是有正常的复制log的。初想一下，复制过程中发生删除的地方也就是和过期的IndexCommit相关联的索引文件被删除，但也不会删除目录阿！！！莫不是被系统外界下的黑手？</p>
<p>  	回家的地铁上就在琢磨，复制过程有删除目录的操作吗？有！！！复制的最后会删除临时目录！！！那会不会是临时目录和目标索引目录重名了，结果复制后把索引目录给删除了？回家再回看错误日志，这临时索引目录名称是：index.20101122031500，而出错当时是下午的15:15,而不是凌晨的03:15，查找创建临时索引目录的代码：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"> <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #003399;">File</span> createTempindexDir<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> SolrCore core<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">String</span> tmpIdxDirName <span style="color: #339933;">=</span> <span style="color: #0000ff;">&quot;index.&quot;</span> <span style="color: #339933;">+</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">SimpleDateFormat</span><span style="color: #009900;">&#40;</span>SnapShooter.<span style="color: #006633;">DATE_FMT</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">format</span><span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">Date</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">File</span> tmpIdxDir <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">File</span><span style="color: #009900;">&#40;</span>core.<span style="color: #006633;">getDataDir</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>, tmpIdxDirName<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    tmpIdxDir.<span style="color: #006633;">mkdirs</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">return</span> tmpIdxDir<span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span></pre></div></div>

<p>  SnapShooter.DATE_FMT的定义是：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">String</span> DATE_FMT <span style="color: #339933;">=</span> <span style="color: #0000ff;">&quot;yyyyMMddhhmmss&quot;</span><span style="color: #339933;">;</span></pre></div></div>

<p>  注意时间串格式中的小时是“hh”（范围是01-12）而不是“HH”（范围是00-23），这就造成从库如果在当天上午某个时间点全复制，而不幸的，在下午的相应时刻复制又来了，就使得临时索引目录和当前索引目录名称相同，而在复制的最后误删掉当前索引目录。从Bug效果来说，这绝对有破坏性，并且其真的是够低级的。<br />
  我后来check了solr的trunk，发现这个bug已经被修复了，log显示是：<br />
<font color="blue"><br />
  r964312 | koji | 2010-07-15 14:32:22 +0800 (四, 2010-07-15) | 1 行<br />
  SOLR-1995: Use proper format for SimpleDateFormat<br />
</font><br />
  找到https://issues.apache.org/jira/browse/SOLR-1995，发现修复的初衷和我所出现的问题还不一样，它的破坏性也要小的多。话说又说回来，这个bug其实是很隐蔽的，触发的条件是：1）从库要在当天的上午某点进行全复制,如果使用DIH复制，这种情况是不会出现的，而我的系统的全复制通常是在下午，所以之前没出现该问题。2）下午的相应时刻也要发生复制，这要求复制间隔能被12小时整除（我的5分钟间隔中招了），并且当时主库的索引有更新变化（我的系统10多个库当时就有部分库就因为那个时间段没有更新而没有中招）。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/11/414.html/feed</wfw:commentRss>
		<slash:comments>5</slash:comments>
		</item>
		<item>
		<title>httpclient的并发连接问题</title>
		<link>http://www.kafka0102.com/2010/11/405.html</link>
		<comments>http://www.kafka0102.com/2010/11/405.html#comments</comments>
		<pubDate>Fri, 19 Nov 2010 17:56:18 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[java]]></category>
		<category><![CDATA[httpclient]]></category>
		<category><![CDATA[solr]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=405</guid>
		<description><![CDATA[	昨天的搜索系统又出状况了，几个库同时重建索引变得死慢。经过一个上午的复现分析，确定问题出现httpclient的使用上。搜索系统在重建索引时，是并发多个线程（默认是8个）不停的从PHP客户端取数据（当然，从另一个角度来说，搜索系统是客户端，PHP端是服务端），取回后放到一个队列里由单独的一个线程更新索引。在测试环境复现发现，对于一个请求，PHP端打印耗时是1-2秒，但搜索端打印在4-6秒。这种耗时差别也就两种可能性，一个是PHP端返回到搜索端接受完耗时太长，另一个就是搜索端在真正发给PHP端数据前等待了很久。因为有了之前的jetty7的困顿，起初我怀疑是传输数据的问题。因为请求数据的代码部分我只是简单的使用了httpclient，所以只能从httpclient着手分析。我想到把PHP端和搜索端的请求起始和结束时间都打出来对照一下，不过在这样做之前我把搜索端的并发请求线程数调到了1,看下单线程情况下效果如何，结果惊奇地发现PHP端和搜索端的耗时相近。所以，可以确定，是httpclient的并发连接处理上可能存在问题。]]></description>
			<content:encoded><![CDATA[<p>	昨天的搜索系统又出状况了，几个库同时重建索引变得死慢。经过一个上午的复现分析，确定问题出现httpclient的使用上（我使用的是3.1这个被广泛使用的遗留版本）。搜索系统在重建索引时，是并发多个线程（默认是8个）不停的从PHP客户端取数据（当然，从另一个角度来说，搜索系统是客户端，PHP端是服务端），取回后放到一个队列里由单独的一个或多个线程更新索引。在测试环境复现发现，对于一个请求，PHP端打印耗时是1-2秒，但搜索端打印在4-6秒。这种耗时差别也就两种可能性，一个是PHP端返回到搜索端接受完耗时太长，另一个就是搜索端在真正发给PHP端数据前等待了很久。因为有了之前的jetty7的困顿，起初我怀疑是传输数据的问题。因为请求数据的代码部分我只是简单的使用了httpclient，所以只能从httpclient着手分析。我想到把PHP端和搜索端的请求起始和结束时间都打出来对照一下，不过在这样做之前我把搜索端的并发请求线程数调到了1,看下单线程情况下效果如何，结果惊奇地发现PHP端和搜索端的耗时相近。所以，可以确定，是httpclient的并发连接处理上可能存在问题。不消说，翻开httpclient API中和配置相关的接口，结果找到HttpConnectionManagerParams类中下面两个函数：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">	<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> setDefaultMaxConnectionsPerHost<span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> maxHostConnections<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
	<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> setMaxTotalConnections<span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> maxTotalConnections<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>

<p>	httpclient在处理请求连接方面使用了连接池，它内部实际上有两种连接池，一种是全局的ConnectionPool，一种是每主机（per-host）HostConnectionPool。参数maxHostConnections就HostConnectionPool中表示每主机可保持连接的连接数，maxTotalConnections是ConnectionPool中可最多保持的连接数。每主机的配置类是HostConfiguration，HttpClient有个int executeMethod(final HostConfiguration hostConfiguration, final HttpMethod method)方法可以指定使用哪个HostConfiguration，不过多数情况都是不显示指定HostConfiguration，这样httpclient就用了默认的HostConfiguration=null，也就是说所有的请求可以认为指自同一个主机。如果不设置这两个参数，httpclient自然会用默认的配置，也就是MultiThreadedHttpConnectionManager中的：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">    <span style="color: #008000; font-style: italic; font-weight: bold;">/** The default maximum number of connections allowed per host */</span>
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> DEFAULT_MAX_HOST_CONNECTIONS <span style="color: #339933;">=</span> <span style="color: #cc66cc;">2</span><span style="color: #339933;">;</span>   <span style="color: #666666; font-style: italic;">// Per RFC 2616 sec 8.1.4</span>
&nbsp;
    <span style="color: #008000; font-style: italic; font-weight: bold;">/** The default maximum number of connections allowed overall */</span>
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> DEFAULT_MAX_TOTAL_CONNECTIONS <span style="color: #339933;">=</span> <span style="color: #cc66cc;">20</span><span style="color: #339933;">;</span></pre></div></div>

<p>    默认的maxHostConnections大小只有2,也就是说，在我并发8个线程请求数据时，实际上会有6个线程处于等待被调度,这也就解释上面的现象了。再看看上面的注释，我从http://www.faqs.org/rfcs/rfc2616.html找到从了RFC 2616 sec 8.1.4  Practical Considerations段落的最后一段：<br />
<font color="blue"><br />
       Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to  another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion.<br />
</font><br />
	看这叙述，也就表明人家httpclient设置maxHostConnections为2是有根有据的。不过，这种设置显然适合的是浏览器这种客户端，但我相信，多数使用httpclient并不希望有这种默认的限制。而它默认的只有20的maxTotalConnections也太有些吝啬了。我后来浏览solr的客户端server类CommonsHttpSolrServer的代码发现了下面的段落，solr要比我更了解httpclient：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">    _httpClient <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span>client <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">?</span> <span style="color: #000000; font-weight: bold;">new</span> HttpClient<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">new</span> MultiThreadedHttpConnectionManager<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">:</span> client<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>client <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #666666; font-style: italic;">// set some better defaults if we created a new connection manager and client</span>
      <span style="color: #666666; font-style: italic;">// increase the default connections</span>
      <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">setDefaultMaxConnectionsPerHost</span><span style="color: #009900;">&#40;</span> <span style="color: #cc66cc;">32</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>  <span style="color: #666666; font-style: italic;">// 2</span>
      <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">setMaxTotalConnections</span><span style="color: #009900;">&#40;</span> <span style="color: #cc66cc;">128</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span> <span style="color: #666666; font-style: italic;">// 20</span>
    <span style="color: #009900;">&#125;</span></pre></div></div>

<p>    对于httpclient，特别指出的是它的MultiThreadedHttpConnectionManager，看名字好像是多线程并发请求似的，其实不是，但它也确实用到了多线程，那是在发现连接不够用时起个等待线程wait信号，这个名称的含义应该是多线程情况线程安全的HttpConnectionManager。<br />
    使用httpclient还有两点经验，一个是创建的MultiThreadedHttpConnectionManager 实例最好是全局的，否则会有多个连接池，而HttpClient是线程安全的，可以多个实例。另一个是，在处理请求的最后，也就是finnaly里中，要调用method.releaseConnection();回收连接，否则连接池就可能会爆了。</p>
<p>补充：写完之后倒在床上，我又想起了几个问题，这里补充上：<br />
1、系统原先重建索引隐约记得速度还是可以的，为什么现在变慢得如此明显？这有两方面的原因，一个是原来系统取数据是用的单线程（我后来发现单取数据取数据速度跟不上更新索引的速度所以改成多线程），另一个是，当时重建没有一下子同时开启数个库。所以，即便是同样的代码，环境变了，效果也可能变了。当这种改变悄然发生了，程序员却没有捕捉到，才会第一直觉感到问题的诡异。<br />
2、对于长时间不能获得连接的情况，httpclient是否有warn日志报出来？因为我使用了httpclient的getResponseBodyAsStream方法，而它会打出warn日志，所以我是关掉了httpclient的warn级别的。所以，我又检查了httpclient的代码，可惜没看到相关warning log，这点httpclient可以改进下。不过httpclient现在都是4时代了，而我使用的还是3.1的，而3.x已经被停止更新了，所以再采用httpclient可以考虑4版本的，尽管现在能见到的代码几乎都是用的3.x系列的。<br />
3、httpclient的文档是否有特别提到连接数配置的情况？我翻看了一下，确实在关于threading一页中有提到。不过，我等使用它时显然没有完整阅毕它的文档。或许，httpclient给出个明显的最佳实践到能引起使用者的注意，否则误用的情况还是会时有发生。不信，google之。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/11/405.html/feed</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>solr拾遗：引用计数</title>
		<link>http://www.kafka0102.com/2010/11/401.html</link>
		<comments>http://www.kafka0102.com/2010/11/401.html#comments</comments>
		<pubDate>Sat, 13 Nov 2010 12:22:07 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[solr]]></category>
		<category><![CDATA[RefCounted]]></category>
		<category><![CDATA[SolrIndexSearcher]]></category>
		<category><![CDATA[SolrQueryRequest]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=401</guid>
		<description><![CDATA[	据我不完全统计，solr代码中使用引用计数的用途有两种：一个是引用资源，一个是引用对象。技术上来说引用计数的使用没多少可大说的，不过如果没有正确的close获得的资源和对象，泄漏的bug就出现了。]]></description>
			<content:encoded><![CDATA[<p>	据我不完全统计，solr代码中使用引用计数的用途有两种：一个是引用资源，一个是引用对象。技术上来说引用计数的使用没多少可大说的，不过如果没有正确的close获得的资源和对象，泄漏的bug就出现了。</p>
<h2>引用资源</h2>
<p>	solr中供使用的IndexSearcher是SolrIndexSearcher，使用时它是寄存于RefCounted。RefCounted的代码如下：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">abstract</span> <span style="color: #000000; font-weight: bold;">class</span> RefCounted<span style="color: #339933;">&lt;</span>Type<span style="color: #339933;">&gt;</span> <span style="color: #009900;">&#123;</span>
  <span style="color: #000000; font-weight: bold;">protected</span> <span style="color: #000000; font-weight: bold;">final</span> Type resource<span style="color: #339933;">;</span>
  <span style="color: #000000; font-weight: bold;">protected</span> <span style="color: #000000; font-weight: bold;">final</span> AtomicInteger refcount <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> AtomicInteger<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> RefCounted<span style="color: #009900;">&#40;</span>Type resource<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">resource</span> <span style="color: #339933;">=</span> resource<span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">int</span> getRefcount<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> refcount.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">final</span> RefCounted<span style="color: #339933;">&lt;</span>Type<span style="color: #339933;">&gt;</span> incref<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    refcount.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000000; font-weight: bold;">this</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">final</span> Type get<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> resource<span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> decref<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>refcount.<span style="color: #006633;">decrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">==</span> <span style="color: #cc66cc;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      close<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">protected</span> <span style="color: #000000; font-weight: bold;">abstract</span> <span style="color: #000066; font-weight: bold;">void</span> close<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>	SolrIndexSearcher关联于SolrCore。SolrCore成员变量</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">private</span> RefCounted<span style="color: #339933;">&lt;</span>SolrIndexSearcher<span style="color: #339933;">&gt;</span> _searcher</pre></div></div>

<p>表示正在使用的SolrIndexSearcher，而成员变量</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> LinkedList<span style="color: #339933;">&lt;</span>RefCounted<span style="color: #339933;">&lt;</span>SolrIndexSearcher<span style="color: #339933;">&gt;&gt;</span> _searchers</pre></div></div>

<p>保持了未close的SolrIndexSearcher列表。在主从模式下，当新的索引到来时，solr会创建了新的</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">RefCounted<span style="color: #339933;">&lt;</span>SolrIndexSearcher<span style="color: #339933;">&gt;</span></pre></div></div>

<p>实例赋为_searcher并添加到_searchers。上面的RefCounted的close方法是抽象的，在SolrCore中有其实现：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">  <span style="color: #000000; font-weight: bold;">private</span> RefCounted<span style="color: #339933;">&lt;</span>SolrIndexSearcher<span style="color: #339933;">&gt;</span> newHolder<span style="color: #009900;">&#40;</span>SolrIndexSearcher newSearcher<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    RefCounted<span style="color: #339933;">&lt;</span>SolrIndexSearcher<span style="color: #339933;">&gt;</span> holder <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> RefCounted<span style="color: #339933;">&lt;</span>SolrIndexSearcher<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#40;</span>newSearcher<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> close<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
          <span style="color: #000000; font-weight: bold;">synchronized</span><span style="color: #009900;">&#40;</span>searcherLock<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
            <span style="color: #666666; font-style: italic;">// it's possible for someone to get a reference via the _searchers queue</span>
            <span style="color: #666666; font-style: italic;">// and increment the refcount while RefCounted.close() is being called.</span>
            <span style="color: #666666; font-style: italic;">// we check the refcount again to see if this has happened and abort the close.</span>
            <span style="color: #666666; font-style: italic;">// This relies on the RefCounted class allowing close() to be called every</span>
            <span style="color: #666666; font-style: italic;">// time the counter hits zero.</span>
            <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>refcount.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">&gt;</span> <span style="color: #cc66cc;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #000000; font-weight: bold;">return</span><span style="color: #339933;">;</span>
            _searchers.<span style="color: #006633;">remove</span><span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">this</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
          <span style="color: #009900;">&#125;</span>
          resource.<span style="color: #006633;">close</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #003399;">IOException</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
          log.<span style="color: #006633;">error</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;Error closing searcher:&quot;</span> <span style="color: #339933;">+</span> SolrException.<span style="color: #006633;">toStr</span><span style="color: #009900;">&#40;</span>e<span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
      <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
    holder.<span style="color: #006633;">incref</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>  <span style="color: #666666; font-style: italic;">// set ref count to 1 to account for this._searcher</span>
    <span style="color: #000000; font-weight: bold;">return</span> holder<span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span></pre></div></div>

<p>  可以看到，当调用RefCounted的close方法，如果引用计数refcount不大于0,则会调用SolrIndexSearcher的close方法来释放其保持的文件句柄，并且将其从_searchers中移除。<br />
  再来看查询时对SolrIndexSearcher的使用。每个请求是要构造SolrQueryRequest，看下SolrQueryRequestBase代码片断：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">  <span style="color: #000000; font-weight: bold;">protected</span> RefCounted<span style="color: #339933;">&lt;</span>SolrIndexSearcher<span style="color: #339933;">&gt;</span> searcherHolder<span style="color: #339933;">;</span>
  <span style="color: #000000; font-weight: bold;">public</span> SolrIndexSearcher getSearcher<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">if</span><span style="color: #009900;">&#40;</span>core <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//a request for a core admin will no have a core</span>
    <span style="color: #666666; font-style: italic;">// should this reach out and get a searcher from the core singleton, or</span>
    <span style="color: #666666; font-style: italic;">// should the core populate one in a factory method to create requests?</span>
    <span style="color: #666666; font-style: italic;">// or there could be a setSearcher() method that Solr calls</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>searcherHolder<span style="color: #339933;">==</span><span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      searcherHolder <span style="color: #339933;">=</span> core.<span style="color: #006633;">getSearcher</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">return</span> searcherHolder.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> close<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>searcherHolder<span style="color: #339933;">!=</span><span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      searcherHolder.<span style="color: #006633;">decref</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      searcherHolder <span style="color: #339933;">=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
  <span style="color: #009900;">&#125;</span></pre></div></div>

<p> 当调用SolrQueryRequestBase的getSearcher()时，如果是第一次调用，会转而调用core.getSearcher()，其会使得到的searcherHolder的引用计数增一。而对称的，SolrQueryRequestBase的close()方法使searcherHolder的引用计数减一，一增一减平衡了。这也使得当新的索引到来时，仍旧提供查询服务的SolrIndexSearcher不会立即关闭，直到其引用计数减为0才关闭。所以，在使用SolrQueryRequest时，要确保请求的最后调用其close方法，否则那些无用的SolrIndexSearcher就不会被释放，直到句柄耗尽或者OOM掉。</p>
<h2>引用对象</h2>
<p>	引用对象的典型使用是SolrCore，看下相关代码：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> AtomicInteger refCount <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> AtomicInteger<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">1</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">void</span> open<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    refCount.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> close<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000066; font-weight: bold;">int</span> count <span style="color: #339933;">=</span> refCount.<span style="color: #006633;">decrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>count <span style="color: #339933;">&gt;</span> <span style="color: #cc66cc;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #000000; font-weight: bold;">return</span><span style="color: #339933;">;</span> <span style="color: #666666; font-style: italic;">// close is called often, and only actually closes if nothing is using it.</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>count <span style="color: #339933;">&lt;</span> <span style="color: #cc66cc;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      log.<span style="color: #006633;">error</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;Too many close [count:{}] on {}. Please report this exception to solr-user@lucene.apache.org&quot;</span>, count, <span style="color: #000000; font-weight: bold;">this</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">return</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #666666; font-style: italic;">//释放对象和资源</span>
  <span style="color: #009900;">&#125;</span></pre></div></div>

<p>	引用计数就是refCount了。SolrCore实例的获得是通过调用CoreContainer的 SolrCore getCore(String name)得到，其实现是：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">public</span> SolrCore getCore<span style="color: #009900;">&#40;</span><span style="color: #003399;">String</span> name<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">synchronized</span><span style="color: #009900;">&#40;</span>cores<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      SolrCore core <span style="color: #339933;">=</span> cores.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>name<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>core <span style="color: #339933;">!=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span>
        core.<span style="color: #006633;">open</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>  <span style="color: #666666; font-style: italic;">// increment the ref count while still synchronized</span>
      <span style="color: #000000; font-weight: bold;">return</span> core<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
  <span style="color: #009900;">&#125;</span></pre></div></div>

<p>  也就是说CoreContainer显示的open了SolrCore，所以在得到SolrCore实例后，也需要显示的close它。因为SolrQueryRequestBase用到了SolrCore，所以在处理请求的最后，要确保调用了SolrCore的close方法。当然，对于查询端，SolrCore实例在整个生命周期内通常并不会真正被close，除非显示的调用了reload等操作。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/11/401.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>oracle提升mysql售价，误解与无视</title>
		<link>http://www.kafka0102.com/2010/11/397.html</link>
		<comments>http://www.kafka0102.com/2010/11/397.html#comments</comments>
		<pubDate>Sat, 06 Nov 2010 18:13:38 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[mysql]]></category>
		<category><![CDATA[oracle]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=397</guid>
		<description><![CDATA[前几天就在twitter上看到oracle提升mysql售价的消息，推上的英文圈对oracle的这一举动议论纷纷，但中文圈很是平静，倒是像 csdn、javaeye这样的媒体做头条报道：Oracle提高了MySQL的售价。这一新闻难免又引起大家对mysql前景的悲观，对迁移存储系统到postgre、nosql等蠢蠢欲动。其实多数mysql使用者可以轻松的忽略这一新闻，因为人家 oracle提价mysql影响的是买收费版本mysql的用户，并不会影响我等使用免费版本的用户。
]]></description>
			<content:encoded><![CDATA[<p>前几天就在twitter上看到oracle提升mysql售价的消息，推上的英文圈对oracle的这一举动议论纷纷，但中文圈很是平静，倒是像csdn、javaeye这样的媒体做头条报道：<a href="http://www.javaeye.com/news/18425" target="_blank">Oracle提高了MySQL的售价</a>。这一新闻难免又引起大家对mysql前景的悲观，对迁移存储系统到postgre、nosql等蠢蠢欲动。其实多数mysql使用者可以轻松的忽略这一新闻，因为人家oracle提价mysql影响的是买收费版本mysql的用户，并不会影响我等使用免费版本的用户。</p>
<p>国内媒体在转载这一新闻时也没有深究其信息的准确性，对于一些概念解释上的偏差，容易引起误解。这一新闻的源头就是<a href="http://www.mysql.com/products/" target="_blank">http://www.mysql.com/products/</a>的价格表上的变化。从该网页可看到，mysql版本有5个，免费的版本有Community Edition和Classic Edition（现有页面没有这个称呼，它就是MySQL as an Embedded Database），收费的版本有Enterprise Edition、Standard Edition、Cluster Carrier Grade Edition。oracle对mysql售价的调整有两方面：</p>
<p>1、提高三种收费版本的价格，比如Standard Edition现在2000$,而原来的价格只有599$，这对付费用户确实有些影响。这里插段说明，可以将数据库使用者分两种，一种是使用免费的，出了问题自己搞定或搞不定，比如广大的互联网从业人员;另一种是花钱买保障的，产品使用中出了问题直接找提供商解决，比如使用IT产品的企业用户，它们买的不单单是个软件，更是其提供的服务，这后者才是oracle的目标用户（你facebook纵有那么庞大的mysql集群，也没有为oracle贡献几毛钱的利润，就不是oracle的目标用户）。对于数据库产品来说，像oracle数据库高昂的价格是中小企业无力承担的，转而使用价格低廉得多的mysql就很划算。oracle收购mysql（sun）后，oracle数据库面向大型企业，mysql面向中小企业，高低端都覆盖到了。oracle显然认为sun时期的mysql售价过于保守了，要通过提高价格赚更多的钱。这一举动是否会使其原有用户放弃mysql转用其他产品也未可知，但就数据库产品来说，像oracle、db2、sql server都太贵，便宜的也就mysql、postgre了。这次价格调整也给了其他mysql商业支持企业的一些机会，比如由一些mysql专家组成的SkySQL公司，就借此宣传了<a href="http://www.skysql.com/en/letter-to-mysql-customers" target="_blank">其更低廉的mysql商业支持</a>，Percona也站出来表明其<a href="http://www.mysqlperformanceblog.com/2010/11/05/perconas-commitments-to-mysql-users/" target="_blank">拥抱用户的立场</a> 。</p>
<p>2、oracle对Classic Edition不再提供对innodb SE的支持，使得免费的嵌入式mysql只支持myisam引擎，oracle此举是要毁掉免费的嵌入式mysql，驱使使用嵌入式mysql的ISV、OEMs、VAR们都使用收费版本。这一变化对普通用户没什么影响，不过有人错以为Classic Edition是Community Edition，引发不必要的恐慌。</p>
<p>比较mysql的免费和收费版本的功能，收费版本多了些如Enterprise Backup、Enterprise Monitor等高级功能，可我们不都从来没用上innodb的backup工具吗？也就不值一说了。只是，像oracle只顾谋取暴利的做法总是会让热爱开源事业的人反感。最近oracle宣布在openjdk之外推出收费版本的消息，也是勾起大家的纷纷议论，但mysql、java的气候显然不是oracle一家能阻挡的了的。话又说回来，mysql这些年在sun的温柔乡里发展得明显有些迟缓，如果mysql卖得好，也许oracle会投入更多资源。就算oracle手中的mysql凋零，如maria、drizzle等分支够争气，也会是用户平滑迁移的好选择。而java这块，随着jrocket和hotspot的融合，也会进一步增强jvm，即便是免费的版本。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/11/397.html/feed</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>solr拾遗：CopyField</title>
		<link>http://www.kafka0102.com/2010/10/394.html</link>
		<comments>http://www.kafka0102.com/2010/10/394.html#comments</comments>
		<pubDate>Sat, 30 Oct 2010 15:51:46 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[solr]]></category>
		<category><![CDATA[copyfiled]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=394</guid>
		<description><![CDATA[	solr的index schema中，除了支持基本数值类型的field，还支持一些特别的field，比如较常用的CopyField。以下面的schema配置片断为例：

&#60;schema name=&#34;eshequn.post.db_post.0&#34; version=&#34;1.1&#34;
    xmlns:xi=&#34;http://www.w3.org/2001/XInclude&#34;&#62;
     &#60;fields&#62;
     	&#60;!-- for title --&#62;
        &#60;field name=&#34;t&#34; type=&#34;text&#34; indexed=&#34;true&#34; stored=&#34;false&#34; /&#62;
        &#60;!-- for abstract --&#62;
        &#60;field name=&#34;a&#34; type=&#34;text&#34; [...]]]></description>
			<content:encoded><![CDATA[<p>	solr的index schema中，除了支持基本数值类型的field，还支持一些特别的field，比如较常用的CopyField。以下面的schema配置片断为例：</p>

<div class="wp_syntax"><div class="code"><pre class="xml" style="font-family:monospace;"><span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;schema</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;eshequn.post.db_post.0&quot;</span> <span style="color: #000066;">version</span>=<span style="color: #ff0000;">&quot;1.1&quot;</span></span>
<span style="color: #009900;">    <span style="color: #000066;">xmlns:xi</span>=<span style="color: #ff0000;">&quot;http://www.w3.org/2001/XInclude&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>
     <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;fields<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
     	<span style="color: #808080; font-style: italic;">&lt;!-- for title --&gt;</span>
        <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;field</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;t&quot;</span> <span style="color: #000066;">type</span>=<span style="color: #ff0000;">&quot;text&quot;</span> <span style="color: #000066;">indexed</span>=<span style="color: #ff0000;">&quot;true&quot;</span> <span style="color: #000066;">stored</span>=<span style="color: #ff0000;">&quot;false&quot;</span> <span style="color: #000000; font-weight: bold;">/&gt;</span></span>
        <span style="color: #808080; font-style: italic;">&lt;!-- for abstract --&gt;</span>
        <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;field</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;a&quot;</span> <span style="color: #000066;">type</span>=<span style="color: #ff0000;">&quot;text&quot;</span> <span style="color: #000066;">indexed</span>=<span style="color: #ff0000;">&quot;true&quot;</span> <span style="color: #000066;">stored</span>=<span style="color: #ff0000;">&quot;false&quot;</span> <span style="color: #000000; font-weight: bold;">/&gt;</span></span>
        <span style="color: #808080; font-style: italic;">&lt;!-- for title and abstract --&gt;</span>
        <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;field</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;ta&quot;</span> <span style="color: #000066;">type</span>=<span style="color: #ff0000;">&quot;text&quot;</span> <span style="color: #000066;">indexed</span>=<span style="color: #ff0000;">&quot;true&quot;</span> <span style="color: #000066;">stored</span>=<span style="color: #ff0000;">&quot;false&quot;</span> <span style="color: #000066;">multiValued</span>=<span style="color: #ff0000;">&quot;true&quot;</span><span style="color: #000000; font-weight: bold;">/&gt;</span></span>
    <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/fields<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
    <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;copyField</span> <span style="color: #000066;">source</span>=<span style="color: #ff0000;">&quot;t&quot;</span> <span style="color: #000066;">dest</span>=<span style="color: #ff0000;">&quot;ta&quot;</span> <span style="color: #000000; font-weight: bold;">/&gt;</span></span>
    <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;copyField</span> <span style="color: #000066;">source</span>=<span style="color: #ff0000;">&quot;a&quot;</span> <span style="color: #000066;">dest</span>=<span style="color: #ff0000;">&quot;ta&quot;</span> <span style="color: #000000; font-weight: bold;">/&gt;</span></span>
<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/schema<span style="color: #000000; font-weight: bold;">&gt;</span></span></span></pre></div></div>

<p>	字段t是文章的标题，字段a是文章的摘要，字段ta是文章标题和摘要的联合。添加索引文档时，只需要传入t和a字段的内容，solr会自动索引ta字段。这算不上多高级的功能，不过如果让你来实现这个功能，你会怎么做呢？我接手的搜索系统原来就有类似的功能，它的做法是，将t和a字段的文本合并，塞到ta字段，无可厚非的做法。不过，有人注意到lucene的Document类提供的public final Field[] getFields(String name)类似函数不？也就是说，lucene中的一个name可以对应多个Field。solr在添加索引时，会检查field name是不是copyField集合中的source，是的话就以其value构造dest field。如果dest由多个source构成，就需要将其指定为multiValued。</p>
<p>	对于查询来说，如果查询字段要来自多个字段，一种选择是使用CopyField，化多个字段为一个字段，缺点是不能区分各个字段的重要度差别。比如文章的标题和摘要，标题就要比摘要重要性更强，如果有这方面的要求，可以选择查询多个字段的做法。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/10/394.html/feed</wfw:commentRss>
		<slash:comments>8</slash:comments>
		</item>
		<item>
		<title>各种java序列化工具性能对比</title>
		<link>http://www.kafka0102.com/2010/10/383.html</link>
		<comments>http://www.kafka0102.com/2010/10/383.html#comments</comments>
		<pubDate>Sun, 24 Oct 2010 10:58:42 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[java]]></category>
		<category><![CDATA[序列化]]></category>
		<category><![CDATA[评测]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=383</guid>
		<description><![CDATA[看到一个很不错的工具http://github.com/eishay/jvm-serializers/，可以用它来评测各种流行的java序列化反序列化工具的性能，使用上也很简单。想试试该工具的，下载源码后参考起README操作即可。而我更关心的是，是各种工具的性能对比，以作选择的一个衡量标准，也就是http://github.com/eishay/jvm-serializers/wiki的图示和数据。本文也就简单转摘其图示，图示中的java-manual指的是根据对象（数据）格式手工操作（当然是最快的，但不具有通用性），java- buildin-in就是内置的序列化方式（ObjectOutputStream、ObjectInputStream），其他工具的使用版本可以查看其wiki。]]></description>
			<content:encoded><![CDATA[<p>看到一个很不错的工具<a href="http://github.com/eishay/jvm-serializers/" target="_blank">http://github.com/eishay/jvm-serializers/</a>，可以用它来评测各种流行的java序列化反序列化工具，使用上也很简单。想试试该工具的，下载源码后参考起README操作即可。而我更关心的是，是各种工具的性能对比，以作选择的一个衡量标准，也就是<a href="http://github.com/eishay/jvm-serializers/wiki" target="_blank">http://github.com/eishay/jvm-serializers/wiki</a>的图示和数据。本文也就简单转摘其图示，图示中的java-manual指的是根据对象（数据）格式手工操作（当然是最快的，但不具有通用性），java-buildin-in就是内置的序列化方式（ObjectOutputStream、ObjectInputStream），其他工具的使用版本可以查看其wiki。</p>
<h2>1、Total Time (“total”)</h2>
<p>创建一个对象，将其序列化成一个字节数组，然后再反序列化成一个对象。</p>
<p><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/chart1.png"><img class="alignleft size-full wp-image-384" title="chart1" src="http://www.kafka0102.com/wp-content/uploads/2010/10/chart1.png" alt="" width="700" height="402" /></a></p>
<h2>2、Serialization Time (“ser”)</h2>
<p>创建一个对象，将其序列化成一个字节数组。</p>
<p><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/chart2.png"><img class="alignleft size-full wp-image-385" title="chart2" src="http://www.kafka0102.com/wp-content/uploads/2010/10/chart2.png" alt="" width="700" height="402" /></a></p>
<h2>3、Deserialization Time (“deser+deep”)</h2>
<p>相比于序列化，反序列化更耗时。为了更公平的比较，jvm-serializers在反序列化测试时访问了反序列化得到的对象的所有字段（也就是deep的含义），因为部分工具反序列化时“偷懒”而没有做足工作。</p>
<p><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/chart3.png"><img class="alignleft size-full wp-image-386" title="chart3" src="http://www.kafka0102.com/wp-content/uploads/2010/10/chart3.png" alt="" width="700" height="402" /></a></p>
<h2>4、Serialized Size (“size”)</h2>
<p>序列化数据的大小，这个大小会依赖于使用的数据。</p>
<p><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/chart4.png"><img class="alignleft size-full wp-image-387" title="chart4" src="http://www.kafka0102.com/wp-content/uploads/2010/10/chart4.png" alt="" width="700" height="402" /></a></p>
<h2>5、Serialization Compressed Size (“size+dfl”)</h2>
<p>使用java内置的DEFLATE(zlib)压缩的序列化数据的大小。</p>
<p><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/chart5.png"><img class="alignleft size-full wp-image-388" title="chart5" src="http://www.kafka0102.com/wp-content/uploads/2010/10/chart5.png" alt="" width="700" height="402" /></a></p>
<h2>6、Object Creation Time (“create”)</h2>
<p>对象创建耗时很短（平均100纳秒）所以通常的比较没什么意义。不过，不同工具创建的对象在表现上会有不同。有的工具只是创建普通的java类，你可以直接访问其字段，而有的使用get/set方法，有的使用builder模式。<br />
<a href="http://www.kafka0102.com/wp-content/uploads/2010/10/chart6.png"><img class="alignleft size-full wp-image-389" title="chart6" src="http://www.kafka0102.com/wp-content/uploads/2010/10/chart6.png" alt="" width="700" height="402" /></a></p>
<p>分析这些对比，java内置的序列化方式性能很差（这才催生了各种序列化工具）。在这些工具中，protostuff表现极为出色，盖过了名头响亮的protobuff和thrift。通用格式中，json要比xml强不少，而不同工具对同样格式的性能表现也有差别，这也给了选择工具的一个指导。另一个值得一提的是bson，尽管jvm-serializers没有包含它，相信性能上应该不错。也可以参考jvm-serializers已有工具评测代码的实现，添加比如处理bson、php等格式的序列化工具的评测。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/10/383.html/feed</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>mysql的多线程复制</title>
		<link>http://www.kafka0102.com/2010/10/380.html</link>
		<comments>http://www.kafka0102.com/2010/10/380.html#comments</comments>
		<pubDate>Sun, 24 Oct 2010 08:39:17 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[mysql]]></category>
		<category><![CDATA[mysql 复制]]></category>
		<category><![CDATA[slave master]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=380</guid>
		<description><![CDATA[对于使用主从复制的mysql用户来说，经常会遇到的问题是，当主库的写压力增大时，基于单线程复制的从库跟不上主库的写速度，造成应用从从库读到脏数据。如果在数据库前面使用了cache，那么这种脏数据的影响就更恶劣。更糟糕的是，如果在主库上执行了耗时很长的sql，那么从库就会被完全阻塞，这也限制了程序员们不要在主库执行耗时长的语句，更不要提那些alter schema的语句。对于mysql主从的这个缺陷，可以使用一些其他方式规避它，比如做sharding处理，控制每个master的写压力在合适的范围内，也可以去主从，通过应用来分发请求到各个db（应用相当于主库），但这些做法相比复制无疑会复杂的多。]]></description>
			<content:encoded><![CDATA[<p>对于使用主从复制的mysql用户来说，经常会遇到的问题是，当主库的写压力增大时，基于单线程复制的从库跟不上主库的写速度，造成应用从从库读到脏数据。如果在数据库前面使用了cache，那么这种脏数据的影响就更恶劣。更糟糕的是，如果在主库上执行了耗时很长的sql，那么从库就会被完全阻塞，这也限制了程序员们不要在主库执行耗时长的语句，更不要提那些alter schema的语句。对于mysql主从的这个缺陷，可以使用一些其他方式规避它，比如做sharding处理，控制每个master的写压力在合适的范围内，也可以去主从，通过应用来分发请求到各个db（应用相当于主库），但这些做法相比复制无疑会复杂的多。</p>
<p>我很早就有的一个想法是，为什么不做一个mysql的多线程复制（并行复制），使得从库的写速度能跟上主库，而是一直使用这种低效的单线程复制？那时我没有从mysql官方看到这方面的消息。最近，看到Baron Schwartz的<a href="http://www.mysqlperformanceblog.com/2010/10/20/mysql-limitations-part-1-single-threaded-replication/ " target="_blank">MySQL Limitations Part 1: Single-Threaded Replication</a>，值得分享和讨论一下这个问题。</p>
<p>Baron Schwartz在文章中提到了多线程复制的3种思路，列举如下:</p>
<p>1）每个数据库一个线程。这样做的前提是各个数据库之间是独立的，主库不需要做什么修改，从库上对每个数据库起一个线程执行复制过来的binary log。这种方式的优点在于，基于现有的复制机制修改起来较为简单，缺点就是它不具有通用性。但我想，大多数应用的事务都不是跨数据库的，另一方面，是否可以做成，通过配置，配置哪些独立的数据库可以起独立线程，而关联的数据库还是使用原来的复制方式？</p>
<p>2）在从库上由一个协调线程（coordinator）负责分发任务给worker线程池。协调线程读relay log到一个事务段（而不是一个语句），将整个事务交由某个工作线程执行，工作线程执行到COMMIT，但不是真正的commit，然后回报给协调线程。协调线程将确保所有的事务以相同的顺序开始和提交。如果出现死锁或者锁等待超时等错误，协调线程要使工作线程回滚，并且重试或者串行化执行有问题的事务。相比于1），该方法具有通用性，缺点是实现起来会更复杂，并且当出现死锁等问题时，是否能正确的处理好也是个问题。</p>
<p>3）主库上有多个binlog，每个数据库一个，这样从库的复制单位就是单个数据库的binlog。这也使得，从库可以从多个主库复制，这对于一些场景会很有用。该方法的缺点是，对现有的复制机制改动很大，包括现有的复制配置、管理命令等都需要做很大的改变。</p>
<p>说完上面的实现思路，接下来的问题就是，mysql是否会做多线程复制（且不说它使用哪种方式）？在<a href="http://www.mysqlperformanceblog.com/2010/10/20/mysql-limitations-part-1-single-threaded-replication/ " target="_blank">MySQL Limitations Part 1: Single-Threaded Replication</a>的评论中，有人给出了这个<a href="http://forge.mysql.com/wiki/ReplicationFeatures/ParallelSlave" target="_blank">http://forge.mysql.com/wiki/ReplicationFeatures/ParallelSlave</a>链接，就是说mysql从5.1开始做这个事情，只是没想到其如此低调，连Baron Schwartz似乎都不知晓。在<a href="http://forge.mysql.com/worklog/task.php?id=4648" target="_blank">http://forge.mysql.com/worklog/task.php?id=4648</a>可以看到mysql对多线程复制的一些思路，它是基于5.1的行复制（估计现在多数应用还是基于语句复制），实现思路似乎和上面的2）有些相像。不知道以mysql的办事效率，该功能何年何月才能成熟起来。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/10/380.html/feed</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>solr filter query的误用</title>
		<link>http://www.kafka0102.com/2010/10/374.html</link>
		<comments>http://www.kafka0102.com/2010/10/374.html#comments</comments>
		<pubDate>Fri, 22 Oct 2010 02:06:01 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[solr]]></category>
		<category><![CDATA[filter query]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=374</guid>
		<description><![CDATA[在我整理solr SolrIndexSearcher性能问题分析的时候，我就在想，我是不是误用了SolrIndexSearcher，才出现我所以为的性能问题。当我想到其实现特点，我恍惚确定确实是这样的，根源就是我误用 filter query。上篇文章暂且放下，这篇做个补充。如我上文所述，fq=fid:1这个条件匹配的文档数会很多，不过，如果开启了filter cache，那么只会在第一次调用时慢些，后续的调用都会命中cache而提升速度，而通过query预热是可以解决初次调用的低速问题。所以，如果要使用filter query，就要开启filter cache，并确保filter cache能容纳所有的filter query。这也需要对应用的查询特点做好分析。以fq=atm:[int_time1 TO int_time2]为例，我之前是将它放到filter query中，不过因为每次查询的int_time1和int_time2都几乎不相同，使得总不能命中filter cache，严重影响了查询性能。另一方面，我在做压力测试时也发现，当filter query结果充满了filter cache，最终使得程序内存耗尽。]]></description>
			<content:encoded><![CDATA[<p>在我整理<a title="Permanent Link to solr SolrIndexSearcher性能问题分析" rel="bookmark" href="../2010/10/366.html">solr  SolrIndexSearcher性能问题分析</a>的时候，我就在想，我是不是误用了SolrIndexSearcher，才出现我所以为的性能问题。当我想到其实现特点，我恍惚确定确实是这样的，根源就是我误用filter query。上篇文章暂且放下，这篇做个补充。如我上文所述，fq=fid:1这个条件匹配的文档数会很多，不过，如果开启了filter cache，那么只会在第一次调用时慢些，后续的调用都会命中cache而提升速度，而通过query预热是可以解决初次调用的低速问题。所以，如果要使用filter query，就要开启filter cache，并确保filter cache能容纳所有的filter query。这也需要对应用的查询特点做好分析。以fq=atm:[int_time1 TO int_time2]为例，我之前是将它放到filter query中，不过因为每次查询的int_time1和int_time2都几乎不相同，使得总不能命中filter cache，严重影响了查询性能。另一方面，我在做压力测试时也发现，当filter query结果充满了filter cache，最终使得程序内存耗尽。</p>
<p>对filter query的使用，还有个注意点，就是solr是按每个fq参数缓存结果的，所以fq=fid:1、fq=fid:2、fq=(fid:1 OR fid:2)是3个缓存项。而像atm:[int_time1 TO int_time2]这样的范围查询，如果个数不是可枚举的，就不要使用它作为fq。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/10/374.html/feed</wfw:commentRss>
		<slash:comments>6</slash:comments>
		</item>
		<item>
		<title>solr SolrIndexSearcher性能问题分析</title>
		<link>http://www.kafka0102.com/2010/10/366.html</link>
		<comments>http://www.kafka0102.com/2010/10/366.html#comments</comments>
		<pubDate>Thu, 21 Oct 2010 15:22:52 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[solr]]></category>
		<category><![CDATA[lucene]]></category>
		<category><![CDATA[性能]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=366</guid>
		<description><![CDATA[基于solr的新搜索系统已经使用了有段时间，不过之前上的几个库都很小，也没发现什么性能问题。最近上了thread和post库后，性能问题显现出来。这段时间解决的性能问题有好几个，本文只着重于SolrIndexSearcher的search问题。thread库索引大小有400M多，记录数有400多万，原来的基于lucene的系统的查询处理耗时一般在10-30ms，而同样的请求，新系统耗时在50-100ms。而post库索引大小有 6G多，记录数有3000多万，旧系统的查询处理时间一般在30-80ms，而新系统往往需要400-800ms，性能差距还是很明显的。
因为SolrIndexSearcher是基于lucene的IndexSearcher，想必应该是青出于蓝胜于蓝，并且其wiki上给出的性能数据也很不错，所以当我把新系统做完后并没有去细致的测试其性能。但现在系统的性能相比之下确实有些差，并且因为要做多个库的整合搜索，就要避免像thread 这样耗时长的库就成为短板。总之，问题需要定位和解决。]]></description>
			<content:encoded><![CDATA[<h2>问题背景</h2>
<p>基于solr的新搜索系统已经使用了有段时间，不过之前上的几个库都很小，也没发现什么性能问题。最近上了thread和post库后，性能问题显现出来。这段时间解决的性能问题有好几个，本文只着重于SolrIndexSearcher的search问题。thread库索引大小有400M多，记录数有400多万，原来的基于lucene的系统的查询处理耗时一般在10-30ms，而同样的请求，新系统耗时在50-100ms。而post库索引大小有6G多，记录数有3000多万，旧系统的查询处理时间一般在30-80ms，而新系统往往需要400-800ms，性能差距还是很明显的。</p>
<p>因为SolrIndexSearcher是基于lucene的IndexSearcher，想必应该是青出于蓝胜于蓝，并且其wiki上给出的性能数据也很不错，所以当我把新系统做完后并没有去细致的测试其性能。但现在系统的性能相比之下确实有些差，并且因为要做多个库的整合搜索，就要避免像thread这样耗时长的库就成为短板。总之，问题需要定位和解决。</p>
<h2>原因分析</h2>
<p>分析thread应用给搜索server发来的query，query除了包含用户输入的文本，还有两个filter query:1)是fq=fid:1，fid是int型，表示选择某个版块;2）是fq=atm:[int_time1 TO int_time2]，atm是int型，表示帖子发表的时间，默认int_time2是请求时的时间，int_time1是比int_time2小6个月的起始时间。实际测试发现，当去掉两个fq，处理耗时很短（即便有sort返回结果数很多），而两个fq同时存在或则其中某一个存在时，耗时都会变长，尤其是匹配的结果总数较多时。<br />
因为query分为filter query和text query，基于lucene的IndexSearcher实现方式可以有两种：1）是将各filter query和text query联合起来构成一个大的Query（就是个BooleanQuery，BooleanClause之间是MUST关系）；2）是将filter query转成Filter，和text query区分开。旧系统使用的就是方式2。因为Filter是基于Query构造的，所以两种方式按说性能应该差不多。</p>
<p>翻看SolrIndexSearcher的代码，它除了继承IndexSearcher的一系列search方法，还增加了search(QueryResult qr, QueryCommand cmd)，也是我使用的接口。我当然可以认为，solr的这个search方法会对查询过程做了更高效的实现。除去为提升性能（或者降低性能）引入的三种cache代码，以及一些细枝末节的分支代码，SolrIndexSearcher的search方法主要处理过程其实也简单：</p>
<p>1）将一或多个fq构成List&lt;Query&gt;，然后调用getDocSet(final List&lt;Query&gt; queries)得到DocSet，DocSet存放的是匹配的doc id列表，getDocSet方法内部就是遍历queries，对每一个query执行getPositiveDocSet(final Query q)得到该query的DocSet，然后将得到的多个DocSet做合并，得到满足所有fq的DocSet。</p>
<p>2）由DocSet的Filter getTopFilter()方法得到Filter，再联合TopFieldCollector或者TopScoreDocCollector，以及由text query构造的Qeury，调用lucene的void search(Query query, Filter filter, Collector results)，再结合一些扫尾工作，完成一次查询过程。</p>
<p>单看这个过程，也很难发现问题在哪。为了跟踪整个查询的执行路径，我索性将SolrIndexSearcher从solr源码中提取出来，这其间也是费了一些周折，因为SolrIndexSearcher使用了一些包级别的代码并且我需要构造的SolrIndexSearcher2不能直接从SolrQueryRequest获得。通过加入一些代码段的耗时统计，发现上述1）耗时占70%以上，再分析发现，问题出在查询fq的DocSet getDocSetNC(final Query query, final DocSet filter)方法中使用的DocSetCollector。DocSetCollector的代码如下：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">class</span> DocSetCollector <span style="color: #000000; font-weight: bold;">extends</span> Collector <span style="color: #009900;">&#123;</span>
  <span style="color: #000066; font-weight: bold;">int</span> pos<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>
  OpenBitSet bits<span style="color: #339933;">;</span>
  <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> maxDoc<span style="color: #339933;">;</span>
  <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> smallSetSize<span style="color: #339933;">;</span>
  <span style="color: #000066; font-weight: bold;">int</span> base<span style="color: #339933;">;</span>
  <span style="color: #666666; font-style: italic;">// in case there aren't that many hits, we may not want a very sparse</span>
  <span style="color: #666666; font-style: italic;">// bit array.  Optimistically collect the first few docs in an array</span>
  <span style="color: #666666; font-style: italic;">// in case there are only a few.</span>
  <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span><span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> scratch<span style="color: #339933;">;</span>
  DocSetCollector<span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> smallSetSize, <span style="color: #000066; font-weight: bold;">int</span> maxDoc<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">smallSetSize</span> <span style="color: #339933;">=</span> smallSetSize<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">maxDoc</span> <span style="color: #339933;">=</span> maxDoc<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">scratch</span> <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #000066; font-weight: bold;">int</span><span style="color: #009900;">&#91;</span>smallSetSize<span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> collect<span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> doc<span style="color: #009900;">&#41;</span> <span style="color: #000000; font-weight: bold;">throws</span> <span style="color: #003399;">IOException</span> <span style="color: #009900;">&#123;</span>
    doc <span style="color: #339933;">+=</span> base<span style="color: #339933;">;</span>
    <span style="color: #666666; font-style: italic;">// optimistically collect the first docs in an array</span>
    <span style="color: #666666; font-style: italic;">// in case the total number will be small enough to represent</span>
    <span style="color: #666666; font-style: italic;">// as a small set like SortedIntDocSet instead...</span>
    <span style="color: #666666; font-style: italic;">// Storing in this array will be quicker to convert</span>
    <span style="color: #666666; font-style: italic;">// than scanning through a potentially huge bit vector.</span>
    <span style="color: #666666; font-style: italic;">// FUTURE: when search methods all start returning docs in order, maybe</span>
    <span style="color: #666666; font-style: italic;">// we could have a ListDocSet() and use the collected array directly.</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>pos <span style="color: #339933;">&amp;</span>lt<span style="color: #339933;">;</span> scratch.<span style="color: #006633;">length</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      scratch<span style="color: #009900;">&#91;</span>pos<span style="color: #009900;">&#93;</span><span style="color: #339933;">=</span>doc<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">else</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #666666; font-style: italic;">// this conditional could be removed if BitSet was preallocated, but that</span>
      <span style="color: #666666; font-style: italic;">// would take up more memory, and add more GC time...</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>bits<span style="color: #339933;">==</span><span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> bits <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> OpenBitSet<span style="color: #009900;">&#40;</span>maxDoc<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      bits.<span style="color: #006633;">fastSet</span><span style="color: #009900;">&#40;</span>doc<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    pos<span style="color: #339933;">++;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> DocSet getDocSet<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>pos<span style="color: #339933;">&amp;</span>lt<span style="color: #339933;">;=</span>scratch.<span style="color: #006633;">length</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #666666; font-style: italic;">// assumes docs were collected in sorted order!</span>
      <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000000; font-weight: bold;">new</span> SortedIntDocSet<span style="color: #009900;">&#40;</span>scratch, pos<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">else</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #666666; font-style: italic;">// set the bits for ids that were collected in the array</span>
      <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span> i</pre></div></div>

<p>可以看到，DocSetCollector使用两个变量来保存collect的doc id，当collect的doc数小于smallSetSize时使用数组scratch，否则使用OpenBitSet bits。getDocSetNC中构造DocSetCollector的代码是：DocSetCollector collector = new DocSetCollector(maxDoc()&gt;&gt;6, maxDoc());。问题就出在OpenBitSet，以thread用例来说，满足fq=fid:1的doc数有200万之多（总doc数有400万）,而fq=atm:[int_time1 TO int_time2]有时也有百万之多，这使得DocSetCollector在collect时使用OpenBitSet.fastSet(int index)置位。尽管OpenBitSet要比BitSet高效，但我在本机测试发现，OpenBitSet200万次的fastSet需要50ms，所以不难理解，匹配fq的文档数越多，DocSetCollector就越慢，进而SolrIndexSearcher就越慢。不过，另一方面，当多个fq的DocSet做合并后，实际有效的DocSet大小可能很小，而再和text query做合并后，得到的DocSet就会更小。所以，当索引的文档多时，solr的这种处理效率上就低得多。</p>
<p>我粗略浏览了lucene的相关代码，lucence在通过各种Scorer操作匹配Query的结果时没有使用OpenBitSet，而是主要使用队列、堆等集合来操作匹配结果的收集、合并等操作（各种Scorer的具体实现没有细看，有时间再看吧），而Collector.collect的会是最后真正匹配的结果。从实际测试的效果来看，性能要比solr提升数倍。最后，我也就放弃了search(QueryResult qr, QueryCommand cmd)，而是使用lucene的，虽然不能使用上solr的三种cache，但性能还是令人满意的。</p>
<p><font color="red">updated：该文分析问题的角度上出现了偏差，<a href="http://www.kafka0102.com/2010/10/374.html">solr filter query的误用</a>做了一些补充和修正。</font></p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/10/366.html/feed</wfw:commentRss>
		<slash:comments>8</slash:comments>
		</item>
		<item>
		<title>分享：Scalable System Design Patterns</title>
		<link>http://www.kafka0102.com/2010/10/350.html</link>
		<comments>http://www.kafka0102.com/2010/10/350.html#comments</comments>
		<pubDate>Sun, 17 Oct 2010 13:11:27 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[architecture]]></category>
		<category><![CDATA[架构]]></category>
		<category><![CDATA[模式]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=350</guid>
		<description><![CDATA[Scalable System Design Patterns 一文概括了几种常见的系统设计模式。配图很漂亮，我就索性摘过来，推荐感兴趣的继续围观其博客。]]></description>
			<content:encoded><![CDATA[<p><a href="http://horicky.blogspot.com/2010/10/scalable-system-design-patterns.html" target="_blank"> Scalable System Design Patterns </a>一文概括了几种常见的系统设计模式。配图很漂亮，我就索性摘过来，推荐感兴趣的继续围观其博客。</p>
<h2>1、Load Balancer</h2>
<p>该模式中，一个分发器基于某种策略确定由哪个worker实例处理请求。应用最好是无状态的，以使任何一个worker实例都能同等处理请求。大量的网站都会用到负载均衡器这个模式的。</p>
<p style="text-align: left;"><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/p1.png"><img class="aligncenter size-full wp-image-351" title="p1" src="http://www.kafka0102.com/wp-content/uploads/2010/10/p1.png" alt="" width="611" height="409" /></a></p>
<h2>2、Scatter and Gather</h2>
<p>该模式中，分发器将请求转发给多个worker实例，每个worker实例处理完返回给分发器，分发器将worker们返回的结果再加工后再返回给客户端。以搜索为例，通常得AS、BS架构就是这种典型模式。</p>
<p style="text-align: left;"><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/P2.png"><img class="alignleft size-full wp-image-353" title="P2" src="http://www.kafka0102.com/wp-content/uploads/2010/10/P2.png" alt="" width="619" height="358" /></a></p>
<h2>3、Result Cache</h2>
<p>承接上个模式，这个模式只是在分发器处理时加了一步查询结果缓存，这都能算是模式！</p>
<p style="text-align: left;"><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/P3.png"><img class="alignleft size-full wp-image-354" title="P3" src="http://www.kafka0102.com/wp-content/uploads/2010/10/P3.png" alt="" width="619" height="358" /></a></p>
<h2>4、Shared Space</h2>
<p>这个模式还有个更广泛的名字&#8211;“黑板模式”。实现来说，就是在处理流程中，存在一个全局传递的对象，它可能包含了请求参数、中间状态、响应结果等各种信息，供流程中的各个组件对其进行操作。在一些web框架和应用框架中，都可见这个模式的使用。</p>
<p style="text-align: left;"><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/P4.png"><img class="alignleft size-full wp-image-355" title="P4" src="http://www.kafka0102.com/wp-content/uploads/2010/10/P4.png" alt="" width="611" height="409"/></a></p>
<h2>5、Pipe and Filter</h2>
<p>这个模式又以“面向数据流编程”知名，是很通用的企业集成模式。</p>
<p style="text-align: left;"><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/P5.png"><img class="alignleft size-full wp-image-356" title="P5" src="http://www.kafka0102.com/wp-content/uploads/2010/10/P5.png" alt="" width="611" height="409"/></a>
</p>
<h2>6、Map Reduce</h2>
<p>因为google和hadoop，这个模式几乎都了解些，尽管多数人都没亲身应用过。</p>
<p style="text-align: left;"><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/P7.png"><img class="alignleft size-full wp-image-357" title="P7" src="http://www.kafka0102.com/wp-content/uploads/2010/10/P7.png" alt="" width="611" height="409"/></a>
</p>
<h2>7、Bulk Synchronous Parellel</h2>
<p>该模型基于一个master协调，所有的worker同步（lock-step）执行。<br />
该模式被用于Google Pregel Graph Processing  <a href="http://horicky.blogspot.com/2010/07/google-pregel-graph-processing.html">google-pregel-graph-processing</a> 和<a href="http://incubator.apache.org/hama/">Hama</a>。</p>
<p style="text-align: left;"><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/P8.png"><img class="alignleft size-full wp-image-358" title="P8" src="http://www.kafka0102.com/wp-content/uploads/2010/10/P8.png" alt="" width="611" height="409"/></a></p>
<h2>8、Execution Orchestrator</h2>
<p>又一个不很了解的模式，似乎是一个和map reduce有一拼的分布式计算模型，似乎是微软的创造：<a href="http://research.microsoft.com/en-us/projects/dryad/ " target="_blank">Microsoft&#8217;s Dryad project</a>。</p>
<p style="text-align: left;"><a href="http://www.kafka0102.com/wp-content/uploads/2010/10/P82.png"><img class="alignleft size-full wp-image-359" title="P82" src="http://www.kafka0102.com/wp-content/uploads/2010/10/P82.png" alt="" width="611" height="409" /></a></p>
<p><br/></p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/10/350.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>twitter的新搜索架构</title>
		<link>http://www.kafka0102.com/2010/10/347.html</link>
		<comments>http://www.kafka0102.com/2010/10/347.html#comments</comments>
		<pubDate>Sat, 09 Oct 2010 19:42:46 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[lucene]]></category>
		<category><![CDATA[search]]></category>
		<category><![CDATA[twitter]]></category>
		<category><![CDATA[zoie]]></category>
		<category><![CDATA[实时检索]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=347</guid>
		<description><![CDATA[当习惯于粗略浏览google reader中的未读条目后便似完成任务般的“全部标记已读”，我就经常性的错过不错的精彩条目，比如这篇 Twitter's New Search Architecture。我是订阅了engineering.twitter.com的，但这并不影响我没有注意到它就把它mark过。今天看到这篇文章时，已经有别的网站将该文翻译成中文了，比如新 Twitter，新搜索，比如 新 Twitter，新搜索，我在这也就算是炒冷饭了。]]></description>
			<content:encoded><![CDATA[<p>当习惯于粗略浏览google reader中的未读条目后便似完成任务般的“全部标记已读”，我就经常性的错过不错的精彩条目，比如这篇 <a href="http://engineering.twitter.com/2010/10/twitters-new-search-architecture.html" target="_blank">Twitter&#8217;s New Search Architecture</a>。我是订阅了engineering.twitter.com的，但这并不影响我没有注意到它就把它mark过。今天看到这篇文章时，已经有别的网站将该文翻译成中文了，比如<a href="http://news.cnblogs.com/n/76524/ " target="_blank">新 Twitter，新搜索</a>，比如 <a href="http://www.techcrunchchina.com/5144" target="_blank">新 Twitter，新搜索</a>，我在这也就算是炒冷饭了。</p>
<p>twitter之前的搜索使用的是被它收购的Summize公司的技术，是基于mysql的搜索，具体技术细节不祥。<a href="http://blog.twitter.com/2008/07/finding-perfect-match.html" target="_blank"> Finding A Perfect Match</a>有提到，Summize也是个初创公司，做的就是twitter搜索，而twitter看中了Summize的技术和团队，便收编它为正规军。不过，随着twitter数据量的激增及对实时搜索的高要求（每秒1000条推的索引更新和12000次查询，一天下来有10亿次查询），原有的技术架构越来越不能满足需求，所以twitter转向了lucene。这除了lucene拥有诸多的优点，也是因为twitter自家有人是lucene的commiter。</p>
<p>当然，lucene也有缺点，文中有提到twitter对lucene的一些改进点如下，这些点已经或将被提到lucene代码中：<br />
（1）改进GC性能。<br />
（2）无锁的数据结构和算法<br />
（3）逆序遍历posting列表（是指希望默认的查询排序方式是按照添加的索引顺序，而不是需要显示指定排序字段，以提高速度？）<br />
（4）有效的早期query终止（是指对查询做了超时限制，避免查询因为读磁盘等原因被hang住？）</p>
<p>就像之前的linkedin开源过一个基于lucene的实时搜索项目<a href="http://code.google.com/p/zoie/" target="_blank">zoie</a> （<a href="http://www.kafka0102.com/2010/05/119.html" target="_blank">用Zoie构建实时检索系统</a> <a href="http://www.kafka0102.com/2010/05/133.html" target="_blank">实时检索系统Zoie实现分析</a>），对于social应用，现在越来越追求搜索的实时性。lucene在2.9版本引入了NRT（Near Real-Time search），算是对实时搜索需求的初始回应，但NRT不是RT，根据应用特点不同，一般能做到秒级别到分钟级别的准实时搜索。lucene的RT搜索目前还在branch中缓慢的孕育，twitter为其贡献着代码并充当了很好的实战的先头兵，这个很赞的说。</p>
<p>期待着lucene和solr的RT搜索能早些出来，这样我可能会考虑迁移现在的站内搜索架构。对于查询量不大更新量更不大的站内搜索来说，将建索引和实时查询做到一个server中，而取代master-slave结构，既简化了架构，也提供了更好的查询效果。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/10/347.html/feed</wfw:commentRss>
		<slash:comments>4</slash:comments>
		</item>
		<item>
		<title>Is Terracotta&#8217;s BigMemory a big thing?</title>
		<link>http://www.kafka0102.com/2010/09/343.html</link>
		<comments>http://www.kafka0102.com/2010/09/343.html#comments</comments>
		<pubDate>Thu, 30 Sep 2010 09:40:11 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[java]]></category>
		<category><![CDATA[BigMemory]]></category>
		<category><![CDATA[ehcache]]></category>
		<category><![CDATA[jvm]]></category>
		<category><![CDATA[terracotta]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=343</guid>
		<description><![CDATA[Terracotta最近推出了个新东西--BigMemory，使得Infoq上的这篇Terracotta's BigMemory Aiming to Eliminate Garbage Collection for Java Caches文章引来长长的争论。像GigaSpaces、Infinispan等和Terracotta有竞争关系的商业或开源的对手，对BigMemory的诸多方面提出了疑问和质疑。如果BigMemory真的如Terracotta吹的那般好，那些竞争对手该情何以堪？就算它真的好，也要挑出它的不是，比如就有评论说现在都分布式年代了，还搞什么单JVM实例优化？暂且放下这些争论，看看BigMemory是个怎样的东西，值得大家如此大动神经。]]></description>
			<content:encoded><![CDATA[<p>Terracotta最近推出了个新东西&#8211;BigMemory，使得Infoq上的这篇<a href="http://www.infoq.com/news/2010/09/bigmemory " target="_blank">Terracotta&#8217;s BigMemory Aiming to Eliminate Garbage Collection for Java Caches</a>文章引来长长的争论。像GigaSpaces、Infinispan等和Terracotta有竞争关系的商业或开源的对手，对BigMemory的诸多方面提出了疑问和质疑。如果BigMemory真的如Terracotta吹的那般好，那些竞争对手该情何以堪？就算它真的好，也要挑出它的不是，比如就有评论说现在都分布式年代了，还搞什么单JVM实例优化？暂且放下这些争论，看看BigMemory是个怎样的东西，值得大家如此大动神经。</p>
<p>JVM GC的问题可以说是“臭名昭著”，尽管现在一台服务器的内存可以动辄几十G到上百G，但单个JVM实例并不能充分利用硬件提供的内存资源，因为JVM内存容量越大，由GC带来的应用“暂停”现象就越明显。所以，通常单个JVM实例的内存容量要控制在一定范围内。这也使得，为了能有效利用机器的内存，一台机器会部署多个JVM实例。而对于需要大规模内存的Java应用，比如Cache系统，想不分布式都不行。</p>
<p>但Terracotta说了，他们的大多数客户并不需要分布式的Cache系统，单个Ehcache实例就够用了，只是客户们希望单个Ehcache实例能支持更大的内存容量，并且不要有性能问题。所以，BigMemory被创造出来了。BigMemory不是在JVM堆上分配内存的（off heap），内存的分配与回收不受GC控制，容量方面可以达到物理内存规模。因为BigMemory不是开源的，Terracotta没有透露出BigMemory的实现原理，但其<a href="http://ehcache.org/documentation/offheap_store.html" target="_blank">使用文档</a>指出，使用BigMemory需要配置MaxDirectMemorySize选项，并且其是100%Java开发的，所以BigMemory应该是基于DirectBuffer的。至于BigMemory的技术难度和复杂性，我还不好揣测，是不是通过DirectBuffer分配了一块大的ByteArray，然后有一套有效的堆外内存管理机制呢？</p>
<p>说说BigMemory的效果吧。以Ehcache为例，它原来支持两层store，即MemStore和DiskStore，MemStore的大小受限于GC影响，一般控制在（4G？,我有些记不清了，它的文档应该有说明的）以下为好；而DiskStore相比于MemStore，性能方面有百倍的差距，更适合做个备份。现在引入BigMemory后，Ehcache的Stroe就有三层：MemStore可以控制在较好的规模（比如2G），使得GC不会对应用服务产生严重的影响，并且能cache住大多数热的内容；BigMemory可以给出更大的容量，cache那些不是很热的内容，因为BigMemory保存的是对象序列化后的内容，所以对BigMemory管理的对象操作有个序列化和反序列化过程，使得BigMemory相比MemStore会有10倍左右的性能损失，但相比于分布式的cache读取，这些性能损失是已经很小了；而DiskStore，它就真的适合做个备份，以提升应用重启动时间。</p>
<p><a href="http://www.kafka0102.com/wp-content/uploads/2010/09/tiered_storage.png"><img class="aligncenter size-full wp-image-344" title="tiered_storage" src="http://www.kafka0102.com/wp-content/uploads/2010/09/tiered_storage.png" alt="" width="493" height="243" /></a></p>
<p>如果Terracotta给出的数据和评测准确，我觉得BigMemory是个很不错的东西。即便是应用的分布式化盛行，但提升单个JVM的内存使用效果对很多场景还是很有价值的，君不见还有大公司热衷于mysql的单机优化吗？不知道是否会有有心人做出个通用的开源的BigMemory出来，甚至集成到JDK中，以造福Javaer们呢？</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/09/343.html/feed</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>Facebook的Online Schema Change for MySQL</title>
		<link>http://www.kafka0102.com/2010/09/338.html</link>
		<comments>http://www.kafka0102.com/2010/09/338.html#comments</comments>
		<pubDate>Wed, 29 Sep 2010 08:43:16 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[mysql]]></category>
		<category><![CDATA[Facebook]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=338</guid>
		<description><![CDATA[Feng的 Facebook 针对 MySQL 开源 Online Schema Change 代码 介绍了 Facebook最新开源的在线修改mysql schema的工具，并对该工具极尽赞扬之辞。于是乎，我这个有大半年没动过mysql的人翻看起该工具的文档Online Schema Change for MySQL，并简单的浏览了其代码OnlineSchemaChange.php。总的来说，尽管其实现思想很朴素，这个工具真的很实用，它借鉴的openark-kit似乎发布了很长时间，只是名头没有Facebook的guys响亮，知道的人似乎不多。openark上的其他工具也很不错，比如ycheckpoint，就是个很不错的监控mysql运行状态的工具。]]></description>
			<content:encoded><![CDATA[<p>Feng的 <a href="http://www.dbanotes.net/opensource/facebook_mysql_online_schema_change.html" target="_blank">Facebook 针对 MySQL 开源 Online Schema Change 代码 </a>介绍了Facebook最新开源的在线修改mysql schema的工具，并对该工具极尽赞扬之辞。于是乎，我这个有大半年没动过mysql的人翻看起该工具的文档<a href="http://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932" target="_blank">Online Schema Change for MySQL</a>，并简单的浏览了其代码<a href="http://bazaar.launchpad.net/~mysqlatfacebook/mysqlatfacebook/tools/annotate/head:/osc/OnlineSchemaChange.php " target="_blank">OnlineSchemaChange.php</a>。总的来说，尽管其实现思想很朴素，这个工具真的很实用，它借鉴的<a href="http://code.openark.org/forge/openark-kit" target="_blank">openark-kit</a>似乎发布了很长时间，只是名头没有Facebook的guys响亮，知道的人似乎不多。openark上的其他工具也很不错，比如<a href="http://code.openark.org/forge/mycheckpoint/" target="_blank">ycheckpoint</a>，就是个很不错的监控mysql运行状态的工具。</p>
<p>对于修改mysql schema的操作，我之前的经验是：在主从模式下，先对从库逐台做修改操作，操作期间该从库不用于提供服务（应用端排除对该从库的读操作，如果脏读在一定程度上可以接受，该从库也可以对外提供服务）;所有从库做完后，将主库切到一个从库，对主库进行修改，修改完成后再切换回去。这个过程看起来有些繁杂，但只要有清晰的操作步骤和熟练操作工种，对于大多数一般规模的主从来说，这个过程是可以接受的，毕竟不是经常修改schema。不过，对于像Facebook动辄上千台数据库的情况，这个过程就会耗时很长很折腾，而人家专业的DBA人数又比不上国内的大公司们，所以很有必要搞出个更高效的解决方案。</p>
<p>对于修改schema，mysql的执行过程主要就是建个copy表，把原表数据拷过去，然后将copy表重命名为原表（对一些特定index的修改操作，InnoDB支持<a href="http://dev.mysql.com/doc/innodb-plugin/1.0/en/innodb-create-index.html" target="_blank">Fast Index Creation</a>），而整个过程是需要加互斥锁的，数据量越大，这个数据拷贝过程就越久，这对在线应用往往是不可接受的。Facebook的改进思路是，在将原表数据拷贝到copy表过程中，通过触发器把外部同时进行的写提交记录到deltas表中，在数据拷贝完成后，再回放deltas表中的数据到copy表，保证copy表不会丢失操作期间的写提交，最后再rename copy表为原表。而在实现细节上，其实要考虑的问题还是很多的，而原文不但阐述了他们怎么做的还阐述了他们为什么这样做，其思考过程本身就很值得学习。</p>
<p>翻看OnlineSchemaChange.php，可以在代码的后面清晰的看到该脚本主体的执行过程，下面做些概述：</p>

<div class="wp_syntax"><div class="code"><pre class="php" style="font-family:monospace;"><span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">init</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">createCopyTable</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">alterCopyTable</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">createDeltasTable</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">createTriggers</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">startSnapshotXact</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">selectTableIntoOutfile</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">dropNCIndexes</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">loadCopyTable</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">replayChanges</span><span style="color: #009900;">&#40;</span><span style="color: #009900; font-weight: bold;">false</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span> <span style="color: #666666; font-style: italic;">// false means not in single xact</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">recreateNCIndexes</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">replayChanges</span><span style="color: #009900;">&#40;</span><span style="color: #009900; font-weight: bold;">false</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span> <span style="color: #666666; font-style: italic;">// false means not in single xact</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">swapTables</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$this</span><span style="color: #339933;">-&gt;</span><span style="color: #004000;">cleanup</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>

<p>1、init：初始化阶段会做些验证操作，也是该脚本需要的约束，比如检查mysql版本（目前只支持5.0.84和5.1.47，应该是facebook只用到这两个版本）、修改的表不能有触发器和外键等。在初始化之前（构造函数中），通过sql_log_bin = 0会将关闭该seesion的sql复制，所以在主库执行该脚本不会影响从库。<br />
2、createCopyTable：直接创建一个和原表一样结构的copy表。<br />
3、alterCopyTable：对copy表执行本次的目标修改操作”alter table&#8230;“。<br />
4、createDeltasTable：创建deltas表，该表保存修改schema期间外部对原表提交的写操作，所以该表除了包含原表所有列，还额外有两列，一列是自增主键，一列是提交的写操作类型（insert、update、delete），以便后面回放。<br />
5、createTriggers：对原表创建insert、update、delete的触发器，触发器的内容就是将产生的更新写到deltas表中。<br />
6、startSnapshotXact：开启事务，同时创建并初始化临时表TEMP_TABLE_IDS_TO_EXCLUDE，这个表保存从deltas表得到的在回放期间需要放弃的记录id（避免重复回放），比如在上面createTriggers后，deltas表可能就会有记录了，初始化时就要将它们从deltas表的主键select到TEMP_TABLE_IDS_TO_EXCLUDE。<br />
7、selectTableIntoOutfile：考虑到减轻mysql的负载，将原表数据拷贝到copy表不是直接进行的，它先是分片（chunked）地将数据输出到多个外部文件（outfile），然后将外部文件的数据导入到copy表中，分片的大小可以指定（默认是500000）。该步骤就是分片导出原表数据到外部文件中。<br />
8、dropNCIndexes：在导入数据到copy表前，先把copy表的非聚簇索引去掉，以加快导入。<br />
9、loadCopyTable：将上面得到的文件数据导入到copy表。<br />
10、replayChanges：这里又用到另一个临时表TEMP_TABLE_IDS_TO_INCLUDE，该表从deltas表得到有效的记录id（排除掉TEMP_TABLE_IDS_TO_EXCLUDE的），用于对copy表回放deltas表的有效记录，回放完成后会将TEMP_TABLE_IDS_TO_INCLUDE的id们扔给TEMP_TABLE_IDS_TO_EXCLUDE。<br />
11、recreateNCIndexes：重建非聚簇索引。<br />
12、replayChanges：再次回放。<br />
13、swapTables：将原表和copy表交换名称，整个过程要锁表并停止对从库的复制，会有短暂的停提交。<br />
14、cleanup：清理现场。</p>
<p>上面只是简单的罗列其执行过程，OnlineSchemaChange.php还要处理不同版本mysql的区别，并需要对各种异常做处理。如果该功能能集成到mysql（innodb），就不会再费nosql们指责mysql修改schema麻烦的口舌了。而现在，至少Facebook提供了一个不错的工具来解决这个问题。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/09/338.html/feed</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>分享：Playfish&#8217;s Social Gaming Architecture</title>
		<link>http://www.kafka0102.com/2010/09/333.html</link>
		<comments>http://www.kafka0102.com/2010/09/333.html#comments</comments>
		<pubDate>Wed, 22 Sep 2010 15:21:09 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[architecture]]></category>
		<category><![CDATA[架构]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=333</guid>
		<description><![CDATA[highscalability.com上的Playfish's Social Gaming Architecture - 50 Million Monthly Users and Growing是篇很不错的大杂烩，内容涉及全球第二大social game公司Playfish的技术架构、产品设计、团队协作、产品运营等诸多经验之谈，尤其对初创团队很有借鉴意义。我这里还只是对文中提到的一些技术经验做些汇总，也推荐大家去浏览原文，重点看看自己感兴趣的段落。但说起架构的事情，其实看多了，往往都是大方向上相似，细节上有自己的考量和特点，以形成自己的架构风格。]]></description>
			<content:encoded><![CDATA[<p>highscalability.com上的<a href="http://highscalability.com/blog/2010/9/21/playfishs-social-gaming-architecture-50-million-monthly-user.html" target="_blank">Playfish&#8217;s Social Gaming Architecture &#8211; 50 Million Monthly Users and Growing</a>是篇很不错的大杂烩，内容涉及全球第二大social game公司Playfish的技术架构、产品设计、团队协作、产品运营等诸多经验之谈，尤其对初创团队很有借鉴意义。我这里还只是对文中提到的一些技术经验做些汇总，也推荐大家去浏览原文，重点看看自己感兴趣的段落。但说起架构的事情，其实看多了，往往都是大方向上相似，细节上有自己的考量和特点，以形成自己的架构风格。</p>
<h2>SOA</h2>
<p>SOA在企业应用中热火朝天，但在互联网应用中SOA被提及的很少。Playfish的SOA并不是使用SOA工业中的技术栈，其思想就是组件化设计。当系统变得复杂以后，松耦合的组件化是必然的趋势，系统内组件间通过约定的接口交互。对于复杂系统来说，组件间的低依赖性是很重要的，当一个组件不能正常提供服务，系统应该是优雅的降级，而不是雪崩般的集体挂掉。说起来简单，但相信很多平时跑的很健康的系统，往往只是在一个组件坏掉后，才发现系统原来并不是看起来的那样可靠。</p>
<h2>The Cloud</h2>
<p>从一开始，Playfish就使用Amazon的基础云服务，国外的很多初创公司都是这样做的。关于使用基础云服务的诸多优点，其实很多文章都有提到，对初创产品来说，对硬件资源的需求变动性很大，像social game，高峰期和平时对硬件的需求可能相差很大，如果自己搭建硬件资源，会有很大的资源浪费，而使用具有可扩展性的云服务无疑更会经济。更重要的是，不用操心于硬件资源管理，团队会更加专注于核心的服务。不过，国内使用基础云服务的似乎不多，《程序员》杂志以前倒是有过一篇介绍国内social game公司使用基础云的文章，但国人还是喜欢什么东西都自己掌控的好，所以能自己折腾就绝不麻烦别人。当然，Playfish并没有使用Amazon的所有云服务，比如Amazon的ELB（Elastic Load Balancing）和RDB（Rational Database），因为将他们的应用迁移过来无疑会很麻烦甚至危险。</p>
<h2>Java</h2>
<p>Playfish主要是基于Java平台的，也是得益于Java的诸多优点，比如甚是丰富的开源库，既可以用于开发Web程序也可以用于开发服务器端程序。当然，Playfish也需要花些精力对JVM做性能调优。而在数据处理分析方面，Hadoop那一套东西是必不可少的。文中提到，Playfish使用Jetty作为应用Server，窃以为可能是做嵌入式使用，如果是独立server，tomcat和resin应该是更好的选择。</p>
<h2>Database System</h2>
<p>数据库方面，Playfish使用的是mysql，从早期的master-slave发展到现在的shard。social game的特点是写多读少，所以传统的主从模式就不合适了。在shard方式，Playfish应该是基于用户id散的，所以可扩展性很强。在schema方面，Playfish将mysql kv化了，为了加快访问速度，将和用户相关的诸多信息塞到blob类型中，这样DB的写压力被分散到后端应用上，而应用又是能很好的扩展的。所以，Playfish对mysql的使用也是nosql化了，但Playfish没有直接使用现成的nosql产品，而是使用更成熟的mysql搭建适合自己的nosql存储方式。在读多写少应用中，像memcached这样的缓存系统是必不可少的，而对写多读少的场景，Playfish有效的利用了flash客户端，使得客户端就像个memcached，写操作时即时更新客户端而不需要从服务器端再次读，而服务器端可以使用消息队列来异步写。</p>
<h2>YAMI4 &#8211; Messaging</h2>
<p>YAMI4是个什么东西？YAMI4是<a href="http://www.inspirel.com/yami4/" target="_blank">http://www.inspirel.com/yami4/</a>，一个很轻量级的消息框架，google的结果很少，用的人似乎也不多。Playfish说是在做了很多的评估后采用了YAMI4。我粗略的浏览了YAMI4的文档，发现这个东西还是很有特点的。YAMI4支持如C++、Java、Python等多种语言，它不是一个独立的消息系统，而是一个嵌入式的消息框架（库）。消息传递有两种模式：1）是点对点的同步模式，可以使用如Thrift等RPC框架进行消息传递，但不适用于Playfish。2）是基于消息的异步模式，通常是使用如ActiveMQ等独立的消息系统（broker），但它会成为单点（当然可以花成本来去单点），并且消息的传递还要经过broker这一跳。而YAMI4是个可以嵌入producer的broker，这样消息可以点对点的传递给consumer，没有了broker的单点的困扰，并且调用可以是异步的。而对消息的数据格式，YAMI4像IDL一样提供了较友好的支持。</p>
<h2>总结</h2>
<p>写的匆忙，止笔于此，回头把YAMI4研究下。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/09/333.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>mongodb MapReduce使用初步</title>
		<link>http://www.kafka0102.com/2010/09/329.html</link>
		<comments>http://www.kafka0102.com/2010/09/329.html#comments</comments>
		<pubDate>Sat, 18 Sep 2010 13:15:02 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[nosql]]></category>
		<category><![CDATA[mapreduce]]></category>
		<category><![CDATA[mongodb]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=329</guid>
		<description><![CDATA[	最近在做搜索的查询日志的统计分析，对每一条查询统计日志，我将其解析出来后以特定字段格式存在mongodb中，定时调度做些统计分析。其中有个需求是，统计某个时间段（每天、每周、每月）各个query的查询次数，展示上就是热门查询query了。考虑到处理的数据量不会很大，解决方法也可以简单来之。我现在使用的方法就是mongodb的MapReduce功能，其实这个需求也可以认为是个group操作，而mongodb的group功能就是基于MapReduce的，但group对结果集的大小是有限制的。本文就针对一个示例介绍一下mongodb MapReduce功能。]]></description>
			<content:encoded><![CDATA[<p>	最近在做搜索的查询日志的统计分析，对每一条查询统计日志，我将其解析出来后以特定字段格式存在mongodb中，定时调度做些统计分析。其中有个需求是，统计某个时间段（每天、每周、每月）各个query的查询次数，展示上就是热门查询query了。考虑到处理的数据量不会很大，解决方法也可以简单来之。我现在使用的方法就是mongodb的MapReduce功能，其实这个需求也可以认为是个group操作，而mongodb的group功能就是基于MapReduce的，但group对结果集的大小是有限制的。本文就针对一个示例介绍一下mongodb MapReduce功能。</p>
<h2>语法介绍</h2>
<p>	MapReduce是mongodb中的一个Command，它的语法格式如下：</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">db.<span style="color: #660066;">runCommand</span><span style="color: #009900;">&#40;</span>
 <span style="color: #009900;">&#123;</span> mapreduce <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>collection<span style="color: #339933;">&gt;,</span>
   map <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>mapfunction<span style="color: #339933;">&gt;,</span>
   reduce <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>reducefunction<span style="color: #339933;">&gt;</span>
   <span style="color: #009900;">&#91;</span><span style="color: #339933;">,</span> query <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>query filter object<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#93;</span>
   <span style="color: #009900;">&#91;</span><span style="color: #339933;">,</span> sort <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>sort the query.  <span style="color: #660066;">useful</span> <span style="color: #000066; font-weight: bold;">for</span> optimization<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#93;</span>
   <span style="color: #009900;">&#91;</span><span style="color: #339933;">,</span> limit <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>number of objects to <span style="color: #000066; font-weight: bold;">return</span> from collection<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#93;</span>
   <span style="color: #009900;">&#91;</span><span style="color: #339933;">,</span> out <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>output<span style="color: #339933;">-</span>collection name<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#93;</span>
   <span style="color: #009900;">&#91;</span><span style="color: #339933;">,</span> keeptemp<span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>true<span style="color: #339933;">|</span>false<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#93;</span>
   <span style="color: #009900;">&#91;</span><span style="color: #339933;">,</span> finalize <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>finalizefunction<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#93;</span>
   <span style="color: #009900;">&#91;</span><span style="color: #339933;">,</span> scope <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>object where fields go into javascript global scope <span style="color: #339933;">&gt;</span><span style="color: #009900;">&#93;</span>
   <span style="color: #009900;">&#91;</span><span style="color: #339933;">,</span> verbose <span style="color: #339933;">:</span> <span style="color: #003366; font-weight: bold;">true</span><span style="color: #009900;">&#93;</span>
 <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>

<p>	对于该Command，必有的3个参数我就不解释了。对于可选参数，这里简要说明如下：<br />
	(1) query是很常用的，它用来在map阶段过滤查询条件的以限定MapReduce操作的记录范围。<br />
	(2) 和query相关的还有sort和limit，我起初以为它俩是用在reduce阶段，实际上和query一起用在map阶段。<br />
	(3) mongodb默认是创建一个临时的collection存储MapReduce结果，当客户端连接关闭或者显示使用collection.drop()，这个临时的collection会被删除掉。这也就说，默认的keeptemp是false，如果keeptemp为true，那么结果collection就是永久的。当然，生成的collection名称并不友好，所以可以指定out表明永久存储的collection的名称（这时不需要再指定keeptemp）。当指定out时，并不是将执行结果直接存储到out，而是同样到临时collection，之后如果out存在则drop掉，最后rename临时collection为out。<br />
	(4) finalize：当MapReduce完成时应用到所有结果上，通常不怎么使用。<br />
	(5) verbose：提供执行时间的统计信息。<br />
	执行结果的格式如下：</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #009900;">&#123;</span> result <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>collection_name<span style="color: #339933;">&gt;,</span>
  counts <span style="color: #339933;">:</span> <span style="color: #009900;">&#123;</span>
       input <span style="color: #339933;">:</span>  <span style="color: #339933;">&lt;</span>number of objects scanned<span style="color: #339933;">&gt;,</span>
       emit  <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>number of times emit was called<span style="color: #339933;">&gt;,</span>
       output <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>number of items <span style="color: #000066; font-weight: bold;">in</span> output collection<span style="color: #339933;">&gt;</span>
  <span style="color: #009900;">&#125;</span> <span style="color: #339933;">,</span>
  timeMillis <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>job_time<span style="color: #339933;">&gt;,</span>
  ok <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span><span style="color: #CC0000;">1</span>_if_ok<span style="color: #339933;">&gt;,</span>
  <span style="color: #009900;">&#91;</span><span style="color: #339933;">,</span> err <span style="color: #339933;">:</span> <span style="color: #339933;">&lt;</span>errmsg_if_error<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#93;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>	更常用的MapReduce命令的helper是：</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">	db.<span style="color: #660066;">collection</span>.<span style="color: #660066;">mapReduce</span><span style="color: #009900;">&#40;</span>mapfunction<span style="color: #339933;">,</span>reducefunction<span style="color: #009900;">&#91;</span><span style="color: #339933;">,</span>options<span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>

<p>	map函数的定义如下，map函数内使用this来操作当前行表示的对象，并且需要使用emit(key,value)方法来向reduce提供参数：</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">	<span style="color: #003366; font-weight: bold;">function</span> map<span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">void</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">-&gt;</span> <span style="color: #000066; font-weight: bold;">void</span></pre></div></div>

<p>	reduce函数定义如下，reduce的key就是emit(key,value)的key，value_array是同个key对应的多个value数组：</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">	<span style="color: #003366; font-weight: bold;">function</span> reduce<span style="color: #009900;">&#40;</span>key<span style="color: #339933;">,</span> value_array<span style="color: #009900;">&#41;</span> <span style="color: #339933;">-&gt;</span> value</pre></div></div>

<p>	MapReduce得到的collection的格式是{&#8220;_id&#8221;:key,&#8221;value&#8221;:<reduce return value>}。</p>
<h2>应用示例</h2>
<p>	这里给出一个假想的无意义的示例，主要是为了说明mongodb MapReduce的使用。每条记录的schema是{&#8220;query&#8221;:<query>,&#8221;cnt&#8221;:<no meaning num>,&#8221;year&#8221;:<year>,&#8221;month&#8221;=><month>}。这个schema的cnt是不需要的，因为每条查询query的cnt都是1,但这里想要稍微复杂一些条件。下面是可在mongodb shell中执行的MapReduce脚本。</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">map <span style="color: #339933;">=</span> <span style="color: #003366; font-weight: bold;">function</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span> 
	emit<span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">this</span>.<span style="color: #660066;">query</span><span style="color: #339933;">,</span> <span style="color: #000066; font-weight: bold;">this</span>.<span style="color: #660066;">cnt</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
reduce <span style="color: #339933;">=</span> <span style="color: #003366; font-weight: bold;">function</span><span style="color: #009900;">&#40;</span>key <span style="color: #339933;">,</span> vals<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
	<span style="color: #003366; font-weight: bold;">var</span> sum <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">;</span>
	<span style="color: #000066; font-weight: bold;">for</span><span style="color: #009900;">&#40;</span><span style="color: #003366; font-weight: bold;">var</span> i <span style="color: #000066; font-weight: bold;">in</span> vals<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
		sum <span style="color: #339933;">+=</span> vals<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
	<span style="color: #009900;">&#125;</span>
	<span style="color: #000066; font-weight: bold;">return</span> sum<span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
res <span style="color: #339933;">=</span> db.<span style="color: #660066;">log_info</span>.<span style="color: #660066;">mapReduce</span><span style="color: #009900;">&#40;</span>map<span style="color: #339933;">,</span>reduce<span style="color: #339933;">,</span><span style="color: #009900;">&#123;</span><span style="color: #3366CC;">&quot;query&quot;</span><span style="color: #339933;">:</span><span style="color: #009900;">&#123;</span><span style="color: #3366CC;">&quot;year&quot;</span><span style="color: #339933;">:</span><span style="color: #CC0000;">2010</span><span style="color: #009900;">&#125;</span><span style="color: #009900;">&#125;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>

<p>执行结果如下：</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #009900;">&#123;</span>
	<span style="color: #3366CC;">&quot;result&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;tmp.mr.mapreduce_1284794393_2&quot;</span><span style="color: #339933;">,</span>
	<span style="color: #3366CC;">&quot;timeMillis&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">72</span><span style="color: #339933;">,</span>
	<span style="color: #3366CC;">&quot;counts&quot;</span> <span style="color: #339933;">:</span> <span style="color: #009900;">&#123;</span>
		<span style="color: #3366CC;">&quot;input&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">1000</span><span style="color: #339933;">,</span>
		<span style="color: #3366CC;">&quot;emit&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">1000</span><span style="color: #339933;">,</span>
		<span style="color: #3366CC;">&quot;output&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">113</span>
	<span style="color: #009900;">&#125;</span><span style="color: #339933;">,</span>
	<span style="color: #3366CC;">&quot;ok&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">1</span><span style="color: #339933;">,</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>	对于&#8221;result&#8221;,它是生成的临时collection名称，这个名称的命名规则是：&#8221;tmp.mr.mapreduce_&#8221;+time(0)+&#8221;_&#8221;+(jobNumber++)<br />
执行db[res.result].find()得到：</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;a&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">521</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aa&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">128</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aaa&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">40</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aaaa&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">4</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aaab&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">9</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aaac&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">13</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aab&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">45</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aaba&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">5</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aabb&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">14</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aabc&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">20</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aac&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">39</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aaca&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">6</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aacb&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">2</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aacc&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">5</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;ab&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">65</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;aba&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">37</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;abaa&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">12</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;abab&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">13</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;abac&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">10</span> <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#123;</span> <span style="color: #3366CC;">&quot;_id&quot;</span> <span style="color: #339933;">:</span> <span style="color: #3366CC;">&quot;abb&quot;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;value&quot;</span> <span style="color: #339933;">:</span> <span style="color: #CC0000;">42</span> <span style="color: #009900;">&#125;</span></pre></div></div>

<h2>Java客户端API使用</h2>
<p>	和JS脚本一样，mongodb Java客户端提供了两个MapReduce接口，分别是：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">public</span> MapReduceOutput mapReduce<span style="color: #009900;">&#40;</span> <span style="color: #003399;">String</span> map , <span style="color: #003399;">String</span> reduce , <span style="color: #003399;">String</span> outputCollection , DBObject query <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000000; font-weight: bold;">public</span> MapReduceOutput mapReduce<span style="color: #009900;">&#40;</span> DBObject command <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>

<p>	MapReduceOutput实现如下：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> MapReduceOutput <span style="color: #009900;">&#123;</span>
&nbsp;
    MapReduceOutput<span style="color: #009900;">&#40;</span> DBCollection from , BasicDBObject raw <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        _collname <span style="color: #339933;">=</span> raw.<span style="color: #006633;">getString</span><span style="color: #009900;">&#40;</span> <span style="color: #0000ff;">&quot;result&quot;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        _coll <span style="color: #339933;">=</span> from._db.<span style="color: #006633;">getCollection</span><span style="color: #009900;">&#40;</span> _collname <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        _counts <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span>BasicDBObject<span style="color: #009900;">&#41;</span>raw.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span> <span style="color: #0000ff;">&quot;counts&quot;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> DBCursor results<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> _coll.<span style="color: #006633;">find</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> drop<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        _coll.<span style="color: #006633;">drop</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> DBCollection getOutputCollection<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> _coll<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">String</span> _collname<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> DBCollection _coll<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> BasicDBObject _counts<span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>	所以，可以调用MapReduceOutput.results()得到DBCursor做后续处理，比如在我的应用场景里，根据value值做降序排序并取limit 1000以得到最热门的一些query。<br />
	由于JavaScript引擎设计上的限制，当前的mongodb MapReduce还只是单线程执行，mongodb也在计划解决这个问题。如果需要多线程处理，可以考虑shard或者在客户端代码控制处理。</p>
<h2>参考文章</h2>
<p>http://www.mongodb.org/display/DOCS/MapReduce</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/09/329.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>[Solr实践]自定义SolrEventListener实现searcher的autowarm策略</title>
		<link>http://www.kafka0102.com/2010/09/326.html</link>
		<comments>http://www.kafka0102.com/2010/09/326.html#comments</comments>
		<pubDate>Sun, 05 Sep 2010 11:50:40 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[solr]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=326</guid>
		<description><![CDATA[	Solr的searcher autowarm（预热）有两个时机，一个是系统启动时（firstSearcher），一个是使用新的searcher替换旧的searcher时（newSearcher）。Solr支持在solrconfig.xml中对SolrCore配置SolrEventListener来实现自定义的autowarm。通常来说，Solr提供的默认实现QuerySenderListener就够用了。在我的需求中，希望solrconfig.xml中配置的SolrEventListener是针对多个SolrCore的，这要是因为我的多个SolrCore共用了一个solrconfig.xml配置。就配置autowarm的查询query来说，简单的就是配置一个常见的query，但如果系统有排序查询（sort），可以配置适宜的sort条件以预热lucene的fieldCache。下面是我自定义的SolrEventListener，效果是，如果SolrCore没有配置query，就使用default的，否则使用自己的。]]></description>
			<content:encoded><![CDATA[<p>	Solr的searcher autowarm（预热）有两个时机，一个是系统启动时（firstSearcher），一个是使用新的searcher替换旧的searcher时（newSearcher）。Solr支持在solrconfig.xml中对SolrCore配置SolrEventListener来实现自定义的autowarm。通常来说，Solr提供的默认实现QuerySenderListener就够用了。在我的需求中，希望solrconfig.xml中配置的SolrEventListener是针对多个SolrCore的，这要是因为我的多个SolrCore共用了一个solrconfig.xml配置。就配置autowarm的查询query来说，简单的就是配置一个常见的query，但如果系统有排序查询（sort），可以配置适宜的sort条件以预热lucene的fieldCache。下面是我自定义的SolrEventListener，效果是，如果SolrCore没有配置query，就使用default的，否则使用自己的。<br />
	实现代码修改自Solr的QuerySenderListener，代码如下：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> TubaQuerySenderListener <span style="color: #000000; font-weight: bold;">implements</span> SolrEventListener <span style="color: #009900;">&#123;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000000; font-weight: bold;">final</span> Logger logger <span style="color: #339933;">=</span> LoggerFactory
  .<span style="color: #006633;">getLogger</span><span style="color: #009900;">&#40;</span>TubaQuerySenderListener.<span style="color: #000000; font-weight: bold;">class</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">protected</span> <span style="color: #000000; font-weight: bold;">final</span> SolrCore core<span style="color: #339933;">;</span>
  <span style="color: #000000; font-weight: bold;">protected</span> List<span style="color: #339933;">&lt;</span>NamedList<span style="color: #339933;">&gt;</span> queryArgs<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> init<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> NamedList args<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">String</span> coreName <span style="color: #339933;">=</span> core.<span style="color: #006633;">getName</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    queryArgs <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span>List<span style="color: #339933;">&lt;</span>NamedList<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#41;</span>args.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>coreName<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>queryArgs <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      queryArgs <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span>List<span style="color: #339933;">&lt;</span>NamedList<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#41;</span>args.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;default&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>queryArgs <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        logger.<span style="color: #006633;">warn</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;TubaQuerySenderListener not valid for core:&quot;</span><span style="color: #339933;">+</span>coreName<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span>
    logger.<span style="color: #006633;">info</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;core[&quot;</span><span style="color: #339933;">+</span>coreName<span style="color: #339933;">+</span><span style="color: #0000ff;">&quot;]register TubaQuerySenderListener : &quot;</span> <span style="color: #339933;">+</span> queryArgs<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> TubaQuerySenderListener<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> SolrCore core<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">core</span> <span style="color: #339933;">=</span> core<span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #008000; font-style: italic; font-weight: bold;">/**
   * Add the {@link org.apache.solr.common.params.EventParams#EVENT} with either the {@link org.apache.solr.common.params.EventParams#NEW_SEARCHER}
   * or {@link org.apache.solr.common.params.EventParams#FIRST_SEARCHER} values depending on the value of currentSearcher.
   * &lt;p/&gt;
   * Makes a copy of NamedList and then adds the parameters.
   *
   *
   * @param currentSearcher If null, add FIRST_SEARCHER, otherwise NEW_SEARCHER
   * @param nlst The named list to add the EVENT value to
   */</span>
  <span style="color: #000000; font-weight: bold;">protected</span> NamedList addEventParms<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> SolrIndexSearcher currentSearcher, <span style="color: #000000; font-weight: bold;">final</span> NamedList nlst<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> NamedList result <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> NamedList<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    result.<span style="color: #006633;">addAll</span><span style="color: #009900;">&#40;</span>nlst<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>currentSearcher <span style="color: #339933;">!=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      result.<span style="color: #006633;">add</span><span style="color: #009900;">&#40;</span>EventParams.<span style="color: #006633;">EVENT</span>, EventParams.<span style="color: #006633;">NEW_SEARCHER</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">else</span> <span style="color: #009900;">&#123;</span>
      result.<span style="color: #006633;">add</span><span style="color: #009900;">&#40;</span>EventParams.<span style="color: #006633;">EVENT</span>, EventParams.<span style="color: #006633;">FIRST_SEARCHER</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">return</span> result<span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> newSearcher<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> SolrIndexSearcher newSearcher, <span style="color: #000000; font-weight: bold;">final</span> SolrIndexSearcher currentSearcher<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>queryArgs <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">return</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">final</span> SolrIndexSearcher searcher <span style="color: #339933;">=</span> newSearcher<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> NamedList nlst <span style="color: #339933;">:</span> queryArgs<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #666666; font-style: italic;">// bind the request to a particular searcher (the newSearcher)</span>
        <span style="color: #000000; font-weight: bold;">final</span> NamedList params <span style="color: #339933;">=</span> addEventParms<span style="color: #009900;">&#40;</span>currentSearcher, nlst<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">final</span> LocalSolrQueryRequest req <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> LocalSolrQueryRequest<span style="color: #009900;">&#40;</span>core,params<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
          @Override <span style="color: #000000; font-weight: bold;">public</span> SolrIndexSearcher getSearcher<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span> <span style="color: #000000; font-weight: bold;">return</span> searcher<span style="color: #339933;">;</span> <span style="color: #009900;">&#125;</span>
          @Override <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> close<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span> <span style="color: #009900;">&#125;</span>
        <span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #000000; font-weight: bold;">final</span> SolrQueryResponse rsp <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> SolrQueryResponse<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        core.<span style="color: #006633;">execute</span><span style="color: #009900;">&#40;</span>core.<span style="color: #006633;">getRequestHandler</span><span style="color: #009900;">&#40;</span>req.<span style="color: #006633;">getParams</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>CommonParams.<span style="color: #006633;">QT</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span>, req, rsp<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #666666; font-style: italic;">// Retrieve the Document instances (not just the ids) to warm</span>
        <span style="color: #666666; font-style: italic;">// the OS disk cache, and any Solr document cache.  Only the top</span>
        <span style="color: #666666; font-style: italic;">// level values in the NamedList are checked for DocLists.</span>
        <span style="color: #000000; font-weight: bold;">final</span> NamedList values <span style="color: #339933;">=</span> rsp.<span style="color: #006633;">getValues</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span> i<span style="color: #339933;">&lt;</span>values.<span style="color: #006633;">size</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span> i<span style="color: #339933;">++</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
          <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> o <span style="color: #339933;">=</span> values.<span style="color: #006633;">getVal</span><span style="color: #009900;">&#40;</span>i<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
          <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>o <span style="color: #000000; font-weight: bold;">instanceof</span> DocList<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
            <span style="color: #000000; font-weight: bold;">final</span> DocList docs <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span>DocList<span style="color: #009900;">&#41;</span>o<span style="color: #339933;">;</span>
            <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> DocIterator iter <span style="color: #339933;">=</span> docs.<span style="color: #006633;">iterator</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span> iter.<span style="color: #006633;">hasNext</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
              newSearcher.<span style="color: #006633;">doc</span><span style="color: #009900;">&#40;</span>iter.<span style="color: #006633;">nextDoc</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #009900;">&#125;</span>
          <span style="color: #009900;">&#125;</span>
        <span style="color: #009900;">&#125;</span>
        req.<span style="color: #006633;">close</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #666666; font-style: italic;">// do nothing... we want to continue with the other requests.</span>
        <span style="color: #666666; font-style: italic;">// the failure should have already been logged.</span>
        logger.<span style="color: #006633;">warn</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;&quot;</span>,e<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span>
    logger.<span style="color: #006633;">info</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;core[&quot;</span><span style="color: #339933;">+</span>core.<span style="color: #006633;">getName</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">+</span><span style="color: #0000ff;">&quot;]TubaQuerySenderListener newSearcher done.&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> postCommit<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">throw</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">UnsupportedOperationException</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>	solrconfig.xml中的配置示例如下，其中firstSearcher和newSearcher的配置是一样的，不过这不意味着它们是必须一样的。</p>

<div class="wp_syntax"><div class="code"><pre class="xml" style="font-family:monospace;">	<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;query<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
		<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;listener</span> <span style="color: #000066;">event</span>=<span style="color: #ff0000;">&quot;firstSearcher&quot;</span></span>
<span style="color: #009900;">			<span style="color: #000066;">class</span>=<span style="color: #ff0000;">&quot;com.tintintech.tuba.search.TubaQuerySenderListener&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>
			<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;arr</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;default&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>
				<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;q&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>手机<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;start&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>0<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;rows&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>10<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
				<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
			<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/arr<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
			<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;arr</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;core1&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>
				<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;q&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>手机<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;start&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>0<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;rows&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>10<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;sort&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>at desc<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
				<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
			<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/arr<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
		<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/listener<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
&nbsp;
		<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;listener</span> <span style="color: #000066;">event</span>=<span style="color: #ff0000;">&quot;newSearcher&quot;</span></span>
<span style="color: #009900;">			<span style="color: #000066;">class</span>=<span style="color: #ff0000;">&quot;com.tintintech.tuba.search.TubaQuerySenderListener&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>
			<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;arr</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;default&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>
				<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;q&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>手机<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;start&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>0<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;rows&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>10<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
				<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
			<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/arr<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
			<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;arr</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;core1&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>
				<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;q&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>手机<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;start&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>0<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;rows&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>10<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
					<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;sort&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>at desc<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
				<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
			<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/arr<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
		<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/listener<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
	<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/query<span style="color: #000000; font-weight: bold;">&gt;</span></span></span></pre></div></div>

]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/09/326.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Solr之困</title>
		<link>http://www.kafka0102.com/2010/08/319.html</link>
		<comments>http://www.kafka0102.com/2010/08/319.html#comments</comments>
		<pubDate>Sat, 21 Aug 2010 20:36:37 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[solr]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=319</guid>
		<description><![CDATA[重写公司的站内搜索。经过前期一段时间对lucene和solr的熟悉，最后决定使用Solr作为新系统的基础框架。现在已经是第一阶段开发的后期，核心代码行数有11000+（不包含admin及client等）。现已实现的功能要比已有系统要丰富些，但综合比较两个系统总的代码量，其实新系统并不多得太多。新系统使用Solr代替了已有系统实现的部分功能，这减少了新系统的代码量，同是新系统实现了已有系统不具有的功能，也增加了一些代码量。开发的这段时间，因为新系统中很多代码是独立于Solr的，所以和Solr的交互也是时断时续，以使得即便到了开发后期我还能发现Solr实现的一些细节带给我的困扰。]]></description>
			<content:encoded><![CDATA[<p>重写公司的站内搜索。经过前期一段时间对lucene和solr的熟悉，最后决定使用Solr作为新系统的基础框架。现在已经是第一阶段开发的后期，核心代码行数有11000+（不包含admin及client等）。现已实现的功能要比已有系统要丰富些，但综合比较两个系统总的代码量，其实新系统并不多得太多。新系统使用Solr代替了已有系统实现的部分功能，这减少了新系统的代码量，同是新系统实现了已有系统不具有的功能，也增加了一些代码量。开发的这段时间，因为新系统中很多代码是独立于Solr的，所以和Solr的交互也是时断时续，以使得即便到了开发后期我还能发现Solr实现的一些细节带给我的困扰。</p>
<p>抛开我所做的系统来说，如果要选择一个站内搜索解决方案，Solr在某些场景下可能是个很不错的选择。因为Solr提供了Web server支持通过Http来更新索引、重建索引、查询等功能，如果需求对上Solr，甚至可以不需要基于Solr做二次开发就直接满足需要。多美妙的事情阿。不过，如果你需要些高级功能，那么可能你需要基于Solr做些工作了。比如，如果索引库很大，可以将索引库拆成多个shard，查询时对多个shard进行，这个功能Solr是支持的；不过，建索引的事情就需要自己搞定了，比如在Solr前面加个Proxy（或者只是个库函数），在建索引时根据特定的策略提交到不同的shard上。这个其实也还好了，但如果我需要一个涉及到多个索引库（各索引库有不同的schema）的查询，比如要做整站搜索，那么Solr的shard查询就用不上了，因为它必须要求各shard的schema一致。而我要做的实际是个通用搜索，这样的问题就有些接踵而至了。尽管和Solr磨合的过程花了不少时间，涉及到对它提供的功能、设计、源码的理解等等，并且有时还要妥协它开发，有时还要舍弃它已实现的功能而另起炉灶。但不可否认的是，对于初涉站内搜索开发的我来说，使用Solr并不是太坏的选择，从中也学到了Solr优秀的地方，同时也看到它不足的地方，都是收获。本文会简单的总结下个人在应用Solr过程中一些不是很爽的地方，爽的地方姑且按下不表。</p>
<p>Solr实现上有个核心东西，就是SolrCore。每个SolrCore对应着一个索引库，几乎所有的操作都是针对单个SolrCore进行的，似乎Solr的初衷就是如此，并没有考虑到多个SolrCore之间的关联。所以，可以看到的是，每个SolrQueryRequest都会关联到一个SolrCore，SolrRequestHandler的获得也是从SolrCore取得的。这糟糕的设计使得，当需要对多个SolrCore做管理时，Solr不得不做出CoreAdminHandler，它虽然实现了SolrRequestHandler接口，但它是脱离于SolrCore的，使得配置上也和其他handler不一样。而Solr的shard查询的支持就更糟糕，它要求shard的SolrCore的schema都是一致的，而不能查询异构的SolrCore。为了解决这个问题，我在Solr基础上加了个VirtualCore（这个概念现在看起来不是很好，或许IndexCore会更好些），VirtualCore里面可以包含一个或多个SolrCore，而很多操作就不是针对SolrCore而是针对VirtualCore了。比如索引库index被拆分成index.0、index.1、index.2，无论索引还是查询，客户端只需要向系统针对core=index进行请求，无需关心index被系统拆分成几个库，这些库被如何分布，系统会通过配置把这些事情做好。对于整站多个库的联合查询，就是针对多个VirtualCore进行，可以通过配置指定各个VirtualCore的请求参数而不需要像Solr那样有严格的约束。</p>
<p>引入了VirtualCore，使得Solr的一些实现不能得手的使用上。首当其冲的就是它的SearchHandler，我不得不在它的基础上重写了一个，它的shard请求异常处理策略也很有问题，如果shard请求中的某个出现异常，它就不会返回结果，这样做的好处是保证返回结果的全局准确性，但却降低了可用性。这里也需要考虑到查询结果cache的问题，如果在Solr前面加了查询结果Cache，那么Solr这种准确性要求就是有必要的。但在我的实现中，是可以有多少shard返回就处理多少，但在异常的情况下就不做查询结果cache处理。</p>
<p>VirtualCore也使得Solr强悍的DIH也用不上了，但即便没有VirtualCore，DIH也很难解决单点提交多个shard索引的问题。DIH直接对索引的SolrCore做重建索引处理，并没有对重建索引过程提供灵活的hook（尽管它确实提供了一些hook）。就我的需求来说，我希望每索引一个文档同时会根据一定的策略来更新摘要数据库，我浏览了DIH的文档和代码，似乎很难做到。而且，DIH是直接在现有索引上做重建的，如果重建时间很长或者出现问题，使得同时进来的更新索引被阻塞，就会影响到正常的服务。</p>
<p>Solr对配置文件的把握上也不够好。Solr对solrconfig.xml文件提供了Java属性值替换配置文件变量，但solr.xml却没有支持，使得线上线下配置文件中充斥着不同的绝对路径。也有好的一方面，比如schema.xml支持Xinclude，使得多个索引库的schema.xml可以共用相同的field type定义。不过，如果多个索引库的schema能集中在一个文件而不是散落成多个文件，管理起来会更方便。这样的问题同时也存在于solrconfig.xml，尽管solrconfig.xml大多数项的配置都是通用的，不过多个索引库时，searcher的warm请求参数可能就会不一样，这使得我在考虑安排时间改写它的默认Lisnter的实现。</p>
<p>Solr的索引复制有一个细节，那就是master和slave保持长连接，master通过调用OutputStream的flush方法不断把数据发送给slave，如果使用Servlet容器，通过Servlet得到OutputStream这样做没什么问题，但如果使用Netty作为服务器框架，并且使用Netty的http实现，那就实现不了这个效果。这使得我不得不放弃Netty改用Jetty了。</p>
<p>再回到查询上，Solr的SearchHandler只会得到doc id list，而不会得到需要的所请求的字段内容，它是在ResponseWriter输出时根据doc id从IndexReader得到需要的字段。在我的设计中，索引只会存储逻辑主键id，得到逻辑主键id后再从另外的摘要库把其他字段取回（或者就是返回id列表给客户端），但我显然需要在ResponseWriter输出前做完这些事情，这使得我并不得不修改request需要返回的字段列表为空。而这个ResponseWriter是需要和SolrCore的schema绑定的，结果对于并不存在的VirtualCore，我还不得不使用上配置为空并且没有索引的fake schema蒙混过去。</p>
<p>还是关于配置，Solr复制slave端配置的master url需要指定参数core，这使得每个SolrCore都有不同的master url而不能共用一个solrconfig.xml，而我真的很希望它们能共用一个solrconfig.xml。其实这个core参数在ReplicationHandler中完全可以得到，Solr没这么做的一个可能的原因是，它支持的请求url格式是http://host/corename/qt?xx=dd，把corename作为url path的一部分让我用起来很不爽，所以我把请求的格式格式改成：http://host/qt?core=aaa&amp;xx=dd，并出下策把Solr和复制相关的代码拷过来，增加了几行代码完事。</p>
<p>问题当然还有，但就像上面提到的，遇到问题总要找个解决方案，尽管有的方案看起来有些二。在回想上面提到的问题之后，我对现在完成的产出的可用性有些怀疑，我到现在还没有完整的测试过这个系统，所以，它还需要我更仔细的打磨。值得庆幸的是，随着对Solr了解的深入，我能更好的驾驭它了。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/08/319.html/feed</wfw:commentRss>
		<slash:comments>6</slash:comments>
		</item>
		<item>
		<title>HttpClient的“Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.”警告释疑</title>
		<link>http://www.kafka0102.com/2010/08/316.html</link>
		<comments>http://www.kafka0102.com/2010/08/316.html#comments</comments>
		<pubDate>Sat, 21 Aug 2010 07:43:19 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[java]]></category>
		<category><![CDATA[httpclient]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=316</guid>
		<description><![CDATA[使用HttpClient，总是报出“Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.”的WARN日志，定位到HttpClient的源码如下：

    public byte&#91;&#93; getResponseBody&#40;&#41; throws IOException &#123;
        if &#40;this.responseBody == null&#41; &#123;
            InputStream instream = getResponseBodyAsStream&#40;&#41;;
     [...]]]></description>
			<content:encoded><![CDATA[<p>使用HttpClient，总是报出“Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.”的WARN日志，定位到HttpClient的源码如下：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">byte</span><span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> getResponseBody<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #000000; font-weight: bold;">throws</span> <span style="color: #003399;">IOException</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">responseBody</span> <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
            <span style="color: #003399;">InputStream</span> instream <span style="color: #339933;">=</span> getResponseBodyAsStream<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>instream <span style="color: #339933;">!=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                <span style="color: #000066; font-weight: bold;">long</span> contentLength <span style="color: #339933;">=</span> getResponseContentLength<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>contentLength <span style="color: #339933;">&gt;</span> <span style="color: #003399;">Integer</span>.<span style="color: #006633;">MAX_VALUE</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span> <span style="color: #666666; font-style: italic;">//guard below cast from overflow</span>
                    <span style="color: #000000; font-weight: bold;">throw</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">IOException</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;Content too large to be buffered: &quot;</span><span style="color: #339933;">+</span> contentLength <span style="color: #339933;">+</span><span style="color: #0000ff;">&quot; bytes&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #009900;">&#125;</span>
                <span style="color: #000066; font-weight: bold;">int</span> limit <span style="color: #339933;">=</span> getParams<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">getIntParameter</span><span style="color: #009900;">&#40;</span>HttpMethodParams.<span style="color: #006633;">BUFFER_WARN_TRIGGER_LIMIT</span>, <span style="color: #cc66cc;">1024</span><span style="color: #339933;">*</span><span style="color: #cc66cc;">1024</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span><span style="color: #009900;">&#40;</span>contentLength <span style="color: #339933;">==</span> <span style="color: #339933;">-</span><span style="color: #cc66cc;">1</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">||</span> <span style="color: #009900;">&#40;</span>contentLength <span style="color: #339933;">&gt;</span> limit<span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                    LOG.<span style="color: #006633;">warn</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;Going to buffer response body of large or unknown size. &quot;</span>
                            <span style="color: #339933;">+</span><span style="color: #0000ff;">&quot;Using getResponseBodyAsStream instead is recommended.&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #009900;">&#125;</span>
                LOG.<span style="color: #006633;">debug</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;Buffering response body&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #003399;">ByteArrayOutputStream</span> outstream <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">ByteArrayOutputStream</span><span style="color: #009900;">&#40;</span>
                        contentLength <span style="color: #339933;">&gt;</span> <span style="color: #cc66cc;">0</span> <span style="color: #339933;">?</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span><span style="color: #009900;">&#41;</span> contentLength <span style="color: #339933;">:</span> DEFAULT_INITIAL_BUFFER_SIZE<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #000066; font-weight: bold;">byte</span><span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> buffer <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #000066; font-weight: bold;">byte</span><span style="color: #009900;">&#91;</span><span style="color: #cc66cc;">4096</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
                <span style="color: #000066; font-weight: bold;">int</span> len<span style="color: #339933;">;</span>
                <span style="color: #000000; font-weight: bold;">while</span> <span style="color: #009900;">&#40;</span><span style="color: #009900;">&#40;</span>len <span style="color: #339933;">=</span> instream.<span style="color: #006633;">read</span><span style="color: #009900;">&#40;</span>buffer<span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">&gt;</span> <span style="color: #cc66cc;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                    outstream.<span style="color: #006633;">write</span><span style="color: #009900;">&#40;</span>buffer, <span style="color: #cc66cc;">0</span>, len<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #009900;">&#125;</span>
                outstream.<span style="color: #006633;">close</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                setResponseStream<span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">responseBody</span> <span style="color: #339933;">=</span> outstream.<span style="color: #006633;">toByteArray</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #009900;">&#125;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">responseBody</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span></pre></div></div>

<p>    报WARN的条件是 ((contentLength == -1) || (contentLength > limit))，也就是说，或者是返回的HTTP头没有指定contentLength，或者是contentLength大于上限（默认是1M）。如果能确定返回结果的大小对程序没有显著影响，这个WARN就可以忽略，可以在日志的配置中把HttpClient的日志级别调到ERROR，不让它报出来。</p>
<p>    当然，这个警告也是有意义的，HttpClient建议使用InputStream getResponseBodyAsStream()代替byte[] getResponseBody()。对于返回结果很大或无法预知的情况，就需要使用InputStream getResponseBodyAsStream()，避免byte[] getResponseBody()可能带来的内存的耗尽问题。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/08/316.html/feed</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>构建健壮的Java基准测试</title>
		<link>http://www.kafka0102.com/2010/08/312.html</link>
		<comments>http://www.kafka0102.com/2010/08/312.html#comments</comments>
		<pubDate>Sat, 14 Aug 2010 17:34:49 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[java]]></category>
		<category><![CDATA[java benchmark]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=312</guid>
		<description><![CDATA[本周遇到几篇和基准测试相关的不错的文章，如果不是因为上周末鼓弄了一下各种锁的性能测试，我或许就会错过它们。两篇文章来自dw，分别是 Robust Java benchmarking, Part 1: Issues 和  Robust Java benchmarking, Part 2: Statistics and solutions，作者还有个专页 Java benchmarking article 提供一个Java基准测试的框架，感兴趣的可参考之。本文算是对Robust Java benchmarking, Part 1: Issues的一个简单的总结。在阅读Robust Java benchmarking两篇文章的过程中，我也看了些其中的参考文章，也有一些不错的可以拜读之。]]></description>
			<content:encoded><![CDATA[<p>本周遇到几篇和基准测试相关的不错的文章，如果不是因为上周末鼓弄了一下各种锁的性能测试，我或许就会错过它们。两篇文章来自dw，分别是<a href="http://www.ibm.com/developerworks/library/j-benchmark1.html" target="_blank"> Robust Java benchmarking, Part 1: Issues</a> 和 <a href="http://www.ibm.com/developerworks/java/library/j-benchmark2/index.html" target="_blank"> Robust Java benchmarking, Part 2: Statistics and solutions</a>，作者还有个专页<a href="http://www.ellipticgroup.com/html/benchmarkingArticle.html" target="_blank"> Java benchmarking article</a> 提供一个Java基准测试的框架，感兴趣的可参考之。本文算是对Robust Java benchmarking, Part 1: Issues的一个简单的总结。在阅读Robust Java benchmarking两篇文章的过程中，我也看了些其中的参考文章，也有一些不错的可以拜读之。</p>
<h2>1、度量执行时间</h2>
<p>基准测试通常的过程是：1）记录开始时间，2）执行代码，3）记录结束时间，4）计算时间差。Java中记录时间程序员们经常使用System.currentTimeMillis，不过该方法在精度上有偏差，这个偏差依操作系统的不同而不同，比如Win98可能会偏差55ms，Linux2.6可能偏差1ms。如果测试的task执行时间在毫秒级别，这个偏差就会影响结果的正确性了。JDK1.5中引入的System.nanoTime就是更好的选择，它在精度和准确性方面表现得都更好。而这种偏差以及其他因素可能引起的偏差，都在提醒我们，做基准测试时要使得task执行较长的时间。</p>
<h2>2、代码预热</h2>
<p>Java的执行过程是很复杂的，这其中会有很多因素影响到基准测试。通常来说，Java代码在开始执行阶段会相对很慢，之后会越来越快，直到达到稳定阶段，这一过程涉及的影响因素主要有：<br />
1）类加载。类加载涉及到文件读取、解析、校验等系列操作，所以在计算真正的task执行前需要先执行几遍task确保类加载都完成了。如果task涉及的条件分支很多，要确保各分支的代码都覆盖到。<br />
2）及时编译。JVM执行的是Java代码被翻译成的字节码，字节码的解释执行速度多数情况下是要比执行机器码慢的。所以，在Java代码执行过程中，JVM会根据执行情况将“热”的字节码编译成机器码加速执行，这就是JIT。Sun（Oracle？）的HotSpot JVM有两种启动模式，即client和server。启动时，client相比server做的优化更少，所以启动得更快。在运行时，同样一段代码，client需要1500次调用server需要10000次调用，使得JVM会将这段“热”代码编译成机器码。所以，如果要使得基准测试是在稳定阶段进行，就需要JVM将task中的代码动态编译成机器码。可以使用CompilationMXBean.getTotalCompilationTime函数以及-XX:+PrintCompilation启动参数查看及时编译情况。<br />
为解决上述两点提到的问题，一个可行的执行基准测试的过程如下：<br />
1. 执行task一次去加载所有的类。<br />
2. 执行task足够多次确保JVM的执行达到稳定阶段。<br />
3. 执行task一些次以得到task执行时间的评估值。<br />
4. 使用步骤3计算n，n是接下来task的执行次数，它要使得task的累积执行时间是足够的大。<br />
5. 度量执行n次task的总的执行时间。<br />
6. 评估单次（t/n）task的执行时间。</p>
<h2>3、动态优化</h2>
<p>动态编译不是一劳永逸的，它还涉及到一系列问题，突出的问题如下：<br />
1）Deoptimization：在Java程序运行过程中，同一段代码并不一定只被编译一次，JVM可能根据执行情况对已经编译过的代码段重新编译，以取得更好的优化效果。因为JVM通常编译的是一段“热”代码，但在运行时可能“热”代码周边的代码（比如一些并不经常走的分支）也“热”起来，这时整个代码段就可能需要重新编译。<br />
2）On-stack replacement：早期的HotSpot在编译“热”方法时，是在下次调用该方法时使用编译的字节码，本次的方法执行还是解释执行。所以，如果该次方法调用后再不调用这一方法，那么这个编译就没什么价值。比如一个极端的例子，在main函数里执行长循环操作，尽管HotSpot及时编译了机器码，但在循环执行过程中用不上，当循环结束，main也结束了，结果白白编译了机器码。所以，HotSpot后来引入了OSR，可以在方法执行过程中字节码替换成机器码。尽管OSR看起来很好，但OSR往往没有对编译的代码做最优化处理，这对基准测试来说就不是最好的选择。为避免OSR的问题，通常不要把所有的代码放到一个方法里。<br />
3）Dead-code elimination：Dead-code的情况很多，比如调用一个有返回值的方法，但返回值从来没有被调用者接收处理，JVM就可以对此做优化。这种优化在很多语言里都有，但对基准测试来说，这可不是好的情况。所以，写基准测试代码时要注意这个问题。</p>
<h2>4、Resource reclamation</h2>
<p>garbage collection and object finalization (GC/OF)是JVM自己的行为，如果程序员任其发展，它往往会在你不知道的时刻影响到基准测试。减少GC/OF的影响，通常有两种方法：1）是task运行时间长些，平衡掉GC的影响，2）是执行足够遍的task，并在每次执行后做System.gc和System.runFinalization处理，化被动为主动。</p>
<h2>5、Caching</h2>
<p>OS Cache和CPU Cache有时也会影响到基准测试。如果是测试文件IO操作，就不能忽略OS Cache的影响。如果是针对数值做测试，就可能要考虑CPU Cache的影响。</p>
<h2>6、其他因素</h2>
<p>除了上述几点，一些外界因素也需要考虑，比如测试执行过程中其他运行中的程序的影响，硬件的影响，JVM参数的影响等等。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/08/312.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Java中各种锁类型的基准性能评测</title>
		<link>http://www.kafka0102.com/2010/08/298.html</link>
		<comments>http://www.kafka0102.com/2010/08/298.html#comments</comments>
		<pubDate>Tue, 10 Aug 2010 15:58:03 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[java]]></category>
		<category><![CDATA[HashMap]]></category>
		<category><![CDATA[lock]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=298</guid>
		<description><![CDATA[周末对Java中各种类型的锁做了基准评测。测试的条件有两个：1）是10、50、100个不同的并发线程，2）是读写比例近似1:1,10:1,100:1,1000:1。测试方法是，对各种加锁的Map方法做性能评测，它们都是实现了MapWrapper接口的封装，测试的就是Map的get和put方法。测试的锁类型有：1）hashtable：直接测试Hashtable，2）synclock：对HashMap的方法直接加synchronized（理论上性能应和Hashtable相当），3）mutexlock：对HashMap的方法显示加Lock锁，4）rwlock：对HashMap加读写锁，5）concrrent：直接使用ConcurrentHashMap的方法，6）对HashMap读操作不加锁，写操作加Lock。]]></description>
			<content:encoded><![CDATA[<p>周末对Java中各种类型的锁做了基准评测。测试的条件有两个：1）是10、50、100个不同的并发线程，2）是读写比例近似1:1,10:1,100:1,1000:1。测试方法是，对各种加锁的Map方法做性能评测，它们都是实现了MapWrapper接口的封装，测试的就是Map的get和put方法。测试的锁类型有：1）hashtable：直接测试Hashtable，2）synclock：对HashMap的方法直接加synchronized（理论上性能应和Hashtable相当），3）mutexlock：对HashMap的方法显示加Lock锁，4）rwlock：对HashMap加读写锁，5）concrrent：直接使用ConcurrentHashMap的方法，6）对HashMap读操作不加锁，写操作加Lock。针对每种测试条件，起的每个线程执行特定次数（10万或100万）次读写操作，写入的内容是随机生成的整数，每种条件下一个线程要跑三遍，取总的处理时间的平均值作为计算结果。<br />
评测的代码如下：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">interface</span> MapWrapper <span style="color: #009900;">&#123;</span>
&nbsp;
  <span style="color: #000066; font-weight: bold;">void</span> put<span style="color: #009900;">&#40;</span><span style="color: #003399;">Object</span> key,<span style="color: #003399;">Object</span> value<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #003399;">Object</span> get<span style="color: #009900;">&#40;</span><span style="color: #003399;">Object</span> key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000066; font-weight: bold;">void</span> clear<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #003399;">String</span> getName<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> HashTableMapWrapper <span style="color: #000000; font-weight: bold;">implements</span> MapWrapper<span style="color: #009900;">&#123;</span>
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> Map<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span> map<span style="color: #339933;">;</span>
&nbsp;
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> HashTableMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> Hashtable<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> clear<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map.<span style="color: #006633;">clear</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> get<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> map.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> put<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> value<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map.<span style="color: #006633;">put</span><span style="color: #009900;">&#40;</span>key, value<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">String</span> getName<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #0000ff;">&quot;hashtable&quot;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> SyncMapWrapper  <span style="color: #000000; font-weight: bold;">implements</span> MapWrapper<span style="color: #009900;">&#123;</span>
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> Map<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span> map<span style="color: #339933;">;</span>
&nbsp;
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> SyncMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> HashMap<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">synchronized</span> <span style="color: #000066; font-weight: bold;">void</span> clear<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map.<span style="color: #006633;">clear</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">synchronized</span> <span style="color: #003399;">Object</span>  get<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> map.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">synchronized</span> <span style="color: #000066; font-weight: bold;">void</span> put<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> value<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map.<span style="color: #006633;">put</span><span style="color: #009900;">&#40;</span>key, value<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">String</span> getName<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #0000ff;">&quot;synclock&quot;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> LockMapWrapper <span style="color: #000000; font-weight: bold;">implements</span> MapWrapper<span style="color: #009900;">&#123;</span>
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> Map<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span> map<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> Lock lock<span style="color: #339933;">;</span>
&nbsp;
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> LockMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> HashMap<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    lock <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> ReentrantLock<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> clear<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    lock.<span style="color: #006633;">lock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
      map.<span style="color: #006633;">clear</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">finally</span> <span style="color: #009900;">&#123;</span>
      lock.<span style="color: #006633;">unlock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> get<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    lock.<span style="color: #006633;">lock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">return</span> map.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #666666; font-style: italic;">// TODO: handle exception</span>
    <span style="color: #009900;">&#125;</span><span style="color: #000000; font-weight: bold;">finally</span> <span style="color: #009900;">&#123;</span>
      lock.<span style="color: #006633;">unlock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> put<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> value<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    lock.<span style="color: #006633;">lock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
      map.<span style="color: #006633;">put</span><span style="color: #009900;">&#40;</span>key, value<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">finally</span> <span style="color: #009900;">&#123;</span>
      lock.<span style="color: #006633;">unlock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">String</span> getName<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #0000ff;">&quot;mutexlock&quot;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> RWLockMapWrapper <span style="color: #000000; font-weight: bold;">implements</span> MapWrapper<span style="color: #009900;">&#123;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> Map<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span> map<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> Lock readLock<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> Lock writeLock<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> RWLockMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> HashMap<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> ReentrantReadWriteLock lock <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> ReentrantReadWriteLock<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    readLock <span style="color: #339933;">=</span> lock.<span style="color: #006633;">readLock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    writeLock <span style="color: #339933;">=</span> lock.<span style="color: #006633;">writeLock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> clear<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    writeLock.<span style="color: #006633;">lock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
      map.<span style="color: #006633;">clear</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">finally</span> <span style="color: #009900;">&#123;</span>
      writeLock.<span style="color: #006633;">unlock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> get<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    readLock.<span style="color: #006633;">lock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">return</span> map.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #666666; font-style: italic;">// TODO: handle exception</span>
    <span style="color: #009900;">&#125;</span><span style="color: #000000; font-weight: bold;">finally</span> <span style="color: #009900;">&#123;</span>
      readLock.<span style="color: #006633;">unlock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> put<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> value<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    writeLock.<span style="color: #006633;">lock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
      map.<span style="color: #006633;">put</span><span style="color: #009900;">&#40;</span>key, value<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">finally</span> <span style="color: #009900;">&#123;</span>
      writeLock.<span style="color: #006633;">unlock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">String</span> getName<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #0000ff;">&quot;rwlock&quot;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> ConcurrentMapWrapper  <span style="color: #000000; font-weight: bold;">implements</span> MapWrapper<span style="color: #009900;">&#123;</span>
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> Map<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span> map<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> ConcurrentMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> ConcurrentHashMap<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> clear<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map.<span style="color: #006633;">clear</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> get<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> map.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> put<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> value<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map.<span style="color: #006633;">put</span><span style="color: #009900;">&#40;</span>key, value<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">String</span> getName<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #0000ff;">&quot;concrrent&quot;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> WriteLockMapWrapper <span style="color: #000000; font-weight: bold;">implements</span> MapWrapper<span style="color: #009900;">&#123;</span>
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> Map<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span> map<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> Lock lock<span style="color: #339933;">;</span>
&nbsp;
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> WriteLockMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    map <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> HashMap<span style="color: #339933;">&lt;</span>Object,Object<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    lock <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> ReentrantLock<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> clear<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    lock.<span style="color: #006633;">lock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
      map.<span style="color: #006633;">clear</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">finally</span> <span style="color: #009900;">&#123;</span>
      lock.<span style="color: #006633;">unlock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> get<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> map.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> put<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> value<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    lock.<span style="color: #006633;">lock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
      map.<span style="color: #006633;">put</span><span style="color: #009900;">&#40;</span>key, value<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">finally</span> <span style="color: #009900;">&#123;</span>
      lock.<span style="color: #006633;">unlock</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  @Override
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">String</span> getName<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #0000ff;">&quot;writelock&quot;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span>
&nbsp;
&nbsp;
<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> BenchMark <span style="color: #009900;">&#123;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">volatile</span> <span style="color: #000066; font-weight: bold;">long</span> totalTime<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">private</span> CountDownLatch latch<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> loop<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> threads<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">float</span> ratio<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> BenchMark<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> loop, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> threads,<span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">float</span> ratio<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">super</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">loop</span> <span style="color: #339933;">=</span> loop<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">threads</span> <span style="color: #339933;">=</span> threads<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">ratio</span> <span style="color: #339933;">=</span> ratio<span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">class</span> BenchMarkRunnable <span style="color: #000000; font-weight: bold;">implements</span> <span style="color: #003399;">Runnable</span><span style="color: #009900;">&#123;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> MapWrapper mapWrapper<span style="color: #339933;">;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> size<span style="color: #339933;">;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> benchmarkRandomReadPut<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> MapWrapper mapWrapper,<span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> loop<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Random</span> random <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">Random</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000066; font-weight: bold;">int</span> writeTime <span style="color: #339933;">=</span> <span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>i<span style="color: #339933;">&lt;</span>loop<span style="color: #339933;">;</span>i<span style="color: #339933;">++</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> n <span style="color: #339933;">=</span> random.<span style="color: #006633;">nextInt</span><span style="color: #009900;">&#40;</span>size<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>mapWrapper.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>n<span style="color: #009900;">&#41;</span> <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
          mapWrapper.<span style="color: #006633;">put</span><span style="color: #009900;">&#40;</span>n, n<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
          writeTime<span style="color: #339933;">++;</span>
        <span style="color: #009900;">&#125;</span>
      <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> BenchMarkRunnable<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> MapWrapper mapWrapper,<span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> size<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">mapWrapper</span> <span style="color: #339933;">=</span> mapWrapper<span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">size</span> <span style="color: #339933;">=</span> size<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    @Override
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> run<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">long</span> start <span style="color: #339933;">=</span> <span style="color: #003399;">System</span>.<span style="color: #006633;">currentTimeMillis</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      benchmarkRandomReadPut<span style="color: #009900;">&#40;</span>mapWrapper,loop<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">long</span> end <span style="color: #339933;">=</span> <span style="color: #003399;">System</span>.<span style="color: #006633;">currentTimeMillis</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      totalTime <span style="color: #339933;">+=</span> end<span style="color: #339933;">-</span>start<span style="color: #339933;">;</span>
      latch.<span style="color: #006633;">countDown</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> benchmark<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> MapWrapper mapWrapper<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">float</span> size <span style="color: #339933;">=</span> loop<span style="color: #339933;">*</span>threads<span style="color: #339933;">*</span>ratio<span style="color: #339933;">;</span>
    totalTime <span style="color: #339933;">=</span> <span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> k<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>k<span style="color: #339933;">&lt;</span><span style="color: #cc66cc;">3</span><span style="color: #339933;">;</span>k<span style="color: #339933;">++</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      latch <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> CountDownLatch<span style="color: #009900;">&#40;</span>threads<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>i<span style="color: #339933;">&lt;</span>threads<span style="color: #339933;">;</span>i<span style="color: #339933;">++</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">Thread</span><span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">new</span> BenchMarkRunnable<span style="color: #009900;">&#40;</span>mapWrapper,<span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span><span style="color: #009900;">&#41;</span>size<span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">start</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
      <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
        latch.<span style="color: #006633;">await</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        e.<span style="color: #006633;">printStackTrace</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
      mapWrapper.<span style="color: #006633;">clear</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #003399;">Runtime</span>.<span style="color: #006633;">getRuntime</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">gc</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #003399;">Runtime</span>.<span style="color: #006633;">getRuntime</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">runFinalization</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #003399;">Thread</span>.<span style="color: #006633;">sleep</span><span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">1000</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> rwratio <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">1.0</span><span style="color: #339933;">/</span>ratio<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #003399;">System</span>.<span style="color: #006633;">out</span>.<span style="color: #006633;">println</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;[&quot;</span><span style="color: #339933;">+</span>mapWrapper.<span style="color: #006633;">getName</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">+</span><span style="color: #0000ff;">&quot;]threadnum[&quot;</span><span style="color: #339933;">+</span>threads<span style="color: #339933;">+</span><span style="color: #0000ff;">&quot;]ratio[&quot;</span><span style="color: #339933;">+</span>rwratio<span style="color: #339933;">+</span><span style="color: #0000ff;">&quot;]avgtime[&quot;</span><span style="color: #339933;">+</span>totalTime<span style="color: #339933;">/</span><span style="color: #cc66cc;">3</span><span style="color: #339933;">+</span><span style="color: #0000ff;">&quot;]&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000066; font-weight: bold;">void</span> benchmark2<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> loop, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> threads,<span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">float</span> ratio<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> BenchMark benchMark <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> BenchMark<span style="color: #009900;">&#40;</span>loop,threads,ratio<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> MapWrapper<span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> wrappers <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> MapWrapper<span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">new</span> HashTableMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>,
        <span style="color: #000000; font-weight: bold;">new</span> SyncMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>,
        <span style="color: #000000; font-weight: bold;">new</span> LockMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>,
        <span style="color: #000000; font-weight: bold;">new</span> RWLockMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>,
        <span style="color: #000000; font-weight: bold;">new</span> ConcurrentMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>,
        <span style="color: #000000; font-weight: bold;">new</span> WriteLockMapWrapper<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>,
    <span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> MapWrapper wrapper <span style="color: #339933;">:</span> wrappers<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      benchMark.<span style="color: #006633;">benchmark</span><span style="color: #009900;">&#40;</span>wrapper<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000066; font-weight: bold;">void</span> test<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    benchmark2<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">1000000</span>,<span style="color: #cc66cc;">10</span>,<span style="color: #cc66cc;">1</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//r:w 1:1</span>
    benchmark2<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">1000000</span>,<span style="color: #cc66cc;">10</span>,0.1f<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//r:w 10:1</span>
    benchmark2<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">1000000</span>,<span style="color: #cc66cc;">10</span>,0.01f<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//r:w 100:1</span>
    benchmark2<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">1000000</span>,<span style="color: #cc66cc;">10</span>,0.001f<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//r:w 1000:1</span>
    <span style="color: #666666; font-style: italic;">/////</span>
    benchmark2<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">1000000</span>,<span style="color: #cc66cc;">50</span>,0.1f<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//r:w 10:1</span>
    benchmark2<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">1000000</span>,<span style="color: #cc66cc;">50</span>,0.01f<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//r:w 100:1</span>
    benchmark2<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">1000000</span>,<span style="color: #cc66cc;">50</span>,0.001f<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//r:w 1000:1</span>
    <span style="color: #666666; font-style: italic;">/////</span>
    benchmark2<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">100000</span>,<span style="color: #cc66cc;">100</span>,1f<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//r:w 10:1</span>
    benchmark2<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">100000</span>,<span style="color: #cc66cc;">100</span>,0.1f<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//r:w 10:1</span>
    benchmark2<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">100000</span>,<span style="color: #cc66cc;">100</span>,0.01f<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//r:w 100:1</span>
    benchmark2<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">100000</span>,<span style="color: #cc66cc;">100</span>,0.001f<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//r:w 1000:1</span>
    <span style="color: #666666; font-style: italic;">////</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000066; font-weight: bold;">void</span> main<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">String</span><span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> args<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    test<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>测试是在本机进行的，JVM参数是： -Xms2048M -Xmx2048M -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:+HeapDumpOnOutOfMemoryError。依次是并发10、50、100个线程测试得到的数据表如下：<br />
<a href="http://www.kafka0102.com/wp-content/uploads/2010/08/lock_data.png"><img src="http://www.kafka0102.com/wp-content/uploads/2010/08/lock_data.png" alt="" title="lock_data" width="556" height="375" class="aligncenter size-full wp-image-305" /></a><br />
</br><br />
基于上面的数据使用openoffice做出的曲线对比图如下：<br />
1、并发10个线程的平均耗时曲线对比图：<br />
<a href="http://www.kafka0102.com/wp-content/uploads/2010/08/10_lock.jpg"><img src="http://www.kafka0102.com/wp-content/uploads/2010/08/10_lock.jpg" alt="" title="10_lock" width="599" height="330" class="aligncenter size-full wp-image-306" /></a><br />
<br />
2、并发50个线程的平均耗时曲线对比图：<br />
<a href="http://www.kafka0102.com/wp-content/uploads/2010/08/50_lock.jpg"><img src="http://www.kafka0102.com/wp-content/uploads/2010/08/50_lock.jpg" alt="" title="50_lock" width="604" height="439" class="aligncenter size-full wp-image-307" /></a><br />
<br />
3、并发100个线程的平均耗时曲线对比图：<br />
<a href="http://www.kafka0102.com/wp-content/uploads/2010/08/100_lock.jpg"><img src="http://www.kafka0102.com/wp-content/uploads/2010/08/100_lock.jpg" alt="" title="100_lock" width="598" height="362" class="aligncenter size-full wp-image-308" /></a><br />
<br />
对于测试数据来说，不必太当真，同样的用例跑几遍数据都会有偏差，但总体上数据间的对比应该基本一致的。对于上面的测试结果，其中并发50个线程时没有读写1:1的数据，这其实是我的一个失误，到整理结果时才发现。但在并发100个线程时，我调整了单线程的循环次数，从上面的100万次调整到10万次，这是因为在100万次时我的机器已经严重CPU耗尽，久久不出结果，并伴有大量全GC。</p>
<p>根据测试结果，可以得出下面的结论：<br />
1、读写比例相近（1:1）时，各种并发的速度差不多，只有ConcurrentHashMap因为分段加锁，性能稍好些。<br />
2、在几种并发及读写比例中，hashtable都表现得很差，即便是和显示加互斥锁的synclock和mutexlock相比。<br />
3、并发线程越多，锁的影响就越有体现。如果加锁的方法处理耗时更长，这种对比就更加明显（比如文件操作等）。<br />
4、直观的数据对比来说，在100线程并发读写1000:1的最大化条件下，表现最差的hashtable和表现最好的concurrent性能比是11:1,其他条件下的比例都在这之内。<br />
5、对比rwlock和writelock（rwlock加读写锁，writelock读不加锁写加互斥锁），读写锁的开销还是有一些的，因读写比例的不同而有几倍的差距。因为concurrent和writelock的区别在于concurrent是分段加锁，所以它俩的读写比例大时差别不大。<br />
6、综合分析，concurrent这种读不加锁、写加分段锁的做法是效果最好的（尽管读有时会脏读的情况）。</p>
<p>对于这种基准测试，如果你有更好更精确的方法，也请不吝分享之。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/08/298.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>[Solr源码分析]LRUCache和FastLRUCache实现分析</title>
		<link>http://www.kafka0102.com/2010/08/293.html</link>
		<comments>http://www.kafka0102.com/2010/08/293.html#comments</comments>
		<pubDate>Sun, 08 Aug 2010 16:01:42 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[cache]]></category>
		<category><![CDATA[solr]]></category>
		<category><![CDATA[HashMap]]></category>
		<category><![CDATA[java]]></category>
		<category><![CDATA[源码分析]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=293</guid>
		<description><![CDATA[	在 [Solr 实践]Solr Cache使用介绍及分析 一文我有对Solr的LRUCache和FastLRUCache做了一些介绍，本文在此基础对其实现做些补充。
1、LRUCache的实现分析
	在分析LRUCache前先对LinkedHashMap做些介绍。LinkedHashMap继承于HashMap，它使用了一个双向链表来存储Map中的Entry顺序关系，这种顺序有两种，一种是LRU顺序，一种是插入顺序，这可以由其构造函数public LinkedHashMap(int initialCapacity,float loadFactor,                   boolean accessOrder)指定。所以，对于get、put、remove等操作，LinkedHashMap除了要做HashMap做的事情，还做些调整Entry顺序链表的工作。
	以get操作为例，如果是LRU顺序（accessOrder为true），Entry的recordAccess方法就调整get到的Entry到链表的头部去：

   public V get&#40;Object key&#41; &#123;
        Entry&#60;K,V&#62; e = &#40;Entry&#60;K,V&#62;&#41;getEntry&#40;key&#41;;
        if &#40;e [...]]]></description>
			<content:encoded><![CDATA[<p>	在 <a href="http://www.kafka0102.com/2010/08/267.html" target="blank">[Solr 实践]Solr Cache使用介绍及分析</a> 一文我有对Solr的LRUCache和FastLRUCache做了一些介绍，本文在此基础对其实现做些补充。</p>
<h2>1、LRUCache的实现分析</h2>
<p>	在分析LRUCache前先对LinkedHashMap做些介绍。LinkedHashMap继承于HashMap，它使用了一个双向链表来存储Map中的Entry顺序关系，这种顺序有两种，一种是LRU顺序，一种是插入顺序，这可以由其构造函数public LinkedHashMap(int initialCapacity,float loadFactor,                   boolean accessOrder)指定。所以，对于get、put、remove等操作，LinkedHashMap除了要做HashMap做的事情，还做些调整Entry顺序链表的工作。<br />
	以get操作为例，如果是LRU顺序（accessOrder为true），Entry的recordAccess方法就调整get到的Entry到链表的头部去：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">   <span style="color: #000000; font-weight: bold;">public</span> V get<span style="color: #009900;">&#40;</span><span style="color: #003399;">Object</span> key<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        Entry<span style="color: #339933;">&lt;</span>K,V<span style="color: #339933;">&gt;</span> e <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span>Entry<span style="color: #339933;">&lt;</span>K,V<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#41;</span>getEntry<span style="color: #009900;">&#40;</span>key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>e <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span>
        e.<span style="color: #006633;">recordAccess</span><span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">this</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">return</span> e.<span style="color: #006633;">value</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span></pre></div></div>

<p>    对于put来说，LinkedHashMap重写了addEntry方法：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">   <span style="color: #000066; font-weight: bold;">void</span> addEntry<span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> hash, K key, V value, <span style="color: #000066; font-weight: bold;">int</span> bucketIndex<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        createEntry<span style="color: #009900;">&#40;</span>hash, key, value, bucketIndex<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #666666; font-style: italic;">// Remove eldest entry if instructed, else grow capacity if appropriate</span>
        Entry<span style="color: #339933;">&lt;</span>K,V<span style="color: #339933;">&gt;</span> eldest <span style="color: #339933;">=</span> header.<span style="color: #006633;">after</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>removeEldestEntry<span style="color: #009900;">&#40;</span>eldest<span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
            removeEntryForKey<span style="color: #009900;">&#40;</span>eldest.<span style="color: #006633;">key</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">else</span> <span style="color: #009900;">&#123;</span>
            <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>size <span style="color: #339933;">&gt;=</span> threshold<span style="color: #009900;">&#41;</span>
                resize<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">2</span> <span style="color: #339933;">*</span> table.<span style="color: #006633;">length</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span></pre></div></div>

<p>    addEntry中调用了boolean removeEldestEntry(Map.Entry<K,V> eldest)方法，默认实现一直返回false，也就是默认的Map是没有容量限制的。LinkedHashMap的子类可以复写该方法，当当前的size大于阈值时返回true，这样LinkedHashMap就可以从Entry顺序链表中删除最旧的Entry。这使得LinkedHashMap具有了Cache的功能，可以存储限量的元素，并具有两种可选的元素淘汰策略（LRU和FIFO），其中的LRU是最常用的。<br />
	Solr的LRUCache是基于LinkedHashMap实现的，所以LRUCache的实现真的很简单，这里列出其中核心的代码片断：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> init<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Map</span> args, <span style="color: #003399;">Object</span> persistence, <span style="color: #000000; font-weight: bold;">final</span> CacheRegenerator regenerator<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
	<span style="color: #666666; font-style: italic;">//一堆解析参数参数初始化的代码</span>
	<span style="color: #666666; font-style: italic;">//map map    </span>
    map <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> LinkedHashMap<span style="color: #009900;">&#40;</span>initialSize, 0.75f, <span style="color: #000066; font-weight: bold;">true</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      @Override
      <span style="color: #000000; font-weight: bold;">protected</span> <span style="color: #000066; font-weight: bold;">boolean</span> removeEldestEntry<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Map.<span style="color: #006633;">Entry</span></span> eldest<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>size<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">&gt;</span> limit<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
          <span style="color: #666666; font-style: italic;">// increment evictions regardless of state.</span>
          <span style="color: #666666; font-style: italic;">// this doesn't need to be synchronized because it will</span>
          <span style="color: #666666; font-style: italic;">// only be called in the context of a higher level synchronized block.</span>
          evictions<span style="color: #339933;">++;</span>
          stats.<span style="color: #006633;">evictions</span>.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
          <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">true</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">false</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>persistence<span style="color: #339933;">==</span><span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #666666; font-style: italic;">// must be the first time a cache of this type is being created</span>
      persistence <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> CumulativeStats<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    stats <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span>CumulativeStats<span style="color: #009900;">&#41;</span>persistence<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">return</span> persistence<span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> put<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> value<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">synchronized</span> <span style="color: #009900;">&#40;</span>map<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>state <span style="color: #339933;">==</span> State.<span style="color: #006633;">LIVE</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        stats.<span style="color: #006633;">inserts</span>.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
      <span style="color: #666666; font-style: italic;">// increment local inserts regardless of state???</span>
      <span style="color: #666666; font-style: italic;">// it does make it more consistent with the current size...</span>
      inserts<span style="color: #339933;">++;</span>
      <span style="color: #000000; font-weight: bold;">return</span> map.<span style="color: #006633;">put</span><span style="color: #009900;">&#40;</span>key,value<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> get<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> key<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">synchronized</span> <span style="color: #009900;">&#40;</span>map<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Object</span> val <span style="color: #339933;">=</span> map.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>state <span style="color: #339933;">==</span> State.<span style="color: #006633;">LIVE</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #666666; font-style: italic;">// only increment lookups and hits if we are live.</span>
        lookups<span style="color: #339933;">++;</span>
        stats.<span style="color: #006633;">lookups</span>.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>val<span style="color: #339933;">!=</span><span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
          hits<span style="color: #339933;">++;</span>
          stats.<span style="color: #006633;">hits</span>.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
      <span style="color: #009900;">&#125;</span>
      <span style="color: #000000; font-weight: bold;">return</span> val<span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
  <span style="color: #009900;">&#125;</span></pre></div></div>

<p>  可以看到，LRUCache对读写操作直接加的互斥锁，多线程并发读写时会有锁的竞争问题。通常来说，Cache系统的读要远多于写，不能并发读是有些不够友好。不过，相比于Solr中其它耗时的操作来说，LRUCache的串行化读往往不会成为系统的瓶颈。LRUCache的优点是，直接套用LinkedHashMap，实现简单，缺点是，因为LinkedHashMap的get操作需要操作Entry顺序链表，所以必须对整个操作加锁。</p>
<h2>2、FastLRUCache的实现分析</h2>
<p>	Solr1.4引入FastLRUCache作为另一种可选的实现。FastLRUCache放弃了LinkedHashMap，而是使用现在很多Java Cache实现中使用的ConcurrentHashMap。但ConcurrentHashMap只提供了高性能的并发存取支持，并没有提供对淘汰数据的支持，所以FastLRUCache主要需要做的就是这件事情。FastLRUCache的存取操作都在ConcurrentLRUCache中实现，所以我们直接过渡到ConcurrentLRUCache的实现。<br />
	ConcurrentLRUCache的存取操作代码如下：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">  <span style="color: #000000; font-weight: bold;">public</span> V get<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> K key<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> CacheEntry<span style="color: #339933;">&lt;</span>K,V<span style="color: #339933;">&gt;</span> e <span style="color: #339933;">=</span> map.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span>key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>e <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>islive<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        stats.<span style="color: #006633;">missCounter</span>.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
      <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>islive<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      e.<span style="color: #006633;">lastAccessed</span> <span style="color: #339933;">=</span> stats.<span style="color: #006633;">accessCounter</span>.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">return</span> e.<span style="color: #006633;">value</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> V remove<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> K key<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> CacheEntry<span style="color: #339933;">&lt;</span>K,V<span style="color: #339933;">&gt;</span> cacheEntry <span style="color: #339933;">=</span> map.<span style="color: #006633;">remove</span><span style="color: #009900;">&#40;</span>key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>cacheEntry <span style="color: #339933;">!=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      stats.<span style="color: #006633;">size</span>.<span style="color: #006633;">decrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">return</span> cacheEntry.<span style="color: #006633;">value</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> put<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> K key, <span style="color: #000000; font-weight: bold;">final</span> V val<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>val <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">final</span> CacheEntry e <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> CacheEntry<span style="color: #009900;">&#40;</span>key, val, stats.<span style="color: #006633;">accessCounter</span>.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> CacheEntry oldCacheEntry <span style="color: #339933;">=</span> map.<span style="color: #006633;">put</span><span style="color: #009900;">&#40;</span>key, e<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000066; font-weight: bold;">int</span> currentSize<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>oldCacheEntry <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      currentSize <span style="color: #339933;">=</span> stats.<span style="color: #006633;">size</span>.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">else</span> <span style="color: #009900;">&#123;</span>
      currentSize <span style="color: #339933;">=</span> stats.<span style="color: #006633;">size</span>.<span style="color: #006633;">get</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>islive<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      stats.<span style="color: #006633;">putCounter</span>.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">else</span> <span style="color: #009900;">&#123;</span>
      stats.<span style="color: #006633;">nonLivePutCounter</span>.<span style="color: #006633;">incrementAndGet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #666666; font-style: italic;">// Check if we need to clear out old entries from the cache.</span>
    <span style="color: #666666; font-style: italic;">// isCleaning variable is checked instead of markAndSweepLock.isLocked()</span>
    <span style="color: #666666; font-style: italic;">// for performance because every put invokation will check until</span>
    <span style="color: #666666; font-style: italic;">// the size is back to an acceptable level.</span>
    <span style="color: #666666; font-style: italic;">// There is a race between the check and the call to markAndSweep, but</span>
    <span style="color: #666666; font-style: italic;">// it's unimportant because markAndSweep actually aquires the lock or returns if it can't.</span>
    <span style="color: #666666; font-style: italic;">// Thread safety note: isCleaning read is piggybacked (comes after) other volatile reads</span>
    <span style="color: #666666; font-style: italic;">// in this method.</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>currentSize <span style="color: #339933;">&gt;</span> upperWaterMark <span style="color: #339933;">&amp;&amp;</span> <span style="color: #339933;">!</span>isCleaning<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>newThreadForCleanup<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">Thread</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
          @Override
          <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> run<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
            markAndSweep<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
          <span style="color: #009900;">&#125;</span>
        <span style="color: #009900;">&#125;</span>.<span style="color: #006633;">start</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">else</span> <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>cleanupThread <span style="color: #339933;">!=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
        cleanupThread.<span style="color: #006633;">wakeThread</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">else</span> <span style="color: #009900;">&#123;</span>
        markAndSweep<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">return</span> oldCacheEntry <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span> <span style="color: #339933;">?</span> <span style="color: #000066; font-weight: bold;">null</span> <span style="color: #339933;">:</span> oldCacheEntry.<span style="color: #006633;">value</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span></pre></div></div>

<p>	所有的操作都是直接调用map（ConcurrentHashMap）的。看下put中的代码，当map容量达到上限并且没有其他线程在清理数据（currentSize > upperWaterMark &#038;&#038; !isCleaning），就调用markAndSweep方法清理数据，可以有3种方式做清理工作：1）在该线程同步执行，2）即时启动新线程异步执行，3）提供单独的清理线程，即时唤醒它异步执行。</p>
<p>	markAndSweep方法那是相当的冗长，这里就不罗列出来。下面叙述下它的思路。</p>
<p>	对于ConcurrentLRUCache中的每一个元素CacheEntry，它有个属性lastAccessed，表示最后访问的数值大小。ConcurrentLRUCache中的stats.accessCounter是全局的自增整数，当put或get Entry时，Entry的lastAccessed会被更新成新自增得到的accessCounter。		ConcurrentLRUCache淘汰数据就是淘汰那些lastAccessed较小的Entry。因为ConcurrentLRUCache没有维护以lastAccessed排序的Entry链表（否则就是LRUCache了），所以淘汰数据时就需要遍历整个Map中的元素来淘汰合适的Entry。这是不是要扯上排序呢？其实不用那么大动干戈。</p>
<p>	这里定义几个变量，wantToKeep表示Map中需要保留的Entry个数，wantToRemove表示需要删除的个数（wantToRemove=map.size-wantToKeep),newestEntry是最大的lastAccessed值（初始是stats.accessCounter），这三个变量初始都是已知的，oldestEntry表示最小的lastAccessed，这个是未知的，可以在遍历Entry时通过比较递进到最小。Map中的Entry有3种:(a)是可以立刻判断出可以被淘汰的，也就是lastAccessed<(oldestEntry+wantToRemove)的，（b）是可以立刻判断出可以被保留的，也就是lastAccessed>(newestEntry-1000)的，（c）除上述两者之外的就是不能准确判断是否需要被淘汰的。对于遍历一趟Map中的Entry来说，极好的情况是如果淘汰掉满足（a）的Entry后Map大小降到了wantToKeep，这种情况的典型代表是对Cache只有get和put操作，使得lastAccessed在Map中能保持连续；极坏的情况是，可能满足（a）的Entry不够多甚至没有。但遍历一趟Map至少有一个效果是，会把需要处理的Entry范围缩小到满足（c）的。如此反复迭代，一定使得Map容量调到wantToKeep。而对这个淘汰，也要考虑一个现实情况是，wantToKeep往往是接近于map.size（比如等于0.9*map.size）的，如果remove操作不是很多，那么并不需要很多次遍历就可以完成清理工作。</p>
<p>	ConcurrentLRUCache淘汰数据的基本思想如上所述。它的执行过程可以分为3个阶段。第一个阶段就是遍历Map中的每个Entry，如果满足（a）就remove，满足（b）则跳过，满足（c）则放到新map中。一遍下来后，如果map.size还大于wantToKeep，第二个阶段就再重复上述过程（实现上，Solr用了个变量numPasses，似乎想做个开关控制遍历几次，当前就固定成一次）。完了如果map.size还大于wantToKeep，第三阶段再遍历一遍Map，但这次使用PriorityQueue来提取出还需要再淘汰的N个最old的Entry，这样一次下来就收工了。需要补充一点，上面提到的wantToKeep在代码中是acceptableWaterMark和lowerWaterMark，也就是如果遍历后达到acceptableWaterMark就算完成，但操作是按lowerWaterMark的要求来。</p>
<p>	这个算法的时间复杂度是2n+kln(k)（k值在实际大多数情况下会很小），相比于直接的堆排，通常会更快些。</p>
<h2>3、总结</h2>
<p>	LRUCache和FastLRUCache两种Cache实现是两种很不同的思路。两者的相同点是，都使用了现成的Map来维护数据。不同点是如何来淘汰数据。LRUCache（也就是LinkedHashMap）格外维护了一个结构，在做存取操作时同时更新该结构，优点在于淘汰操作是O(1)的，缺点是需要对存取操作加互斥锁。FastLRUCache正相反，它没有额外维护新的结构，可以由ConcurrentHashMap支持并发读，但put操作中如果需要淘汰数据，淘汰过程是O(n)的，因为整个过程不加锁，这也只会影响该次put的性能，而FastLRUCache也可选成起独立线程异步执行来降低影响。而另一个Cache实现Ehcache，它在淘汰数据就是同步的，不过它限定了每次淘汰数据的大小（通常都少于5个），所以同步情况下性能不会太受影响。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/08/293.html/feed</wfw:commentRss>
		<slash:comments>5</slash:comments>
		</item>
		<item>
		<title>分析多线程并发写HashMap线程被hang住的原因</title>
		<link>http://www.kafka0102.com/2010/08/286.html</link>
		<comments>http://www.kafka0102.com/2010/08/286.html#comments</comments>
		<pubDate>Fri, 06 Aug 2010 21:05:39 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[java]]></category>
		<category><![CDATA[HashMap]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=286</guid>
		<description><![CDATA[在blogjava上看到一文  谁能帮忙解释一下为什么这个程序会死锁？，激发了我那能害死猫的好奇，所以很费劲的琢磨了这个问题。由于涉及的内容较多，就单独发文阐述一下。
文中提到的问题程序如下：

public class TestLock &#123;
  private final HashMap map = new HashMap&#40;&#41;;
  public TestLock&#40;&#41; &#123;
    final Thread t1 = new Thread&#40;&#41; &#123;
      @Override
      public void run&#40;&#41; &#123;
        for&#40;int i=0; i&#60;500000; i++&#41; &#123;
 [...]]]></description>
			<content:encoded><![CDATA[<p>在blogjava上看到一文 <a href="http://www.blogjava.net/zhvfeng/archive/2010/08/04/327956.html" target="blank"> 谁能帮忙解释一下为什么这个程序会死锁？</a>，激发了我那能害死猫的好奇，所以很费劲的琢磨了这个问题。由于涉及的内容较多，就单独发文阐述一下。</p>
<p>文中提到的问题程序如下：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> TestLock <span style="color: #009900;">&#123;</span>
  <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">HashMap</span> map <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">HashMap</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #000000; font-weight: bold;">public</span> TestLock<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Thread</span> t1 <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">Thread</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      @Override
      <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> run<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">for</span><span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span> i<span style="color: #339933;">&lt;</span><span style="color: #cc66cc;">500000</span><span style="color: #339933;">;</span> i<span style="color: #339933;">++</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
          map.<span style="color: #006633;">put</span><span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">Integer</span><span style="color: #009900;">&#40;</span>i<span style="color: #009900;">&#41;</span>, i<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #003399;">System</span>.<span style="color: #006633;">out</span>.<span style="color: #006633;">println</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;t1 over&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Thread</span> t2 <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">Thread</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      @Override
      <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> run<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">for</span><span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> i<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span> i<span style="color: #339933;">&lt;</span><span style="color: #cc66cc;">500000</span><span style="color: #339933;">;</span> i<span style="color: #339933;">++</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
          map.<span style="color: #006633;">put</span><span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">Integer</span><span style="color: #009900;">&#40;</span>i<span style="color: #009900;">&#41;</span>, i<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #003399;">System</span>.<span style="color: #006633;">out</span>.<span style="color: #006633;">println</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;t2 over&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
    t1.<span style="color: #006633;">start</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    t2.<span style="color: #006633;">start</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">static</span> <span style="color: #000066; font-weight: bold;">void</span> main<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">String</span><span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> args<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">new</span> TestLock<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>	就是启了两个线程，不断的往一个非线程安全的HashMap中put内容，put的内容很简单，key和value都是从0自增的整数（这个put的内容做的并不好，以致于后来干扰了我分析问题的思路）。对HashMap做并发写操作，我原以为只不过会产生脏数据的情况，但反复运行这个程序，会出现线程t1、t2被hang住的情况，多数情况下是一个线程被hang住另一个成功结束，偶尔会两个线程都被hang住。说到这里，你如果觉得不好好学习ConcurrentHashMap而在这瞎折腾就手下留情跳过吧。<br />
	好吧，分析下HashMap的put函数源码看看问题出在哪，这里就罗列出相关代码（jdk1.6）：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">public</span> V put<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> K key, <span style="color: #000000; font-weight: bold;">final</span> V value<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>key <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #000000; font-weight: bold;">return</span> putForNullKey<span style="color: #009900;">&#40;</span>value<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> hash <span style="color: #339933;">=</span> hash<span style="color: #009900;">&#40;</span>key.<span style="color: #006633;">hashCode</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> i <span style="color: #339933;">=</span> indexFor<span style="color: #009900;">&#40;</span>hash, table.<span style="color: #006633;">length</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span>Entry<span style="color: #339933;">&lt;</span>K,V<span style="color: #339933;">&gt;</span> e <span style="color: #339933;">=</span> table<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span> e <span style="color: #339933;">!=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span> e <span style="color: #339933;">=</span> e.<span style="color: #006633;">next</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span><span style="color: #666666; font-style: italic;">//@标记1</span>
      <span style="color: #003399;">Object</span> k<span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>e.<span style="color: #006633;">hash</span> <span style="color: #339933;">==</span> hash <span style="color: #339933;">&amp;&amp;</span> <span style="color: #009900;">&#40;</span><span style="color: #009900;">&#40;</span>k <span style="color: #339933;">=</span> e.<span style="color: #006633;">key</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">==</span> key <span style="color: #339933;">||</span> key.<span style="color: #006633;">equals</span><span style="color: #009900;">&#40;</span>k<span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">final</span> V oldValue <span style="color: #339933;">=</span> e.<span style="color: #006633;">value</span><span style="color: #339933;">;</span>
        e.<span style="color: #006633;">value</span> <span style="color: #339933;">=</span> value<span style="color: #339933;">;</span>
        e.<span style="color: #006633;">recordAccess</span><span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">this</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #000000; font-weight: bold;">return</span> oldValue<span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    modCount<span style="color: #339933;">++;</span>
    addEntry<span style="color: #009900;">&#40;</span>hash, key, value, i<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #000066; font-weight: bold;">void</span> addEntry<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> hash, <span style="color: #000000; font-weight: bold;">final</span> K key, <span style="color: #000000; font-weight: bold;">final</span> V value, <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> bucketIndex<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> Entry<span style="color: #339933;">&lt;</span>K,V<span style="color: #339933;">&gt;</span> e <span style="color: #339933;">=</span> table<span style="color: #009900;">&#91;</span>bucketIndex<span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
    table<span style="color: #009900;">&#91;</span>bucketIndex<span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> Entry<span style="color: #339933;">&lt;</span>K,V<span style="color: #339933;">&gt;</span><span style="color: #009900;">&#40;</span>hash, key, value, e<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>size<span style="color: #339933;">++</span> <span style="color: #339933;">&gt;=</span> threshold<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      resize<span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">2</span> <span style="color: #339933;">*</span> table.<span style="color: #006633;">length</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #000066; font-weight: bold;">void</span> resize<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> newCapacity<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> Entry<span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> oldTable <span style="color: #339933;">=</span> table<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> oldCapacity <span style="color: #339933;">=</span> oldTable.<span style="color: #006633;">length</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>oldCapacity <span style="color: #339933;">==</span> MAXIMUM_CAPACITY<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      threshold <span style="color: #339933;">=</span> <span style="color: #003399;">Integer</span>.<span style="color: #006633;">MAX_VALUE</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">return</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">final</span> Entry<span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> newTable <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> Entry<span style="color: #009900;">&#91;</span>newCapacity<span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
    transfer<span style="color: #009900;">&#40;</span>newTable<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    table <span style="color: #339933;">=</span> newTable<span style="color: #339933;">;</span>
    threshold <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#40;</span>newCapacity <span style="color: #339933;">*</span> loadFactor<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #000066; font-weight: bold;">void</span> transfer<span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> Entry<span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> newTable<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">final</span> Entry<span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span> src <span style="color: #339933;">=</span> table<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> newCapacity <span style="color: #339933;">=</span> newTable.<span style="color: #006633;">length</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">long</span> time1 <span style="color: #339933;">=</span> <span style="color: #003399;">System</span>.<span style="color: #006633;">currentTimeMillis</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span> j <span style="color: #339933;">=</span> <span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span> j <span style="color: #339933;">&lt;</span> src.<span style="color: #006633;">length</span><span style="color: #339933;">;</span> j<span style="color: #339933;">++</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      Entry<span style="color: #339933;">&lt;</span>K,V<span style="color: #339933;">&gt;</span> e <span style="color: #339933;">=</span> src<span style="color: #009900;">&#91;</span>j<span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>e <span style="color: #339933;">!=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        src<span style="color: #009900;">&#91;</span>j<span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #339933;">;</span>
        <span style="color: #000066; font-weight: bold;">int</span> k<span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span><span style="color: #666666; font-style: italic;">//@标记2</span>
        <span style="color: #000000; font-weight: bold;">do</span> <span style="color: #009900;">&#123;</span>
          <span style="color: #000000; font-weight: bold;">final</span> Entry<span style="color: #339933;">&lt;</span>K,V<span style="color: #339933;">&gt;</span> next <span style="color: #339933;">=</span> e.<span style="color: #006633;">next</span><span style="color: #339933;">;</span>
          <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000066; font-weight: bold;">int</span> i <span style="color: #339933;">=</span> indexFor<span style="color: #009900;">&#40;</span>e.<span style="color: #006633;">hash</span>, newCapacity<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
          e.<span style="color: #006633;">next</span> <span style="color: #339933;">=</span> newTable<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
          newTable<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> e<span style="color: #339933;">;</span>
          e <span style="color: #339933;">=</span> next<span style="color: #339933;">;</span>
          <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>k<span style="color: #339933;">++</span> <span style="color: #339933;">&gt;</span> <span style="color: #cc66cc;">1000</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span><span style="color: #666666; font-style: italic;">//@标记3</span>
            <span style="color: #003399;">System</span>.<span style="color: #006633;">out</span>.<span style="color: #006633;">println</span><span style="color: #009900;">&#40;</span><span style="color: #003399;">Thread</span>.<span style="color: #006633;">currentThread</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">getName</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">+</span>
                <span style="color: #0000ff;">&quot;,e==next:&quot;</span><span style="color: #339933;">+</span><span style="color: #009900;">&#40;</span>e<span style="color: #339933;">==</span>e.<span style="color: #006633;">next</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">+</span><span style="color: #0000ff;">&quot;,e==next.next:&quot;</span><span style="color: #339933;">+</span><span style="color: #009900;">&#40;</span>e<span style="color: #339933;">==</span>e.<span style="color: #006633;">next</span>.<span style="color: #006633;">next</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">+</span>
                <span style="color: #0000ff;">&quot;,e:&quot;</span><span style="color: #339933;">+</span>e<span style="color: #339933;">+</span><span style="color: #0000ff;">&quot;,next:&quot;</span><span style="color: #339933;">+</span>e.<span style="color: #006633;">next</span><span style="color: #339933;">+</span><span style="color: #0000ff;">&quot;,eq:&quot;</span><span style="color: #339933;">+</span>e.<span style="color: #006633;">equals</span><span style="color: #009900;">&#40;</span>e.<span style="color: #006633;">next</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
              <span style="color: #003399;">Thread</span>.<span style="color: #006633;">sleep</span><span style="color: #009900;">&#40;</span><span style="color: #cc66cc;">2000</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">final</span> <span style="color: #003399;">Exception</span> e2<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
            <span style="color: #009900;">&#125;</span>
&nbsp;
          <span style="color: #009900;">&#125;</span>
        <span style="color: #009900;">&#125;</span> <span style="color: #000000; font-weight: bold;">while</span> <span style="color: #009900;">&#40;</span>e <span style="color: #339933;">!=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span>
  <span style="color: #009900;">&#125;</span></pre></div></div>

<p>  通过jconsole（或者thread dump），可以看到线程停在了transfer方法的while循环处。这个transfer方法的作用是，当Map中元素数超过阈值需要resize时，它负责把原Map中的元素映射到新Map中。我修改了HashMap，加上了@标记2和@标记3的代码片断，以打印出死循环时的状态，结果死循环线程总是出现类似这样的输出：“Thread-1,e==next:false,e==next.next:true,e:108928=108928,next:108928=108928,eq:true”。<br />
这个输出表明：<br />
  1）这个Entry链中的两个Entry之间的关系是：e=e.next.next，造成死循环。<br />
  2）e.equals(e.next)，但e!=e.next。因为测试例子中两个线程put的内容一样，并发时可能同一个key被保存了多个value，这种错误是在addEntry函数产生的，但这和线程死循环没有关系。</p>
<p>  接下来就分析transfer中那个while循环了。先所说这个循环正常的功能：src[j]保存的是映射成同一个hash值的多个Entry的链表，这个src[j]可能为null，可能只有一个Entry，也可能由多个Entry链接起来。假设是多个Entry，原来的链是(src[j]=a)->b（也就是src[j]=a,a.next=b,b.next=null），经过while处理后得到了(newTable[i]=b)->a。也就是说，把链表的next关系反向了。</p>
<p>  再看看这个while中可能在多线程情况下引起问题的语句。针对两个线程t1和t2,这里它们可能的产生问题的执行序列做些个人分析：</p>
<p>  1）假设同一个Entry列表[e->f->...]，t1先到，t2后到并都走到while中。t1执行“e.next = newTable[i];newTable[i] = e;”这使得e.next=null（初始的newTable[i]为null），newTable[i]指向了e。这时t2执行了“e.next = newTable[i];newTable[i] = e;”，这使得e.next=e，e死循环了。因为循环开始处的“final Entry<K,V> next = e.next;”，尽管e自己死循环了，在最后的“e = next;”后，两个线程都会跳过e继续执行下去。</p>
<p>  2）在while中逐个遍历Entry链表中的Entry而把next关系反向时，newTable[i]成为了被交换的引用，可疑的语句在于“e.next = newTable[i];”。假设链表e->f->g被t1处理成e<-f<-g，newTable[i]指向了g，这时t2进来了，它一执行“e.next = newTable[i];”就使得e->g，造成了死循环。所以，理论上来说，死循环的Entry个数可能很多。尽管产生了死循环，但是t1执行到了死循环的右边，所以是会继续执行下去的，而t2如果执行“final Entry<K,V> next = e.next;”的next为null，则也会继续执行下去，否则就进入了死循环。</p>
<p>  3）似乎情况会更复杂，因为即便线程跳出了死循环，它下一次做resize进入transfer时，有可能因为之前的死循环Entry链表而被hang住（似乎是一定会被hang住）。也有可能，在put检查Entry链表时（@标记1），因为Entry链表的死循环而被hang住。也似乎有可能，活着的线程和死循环的线程同时执行在while里后，两个线程都能活着出去。所以，可能两个线程平安退出，可能一个线程hang在transfer中，可能两个线程都被hang住而又不一定在一个地方。</p>
<p>  4）我反复的测试，出现一个线程被hang住的情况最多，都是e=e.next.next造成的，这主要就是例子put两份增量数据造成的。我如果去掉@标记3的输出，有时也能复现两个线程都被hang住的情况，但加上后就很难复现出来。我又把put的数据改了下，比如让两个线程put范围不同的数据，就能复现出e=e.next，两个线程都被hang住的情况。</p>
<p>  上面罗哩罗嗦了很多，一开始我简单的分析后觉得似乎明白了怎么回事，可现在仔细琢磨后似乎又不明白了许多。有一个细节是，每次死循环的key的大小也是有据可循的，我就不打哈了。感觉，如果样本多些，可能出现问题的原因点会很多，也会更复杂，我姑且不再蛋疼下去。至于有人提到ConcurrentHashMap也有这个问题，我觉得不大可能，因为它的put操作是加锁的，如果有这个问题就不叫线程安全的Map了。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/08/286.html/feed</wfw:commentRss>
		<slash:comments>8</slash:comments>
		</item>
		<item>
		<title>PHP中的“syntax error, unexpected T_PAAMAYIM_NEKUDOTAYIM”错误</title>
		<link>http://www.kafka0102.com/2010/08/281.html</link>
		<comments>http://www.kafka0102.com/2010/08/281.html#comments</comments>
		<pubDate>Fri, 06 Aug 2010 17:20:58 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[php]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=281</guid>
		<description><![CDATA[因为需要，今天晚些在本机使用PHP做些测试，PHP脚本依赖了一堆我也不清楚做什么用的库。结果一跑起来，就报出类似下面的错误：“Parse error: syntax error, unexpected T_PAAMAYIM_NEKUDOTAYIM in /home/kafka/test/test.php on line 8”。查找代码，发现报错的代码类似：“$class_name::func1();”，也就是使用一个表示类名的字符串变量来调用它的静态方法，并且是解析时的语法错误（我第一眼看到::时，脑子里浮现的是C++里的作用域符号，好长时间后才想起PHP里有::也有这种东西，我也是用过 self::doSomething()的）。]]></description>
			<content:encoded><![CDATA[<p>因为需要，今天晚些在本机使用PHP做些测试，PHP脚本依赖了一堆我也不清楚做什么用的库。结果一跑起来，就报出类似下面的错误：“Parse error: syntax error, unexpected T_PAAMAYIM_NEKUDOTAYIM in /home/kafka/test/test.php on line 8”。查找代码，发现报错的代码类似：“$class_name::func1();”，也就是使用一个表示类名的字符串变量来调用它的静态方法，并且是解析时的语法错误（我第一眼看到::时，脑子里浮现的是C++里的作用域符号，好长时间后才想起PHP里::也有这种东西，我也是用过self::doSomething()的）。这代码在测试机和生产机跑着呢，应该不会有问题。就到测试机测试了一下，果然没问题。对比PHP的版本，测试机的是最新的5.3.3,而我的是5.2.13。原因估计就是版本或者配置方面的不同造成了。于是google之，我勒个去，结果一堆页面都是在热火朝天的讨论PAAMAYIM_NEKUDOTAYIM这个怪异的词组什么意思，看得多了，我也明白了，Paamayim Nekudotayim是希伯来语，表示双冒号的意思，也就是double-colon，但却没看到有人提提这个error如何解决。后来总算在官网 http://www.php.net/manual/en/language.oop5.paamayim-nekudotayim.php 找到答案，原来“$class_name::func1();”这种用法是5.3以后才支持的。我晕！重新安装了最新的PHP后，程序正常了。好吧，与时俱进很重要。</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/08/281.html/feed</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>[Solr实践]Solr Cache使用介绍及分析</title>
		<link>http://www.kafka0102.com/2010/08/267.html</link>
		<comments>http://www.kafka0102.com/2010/08/267.html#comments</comments>
		<pubDate>Sat, 31 Jul 2010 21:15:27 +0000</pubDate>
		<dc:creator>kafka0102</dc:creator>
				<category><![CDATA[solr]]></category>
		<category><![CDATA[solr cache]]></category>
		<category><![CDATA[solr 实践]]></category>

		<guid isPermaLink="false">http://www.kafka0102.com/?p=267</guid>
		<description><![CDATA[	本文将介绍Solr查询中涉及到的Cache使用及相关的实现。Solr查询的核心类就是SolrIndexSearcher，每个core通常在同一时刻只由当前的SolrIndexSearcher供上层的handler使用（当切换SolrIndexSearcher时可能会有两个同时提供服务），而Solr的各种Cache是依附于SolrIndexSearcher的，SolrIndexSearcher在则Cache生，SolrIndexSearcher亡则Cache被清空close掉。Solr中的应用Cache有filterCache、queryResultCache、documentCache等，这些Cache都是SolrCache的实现类，并且是SolrIndexSearcher的成员变量，各自有着不同的逻辑和使命，下面分别予以介绍和分析。]]></description>
			<content:encoded><![CDATA[<p>	本文将介绍Solr查询中涉及到的Cache使用及相关的实现。Solr查询的核心类就是SolrIndexSearcher，每个core通常在同一时刻只由当前的SolrIndexSearcher供上层的handler使用（当切换SolrIndexSearcher时可能会有两个同时提供服务），而Solr的各种Cache是依附于SolrIndexSearcher的，SolrIndexSearcher在则Cache生，SolrIndexSearcher亡则Cache被清空close掉。Solr中的应用Cache有filterCache、queryResultCache、documentCache等，这些Cache都是SolrCache的实现类，并且是SolrIndexSearcher的成员变量，各自有着不同的逻辑和使命，下面分别予以介绍和分析。</p>
<h2>1、SolrCache接口实现类</h2>
<p>	Solr提供了两种SolrCache接口实现类：solr.search.LRUCache和solr.search.FastLRUCache。FastLRUCache是1.4版本中引入的，其速度在普遍意义上要比LRUCache更fast些。<br />
	下面是对SolrCache接口主要方法的注释：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">interface</span> SolrCache <span style="color: #009900;">&#123;</span>
  <span style="color: #008000; font-style: italic; font-weight: bold;">/**
   * Solr在解析配置文件构造SolrConfig实例时会初始化配置中的各种CacheConfig，
   * 在构造SolrIndexSearcher时通过SolrConfig实例来newInstance SolrCache，
   * 这会调用init方法。参数args就是和具体实现（LRUCache和FastLRUCache）相关的
   * 参数Map，参数persistence是个全局的东西，LRUCache和FastLRUCache用其来统计
   * cache访问情况（因为cache是和SolrIndexSearcher绑定的，所以这种统计就需要个
   * 全局的注入参数），参数regenerator是autowarm时如何重新加载cache，
   * CacheRegenerator接口只有一个被SolrCache warm方法回调的方法：
   * boolean regenerateItem(SolrIndexSearcher newSearcher,
   * SolrCache newCache, SolrCache oldCache, Object oldKey, Object oldVal)
   */</span>
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> init<span style="color: #009900;">&#40;</span><span style="color: #003399;">Map</span> args, <span style="color: #003399;">Object</span> persistence, CacheRegenerator regenerator<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #008000; font-style: italic; font-weight: bold;">/** :TODO: copy from Map */</span>
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">int</span> size<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #008000; font-style: italic; font-weight: bold;">/** :TODO: copy from Map */</span>
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> put<span style="color: #009900;">&#40;</span><span style="color: #003399;">Object</span> key, <span style="color: #003399;">Object</span> value<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #008000; font-style: italic; font-weight: bold;">/** :TODO: copy from Map */</span>
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">Object</span> get<span style="color: #009900;">&#40;</span><span style="color: #003399;">Object</span> key<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #008000; font-style: italic; font-weight: bold;">/** :TODO: copy from Map */</span>
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> clear<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #008000; font-style: italic; font-weight: bold;">/**
   * 新创建的SolrIndexSearcher autowarm方法，该方法的实现就是遍历已有cache中合适的
   * 范围（因为通常不会把旧cache中的所有项都重新加载一遍），对每一项调用regenerator的
   * regenerateItem方法来对searcher加载新cache项。
   */</span>
  <span style="color: #000066; font-weight: bold;">void</span> warm<span style="color: #009900;">&#40;</span>SolrIndexSearcher searcher, SolrCache old<span style="color: #009900;">&#41;</span> <span style="color: #000000; font-weight: bold;">throws</span> <span style="color: #003399;">IOException</span><span style="color: #339933;">;</span>
  <span style="color: #008000; font-style: italic; font-weight: bold;">/** Frees any non-memory resources */</span>
  <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">void</span> close<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<h3>1.1、solr.search.LRUCache </h3>
<p>LRUCache可配置参数如下：<br />
1）size：cache中可保存的最大的项数，默认是1024<br />
2）initialSize：cache初始化时的大小，默认是1024。<br />
3）autowarmCount：当切换SolrIndexSearcher时，可以对新生成的SolrIndexSearcher做autowarm（预热）处理。autowarmCount表示从旧的SolrIndexSearcher中取多少项来在新的SolrIndexSearcher中被重新生成，如何重新生成由CacheRegenerator实现。在当前的1.4版本的Solr中，这个autowarmCount只能取预热的项数，将来的4.0版本可以指定为已有cache项数的百分比，以便能更好的平衡autowarm的开销及效果。如果不指定该参数，则表示不做autowarm处理。<br />
	实现上，LRUCache直接使用LinkedHashMap来缓存数据，由initialSize来限定cache的大小，淘汰策略也是使用LinkedHashMap的内置的LRU方式，读写操作都是对map的全局锁，所以并发性效果方面稍差。</p>
<h3>1.2、solr.search.FastLRUCache</h3>
<p>在配置方面，FastLRUCache除了需要LRUCache的参数，还可有选择性的指定下面的参数：<br />
1）minSize：当cache达到它的最大数，淘汰策略使其降到minSize大小，默认是0.9*size。<br />
2）acceptableSize：当淘汰数据时，期望能降到minSize，但可能会做不到，则可勉为其难的降到acceptableSize，默认是0.95*size。<br />
3）cleanupThread：相比LRUCache是在put操作中同步进行淘汰工作，FastLRUCache可选择由独立的线程来做，也就是配置cleanupThread的时候。当cache大小很大时，每一次的淘汰数据就可能会花费较长时间，这对于提供查询请求的线程来说就不太合适，由独立的后台线程来做就很有必要。<br />
	实现上，FastLRUCache内部使用了ConcurrentLRUCache来缓存数据，它是个加了LRU淘汰策略的ConcurrentHashMap，所以其并发性要好很多，这也是多数Java版Cache的极典型实现。</p>
<h2>2、filterCache</h2>
<p>	filterCache存储了无序的lucene document id集合，该cache有3种用途：<br />
	1）filterCache存储了filter queries(&#8220;fq&#8221;参数)得到的document id集合结果。Solr中的query参数有两种，即q和fq。如果fq存在，Solr是先查询fq（因为fq可以多个，所以多个fq查询是个取结果交集的过程），之后将fq结果和q结果取并。在这一过程中，filterCache就是key为单个fq（类型为Query），value为document id集合（类型为DocSet）的cache。对于fq为range query来说，filterCache表现出其有价值的一面。<br />
	2）filterCache还可用于facet查询（http://wiki.apache.org/solr/SolrFacetingOverview），facet查询中各facet的计数是通过对满足query条件的document id集合（可涉及到filterCache）的处理得到的。因为统计各facet计数可能会涉及到所有的doc id，所以filterCache的大小需要能容下索引的文档数。<br />
	3）如果solfconfig.xml中配置了&lt;useFilterForSortedQuery/&gt;，那么如果查询有filter（此filter是一需要过滤的DocSet，而不是fq，我未见得它有什么用），则使用filterCache。<br />
下面是filterCache的配置示例：</p>

<div class="wp_syntax"><div class="code"><pre class="xml" style="font-family:monospace;">    <span style="color: #808080; font-style: italic;">&lt;!-- Internal cache used by SolrIndexSearcher for filters (DocSets),</span>
<span style="color: #808080; font-style: italic;">         unordered sets of *all* documents that match a query.</span>
<span style="color: #808080; font-style: italic;">         When a new searcher is opened, its caches may be prepopulated</span>
<span style="color: #808080; font-style: italic;">         or &quot;autowarmed&quot; using data from caches in the old searcher.</span>
<span style="color: #808080; font-style: italic;">         autowarmCount is the number of items to prepopulate.  For LRUCache,</span>
<span style="color: #808080; font-style: italic;">         the prepopulated items will be the most recently accessed items.</span>
<span style="color: #808080; font-style: italic;">      --&gt;</span>
    <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;filterCache</span></span>
<span style="color: #009900;">      <span style="color: #000066;">class</span>=<span style="color: #ff0000;">&quot;solr.LRUCache&quot;</span></span>
<span style="color: #009900;">      <span style="color: #000066;">size</span>=<span style="color: #ff0000;">&quot;16384&quot;</span></span>
<span style="color: #009900;">      <span style="color: #000066;">initialSize</span>=<span style="color: #ff0000;">&quot;4096&quot;</span></span>
<span style="color: #009900;">      <span style="color: #000066;">autowarmCount</span>=<span style="color: #ff0000;">&quot;4096&quot;</span><span style="color: #000000; font-weight: bold;">/&gt;</span></span></pre></div></div>

<p>      对于是否使用filterCache及如何配置filterCache大小，需要根据应用特点、统计、效果、经验等各方面来评估。对于使用fq、facet的应用，对filterCache的调优是很有必要的。</p>
<h2>3、queryResultCache</h2>
<p>	顾名思义，queryResultCache是对查询结果的缓存（SolrIndexSearcher中的cache缓存的都是document id set），这个结果就是针对查询条件的完全有序的结果。下面是它的配置示例：</p>

<div class="wp_syntax"><div class="code"><pre class="xml" style="font-family:monospace;">	    <span style="color: #808080; font-style: italic;">&lt;!-- queryResultCache caches results of searches - ordered lists of</span>
<span style="color: #808080; font-style: italic;">         document ids (DocList) based on a query, a sort, and the range</span>
<span style="color: #808080; font-style: italic;">         of documents requested.</span>
<span style="color: #808080; font-style: italic;">      --&gt;</span>
    <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;queryResultCache</span></span>
<span style="color: #009900;">      <span style="color: #000066;">class</span>=<span style="color: #ff0000;">&quot;solr.LRUCache&quot;</span></span>
<span style="color: #009900;">      <span style="color: #000066;">size</span>=<span style="color: #ff0000;">&quot;16384&quot;</span></span>
<span style="color: #009900;">      <span style="color: #000066;">initialSize</span>=<span style="color: #ff0000;">&quot;4096&quot;</span></span>
<span style="color: #009900;">      <span style="color: #000066;">autowarmCount</span>=<span style="color: #ff0000;">&quot;1024&quot;</span><span style="color: #000000; font-weight: bold;">/&gt;</span></span></pre></div></div>

<p>      缓存的key是个什么结构呢？就是下面的类（key的hashcode就是QueryResultKey的成员变量hc）：</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">public</span> QueryResultKey<span style="color: #009900;">&#40;</span>Query query, List<span style="color: #339933;">&lt;</span>Query<span style="color: #339933;">&gt;</span> filters, Sort sort, <span style="color: #000066; font-weight: bold;">int</span> nc_flags<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">query</span> <span style="color: #339933;">=</span> query<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">sort</span> <span style="color: #339933;">=</span> sort<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">filters</span> <span style="color: #339933;">=</span> filters<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">nc_flags</span> <span style="color: #339933;">=</span> nc_flags<span style="color: #339933;">;</span>
    <span style="color: #000066; font-weight: bold;">int</span> h <span style="color: #339933;">=</span> query.<span style="color: #006633;">hashCode</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>filters <span style="color: #339933;">!=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> h <span style="color: #339933;">^=</span> filters.<span style="color: #006633;">hashCode</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    sfields <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span><span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">sort</span> <span style="color: #339933;">!=</span><span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">?</span> <span style="color: #000000; font-weight: bold;">this</span>.<span style="color: #006633;">sort</span>.<span style="color: #006633;">getSort</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">:</span> defaultSort<span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span>SortField sf <span style="color: #339933;">:</span> sfields<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #666666; font-style: italic;">// mix the bits so that sortFields are position dependent</span>
      <span style="color: #666666; font-style: italic;">// so that a,b won't hash to the same value as b,a</span>
      h <span style="color: #339933;">^=</span> <span style="color: #009900;">&#40;</span>h <span style="color: #339933;">&lt;&lt;</span> <span style="color: #cc66cc;">8</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">|</span> <span style="color: #009900;">&#40;</span>h <span style="color: #339933;">&gt;&gt;&gt;</span> <span style="color: #cc66cc;">25</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>   <span style="color: #666666; font-style: italic;">// reversible hash</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>sf.<span style="color: #006633;">getField</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">!=</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> h <span style="color: #339933;">+=</span> sf.<span style="color: #006633;">getField</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">hashCode</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      h <span style="color: #339933;">+=</span> sf.<span style="color: #006633;">getType</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>sf.<span style="color: #006633;">getReverse</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span> h<span style="color: #339933;">=</span>~h<span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>sf.<span style="color: #006633;">getLocale</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">!=</span><span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> h<span style="color: #339933;">+=</span>sf.<span style="color: #006633;">getLocale</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">hashCode</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>sf.<span style="color: #006633;">getFactory</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">!=</span><span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span> h<span style="color: #339933;">+=</span>sf.<span style="color: #006633;">getFactory</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">hashCode</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    hc <span style="color: #339933;">=</span> h<span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span></pre></div></div>

<p>  因为查询参数是有start和rows的，所以某个QueryResultKey可能命中了cache，但start和rows却不在cache的document id set范围内。当然，document id set是越大命中的概率越大，但这也会很浪费内存，这就需要个参数：queryResultWindowSize来指定document id set的大小。Solr中默认取值为50,可配置，WIKI上的解释很深简单明了：</p>

<div class="wp_syntax"><div class="code"><pre class="xml" style="font-family:monospace;">      <span style="color: #808080; font-style: italic;">&lt;!-- An optimization for use with the queryResultCache.  When a search</span>
<span style="color: #808080; font-style: italic;">         is requested, a superset of the requested number of document ids</span>
<span style="color: #808080; font-style: italic;">         are collected.  For example, of a search for a particular query</span>
<span style="color: #808080; font-style: italic;">         requests matching documents 10 through 19, and queryWindowSize is 50,</span>
<span style="color: #808080; font-style: italic;">         then documents 0 through 50 will be collected and cached.  Any further</span>
<span style="color: #808080; font-style: italic;">         requests in that range can be satisfied via the cache.</span>
<span style="color: #808080; font-style: italic;">    --&gt;</span>
    <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;queryResultWindowSize<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>50<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/queryResultWindowSize<span style="color: #000000; font-weight: bold;">&gt;</span></span></span></pre></div></div>

<p>  相比filterCache来说，queryResultCache内存使用上要更少一些，但它的效果如何就很难说。就索引数据来说，通常我们只是在索引上存储应用主键id，再从数据库等数据源获取其他需要的字段。这使得查询过程变成，首先通过solr得到document id set，再由Solr得到应用id集合，最后从外部数据源得到完成的查询结果。如果对查询结果正确性没有苛刻的要求，可以在Solr之外独立的缓存完整的查询结果（定时作废），这时queryResultCache就不是很有必要，否则可以考虑使用queryResultCache。当然，如果发现在queryResultCache生命周期内，query重合度很低，也不是很有必要开着它。</p>
<h2>4、documentCache</h2>
<p>	又顾名思义，documentCache用来保存&lt;doc_id,document&gt;对的。如果使用documentCache，就尽可能开大些，至少要大过&lt;max_results&gt; * &lt;max_concurrent_queries&gt;，否则因为cache的淘汰，一次请求期间还需要重新获取document一次。也要注意document中存储的字段的多少，避免大量的内存消耗。<br />
	下面是documentCache的配置示例：</p>

<div class="wp_syntax"><div class="code"><pre class="xml" style="font-family:monospace;">	<span style="color: #808080; font-style: italic;">&lt;!-- documentCache caches Lucene Document objects (the stored fields for each document).</span>
<span style="color: #808080; font-style: italic;">      --&gt;</span>
    <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;documentCache</span></span>
<span style="color: #009900;">      <span style="color: #000066;">class</span>=<span style="color: #ff0000;">&quot;solr.LRUCache&quot;</span></span>
<span style="color: #009900;">      <span style="color: #000066;">size</span>=<span style="color: #ff0000;">&quot;16384&quot;</span></span>
<span style="color: #009900;">      <span style="color: #000066;">initialSize</span>=<span style="color: #ff0000;">&quot;16384&quot;</span><span style="color: #000000; font-weight: bold;">/&gt;</span></span></pre></div></div>

<h2>5、User/Generic Caches</h2>
<p>Solr支持自定义Cache，只需要实现自定义的regenerator即可，下面是配置示例：</p>

<div class="wp_syntax"><div class="code"><pre class="xml" style="font-family:monospace;">    <span style="color: #808080; font-style: italic;">&lt;!-- Example of a generic cache.  These caches may be accessed by name</span>
<span style="color: #808080; font-style: italic;">         through SolrIndexSearcher.getCache(),cacheLookup(), and cacheInsert().</span>
<span style="color: #808080; font-style: italic;">         The purpose is to enable easy caching of user/application level data.</span>
<span style="color: #808080; font-style: italic;">         The regenerator argument should be specified as an implementation</span>
<span style="color: #808080; font-style: italic;">         of solr.search.CacheRegenerator if autowarming is desired.</span>
<span style="color: #808080; font-style: italic;">    --&gt;</span>
    <span style="color: #808080; font-style: italic;">&lt;!--</span>
<span style="color: #808080; font-style: italic;">    &lt;cache name=&quot;yourCacheNameHere&quot;</span>
<span style="color: #808080; font-style: italic;">      class=&quot;solr.LRUCache&quot;</span>
<span style="color: #808080; font-style: italic;">      size=&quot;4096&quot;</span>
<span style="color: #808080; font-style: italic;">      initialSize=&quot;2048&quot;</span>
<span style="color: #808080; font-style: italic;">      autowarmCount=&quot;4096&quot;</span>
<span style="color: #808080; font-style: italic;">      regenerator=&quot;org.foo.bar.YourRegenerator&quot;/&gt;</span>
<span style="color: #808080; font-style: italic;">    --&gt;</span></pre></div></div>

<h2>6、The Lucene FieldCache</h2>
<p>	lucene中有相对低级别的FieldCache，Solr并不对它做管理，所以，lucene的FieldCache还是由lucene的IndexSearcher来搞。</p>
<h2>7、autowarm</h2>
<p>	上面有提到autowarm，autowarm触发的时机有两个，一个是创建第一个Searcher时（firstSearcher），一个是创建个新Searcher（newSearcher）来代替当前的Searcher。在Searcher提供请求服务前，Searcher中的各个Cache可以做warm处理，处理的地方通常是SolrCache的init方法，而不同cache的warm策略也不一样。<br />
	1）filterCache：filterCache注册了下面的CacheRegenerator，就是由旧的key查询索引得到新值put到新cache中。</p>

<div class="wp_syntax"><div class="code"><pre class="java" style="font-family:monospace;">	      solrConfig.<span style="color: #006633;">filterCacheConfig</span>.<span style="color: #006633;">setRegenerator</span><span style="color: #009900;">&#40;</span>
              <span style="color: #000000; font-weight: bold;">new</span> CacheRegenerator<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">boolean</span> regenerateItem<span style="color: #009900;">&#40;</span>SolrIndexSearcher newSearcher, SolrCache newCache, SolrCache oldCache, <span style="color: #003399;">Object</span> oldKey, <span style="color: #003399;">Object</span> oldVal<span style="color: #009900;">&#41;</span> <span style="color: #000000; font-weight: bold;">throws</span> <span style="color: #003399;">IOException</span> <span style="color: #009900;">&#123;</span>
                  newSearcher.<span style="color: #006633;">cacheDocSet</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#40;</span>Query<span style="color: #009900;">&#41;</span>oldKey, <span style="color: #000066; font-weight: bold;">null</span>, <span style="color: #000066; font-weight: bold;">false</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                  <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">true</span><span style="color: #339933;">;</span>
                <span style="color: #009900;">&#125;</span>
              <span style="color: #009900;">&#125;</span>
      <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>

<p>      2）queryResultCache：queryResultCache的autowarm不在SolrCache的init（也就是说，不是去遍历已有的queryResultCache中的query key执行查询），而是通过SolrEventListener接口的void newSearcher(SolrIndexSearcher newSearcher, SolrIndexSearcher currentSearcher)方法，来执行配置中特定的query查询，达到显示的预热lucene FieldCache的效果。<br />
      queryResultCache的配置示例如下：</p>

<div class="wp_syntax"><div class="code"><pre class="xml" style="font-family:monospace;">          <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;listener</span> <span style="color: #000066;">event</span>=<span style="color: #ff0000;">&quot;newSearcher&quot;</span> <span style="color: #000066;">class</span>=<span style="color: #ff0000;">&quot;solr.QuerySenderListener&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>
      <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;arr</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;queries&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>
        <span style="color: #808080; font-style: italic;">&lt;!-- seed common sort fields --&gt;</span>
        <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span> <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;q&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>anything<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span> <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;sort&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>name desc price desc populartiy desc<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span> <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
      <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/arr<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
    <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/listener<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
    <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;listener</span> <span style="color: #000066;">event</span>=<span style="color: #ff0000;">&quot;firstSearcher&quot;</span> <span style="color: #000066;">class</span>=<span style="color: #ff0000;">&quot;solr.QuerySenderListener&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>
      <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;arr</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;queries&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>
        <span style="color: #808080; font-style: italic;">&lt;!-- seed common sort fields --&gt;</span>
        <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span> <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;q&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>anything<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span> <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;sort&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>name desc, price desc, populartiy desc<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span> <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
        <span style="color: #808080; font-style: italic;">&lt;!-- seed common facets and filter queries --&gt;</span>
        <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span> <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;q&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>anything<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span> 
              <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;facet.field&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>category<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span> 
              <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;fq&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>inStock:true<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
              <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;str</span> <span style="color: #000066;">name</span>=<span style="color: #ff0000;">&quot;fq&quot;</span><span style="color: #000000; font-weight: bold;">&gt;</span></span>price:[0 TO 100]<span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/str<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
        <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/lst<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
      <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/arr<span style="color: #000000; font-weight: bold;">&gt;</span></span></span>
    <span style="color: #009900;"><span style="color: #000000; font-weight: bold;">&lt;/listener<span style="color: #000000; font-weight: bold;">&gt;</span></span></span></pre></div></div>

<p>    3）documentCache：因为新索引的document id和索引文档的对应关系发生变化，所以documentCache没有warm的过程，落得白茫茫一片真干净。<br />
    尽管autowarm很好，也要注意autowarm带来的开销，这需要在实际中检验其warm的开销，也要注意Searcher的切换频率，避免因为warm和切换影响Searcher提供正常的查询服务。</p>
<h2>8、参考文章</h2>
<p>http://wiki.apache.org/solr/SolrCaching</p>
]]></content:encoded>
			<wfw:commentRss>http://www.kafka0102.com/2010/08/267.html/feed</wfw:commentRss>
		<slash:comments>5</slash:comments>
		</item>
	</channel>
</rss>

